{ "schemaVersion": 3, "dataset": { "version": 3, "date": "2026-08-13", "group": { "id": "ai-infrastructure", "name": "AI Infrastructure" }, "repository": { "id": "vllm", "repo": "vllm-project/vllm", "name": "vLLM", "keywords": [ "vLLM" ] }, "context": { "repository": "vllm-project/vllm", "url": "https://github.com/vllm-project/vllm", "description": "A high-throughput and memory-efficient inference and serving engine for LLMs", "homepage": "https://vllm.ai", "language": "Python", "topics": [ "amd", "blackwell", "cuda", "deepseek", "deepseek-v3", "gpt", "gpt-oss", "inference", "kimi", "llama", "llm", "llm-serving", "model-serving", "moe", "openai", "pytorch", "qwen", "qwen3", "tpu", "transformer" ], "license": "Apache-2.0", "defaultBranch": "main", "stars": 88978, "forks": 20650, "openIssues": 6543, "archived": false, "collectedAt": "2026-08-13T18:02:09.263295+00:00" }, "news": { "repository": "vllm-project/vllm", "collectedAt": "2026-08-13T18:02:09.263295+00:00", "latestRelease": { "repository": "vllm-project/vllm", "tag": "v0.27.1", "title": "v0.27.1", "url": "https://github.com/vllm-project/vllm/releases/tag/v0.27.1", "publishedAt": "2026-08-11T10:47:49Z", "notes": "This is a patch release on top of v0.27.0.\r\n\r\n- Support quantized DSpark Markov heads (#50424)", "highlights": [ "Support quantized DSpark Markov heads (#50424)" ], "prerelease": false }, "upcoming": [], "communityDiscussions": [] }, "runs": [ { "collectedAt": "2026-08-13T12:26:38.318Z", "since": "2026-08-12T12:26:38.318Z", "observedCount": 500, "changedCount": 500 }, { "collectedAt": "2026-08-13T13:48:00.446149Z", "since": "2026-08-12T13:48:00.446149Z", "observedCount": 500, "changedCount": 500 }, { "collectedAt": "2026-08-13T16:19:22.035158Z", "since": "2026-08-12T16:19:22.035158Z", "observedCount": 500, "changedCount": 113 }, { "collectedAt": "2026-08-13T17:43:20.785491Z", "since": "2026-08-12T17:43:20.785491Z", "observedCount": 500, "changedCount": 81 }, { "collectedAt": "2026-08-13T17:47:07.884300Z", "since": "2026-08-12T17:47:07.884300Z", "observedCount": 500, "changedCount": 2 }, { "collectedAt": "2026-08-13T18:01:55.420671Z", "since": "2026-08-12T18:01:55.420671Z", "observedCount": 500, "changedCount": 11 } ], "signals": [ { "id": "github:vllm-project/vllm:issue:23567", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: openai_harmony.HarmonyError: unexpected tokens remaining in message header", "text": "### Your current environment
I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header” In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: https://github.com/openai/harmony/issues/38 But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.
### 🐛 Describe the bug
I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header” In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: https://github.com/openai/harmony/issues/38 But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.
### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/23567", "createdAt": "2025-08-25T15:46:01Z", "updatedAt": "2026-08-13T02:16:11Z", "timestamp": "2026-08-13T02:16:11Z", "metrics": { "reactions": 24, "comments": 47 }, "labels": [ "bug", "stale", "gpt-oss" ], "author": "MoellerAI", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:26431", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: KV Cache Quantization not working on v1 (rtx3090) \"type fp8e4nv not supported in this architecture\"", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version : Could not collect CMake version : version 4.1.0 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.8.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.11 (main, Jun 4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.8.93 CUDA_MODULE_LOADING set to : LAZY GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 Nvidia driver version : 575.57.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: AuthenticAMD Model name: AMD Ryzen 5 7600 6-Core Processor CPU family: 25 Model: 97 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 2 CPU max MHz: 5170.0000 CPU min MHz: 545.0000 BogoMIPS: 7585.79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d Virtualization: AMD-V L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 6 MiB (6 instances) L3 cache: 32 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.14.1 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-ml-py==12.575.51 [pip3] nvidia-nccl-cu12==2.27.3 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pynvml==12.0.0 [pip3] pyzmq==27.1.0 [pip3] torch==2.8.0+cu128 [pip3] torchaudio==2.8.0+cu128 [pip3] torchvision==0.23.0+cu128 [pip3] transformers==4.57.0 [pip3] triton==3.4.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-11 0 N/A GPU1 PHB X 0-11 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 NCCL_VERSION=2.25.1-1 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=12.8.1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True LD_LIBRARY_PATH=/usr/local/cuda/lib64 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 CUDA_MODULE_LOADING=LAZY ```
### 🐛 Describe the bug im using the vllm docker image vllm/vllm-openai:v0.11.0 (same configuration worked in v0.10.2 but it had fallback to V0 ) With this command: command: > --model jart25/Qwen3-Coder-30B-A3B-Instruct-Int4-gptq --uvicorn-log-level \"info\" --gpu-memory-utilization 0.70 --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser \"hermes\" --max-model-len 128000 --dtype \"auto\" --kv_cache_dtype fp8_e5m2 It says ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") But i never specified fp8e4nv anywhere. it works when i disable kv cache quantization This is the Error including the stacktrace (Had to remove a bit from the end, as im hitting a character limit ):
Log output ``` Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 97%|█████████▋| 65/67 [00:06<00:00, 13.95it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:06<00:00, 13.72it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:06<00:00, 10.40it/s] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m Capturing CUDA graphs (decode, FULL): 0%| | 0/35 [00:00.98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_ops.py\", line 1243, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._op(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py\", line 614, in unified_attention_with_output vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self.impl.forward(self, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/triton_attn.py\", line 303, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton_reshape_and_cache_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/ops/triton_reshape_and_cache_flash.py\", line 153, in triton_reshape_and_cache_flash vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] reshape_and_cache_kernel_flash[grid]( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 390, in vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 594, in run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] kernel = self.compile(src, target=target, options=options.__dict__) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 339, in compile vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] module = src.make_ir(options, codegen_fns, module_map, context) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 83, in make_ir vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton.compiler.errors.CompilationError: at 1:0: vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def reshape_and_cache_kernel_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] Traceback (most recent call last): vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 666, in worker_busy_loop vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] output = func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 344, in compile_or_warm_up_model vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] cuda_graph_memory_bytes = self.model_runner.capture_model() vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3463, in capture_model vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._capture_cudagraphs( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3523, in _capture_cudagraphs vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._dummy_run(num_tokens, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 120, in decorate_context vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3152, in _dummy_run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] outputs = self.model( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py\", line 121, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self.runnable(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 675, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] hidden_states = self.model(input_ids, positions, intermediate_tensors, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py\", line 317, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] model_output = self.forward(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 403, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def forward( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 375, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super().__call__(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 929, in _fn vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return fn(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_ops.py\", line 1243, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._op(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py\", line 614, in unified_attention_with_output vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self.impl.forward(self, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/triton_attn.py\", line 303, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton_reshape_and_cache_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/ops/triton_reshape_and_cache_flash.py\", line 153, in triton_reshape_and_cache_flash vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] reshape_and_cache_kernel_flash[grid]( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 390, in vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 594, in run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] kernel = self.compile(src, target=target, options=options.__dict__) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 339, in compile vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] module = src.make_ir(options, codegen_fns, module_map, context) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 83, in make_ir vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton.compiler.errors.CompilationError: at 1:0: vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def reshape_and_cache_kernel_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] WorkerProc hit an exception. vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] Traceback (most recent call last): vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 666, in worker_busy_loop vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] output = func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 344, in compile_or_warm_up_model vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] cuda_graph_memory_bytes = self.model_runner.capture_model() vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3463, in capture_model vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._capture_cudagraphs( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3523, in _capture_cudagraphs vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._dummy_run(num_tokens, vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 120, in decorate_context vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3152, in _dummy_run vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] outputs = self.model( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py\", line 121, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self.runnable(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 675, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] hidden_states = self.model(input_ids, positions, intermediate_tensors, vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py\", line 317, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] model_output = self.forward(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 403, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def forward( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 375, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super().__call__(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 929, in _fn vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return fn(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ```
### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/26431", "createdAt": "2025-10-08T16:25:55Z", "updatedAt": "2026-08-13T02:16:05Z", "timestamp": "2026-08-13T02:16:05Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "bug", "stale" ], "author": "meganoob1337", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:29134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Fully Async Spec-Decoding | Make `seq_lens_cpu` in CommonAttentionMetadata optional", "text": "### Proposal to improve performance Currently fully overlapping input-prep with model forward pass is blocked in the spec-decode case by the following Host<>GPU syncs: 1) `_get_valid_sampled_token_count` (ultimately needed to compute `seq_lens_cpu`): https://github.com/vllm-project/vllm/blob/fe25772aa97beb8bcb07ea49e06a2892b521a7ed/vllm/v1/worker/gpu_model_runner.py#L3109 2) in the `num_speculated_tokens > 1` case by needing to update `seq_lens_cpu` for attention metadata building (specifically for all the speculated tokens after the first one hence only impacting `num_speculated_tokens > 1` context: https://github.com/vllm-project/vllm/pull/26498) Ultimately in-order to realize fully async spec decoding we need to build attention metadata without knowing `seq_lens_cpu` (using device `seq_lens` on device is fine since any metadata building GPU kernels will get queued to after this is updated by the CUDA driver). This is currently not entirely possible for all backends (namely FlashInfer do to D2H or H2D, depending on if a host or device tensor is provided, inside the [plan function](https://github.com/flashinfer-ai/flashinfer/blob/cce4952fdd41b353325e11d99e1fc0b0737961ff/flashinfer/decode.py#L940-L955)). However, there are many (important/default) backends like TRTLLM (inside the FlashInfer backend), FlashAttn, FlashAttn-MLA (for pure decode), FlashMLA (for pure decode) etc. that could achieve this full overlap using only the device `seq_lens` tensor. The proposal would be to update common attention metadata from: ``` @dataclass class CommonAttentionMetadata: ... seq_lens: torch.Tensor seq_lens_cpu: torch.Tensor ``` To something like: ``` @dataclass class CommonAttentionMetadata: ... seq_lens: torch.Tensor _seq_lens_cpu: torch.Tensor | None # WARNING: using this property will mean spec-decode with async-scheduling will not achieve # full overlap due to Host<>GPU sync @property def seq_lens_cpu(self): # Potentially log warning here to encourage developers to avoid this property return self. _seq_lens_cpu or seq_lens.to(\"cpu\") ``` Where `_seq_lens_cpu` would be provided if available/known (e.g. async-scheduling but no-spec-decode or no-async-scheduling). This will allow backends that do not need `seq_lens_cpu` to be able to benefit more form async-scheduling while maintaining support for backends that need `seq_lens_cpu` with little to no perf regression (we could potentially add warnings to developers to encourage them to migrate away from using this). NOTE: other changes to `GPUModelRunner._update_states` would be needed to fully realize this (or model runner v2 https://github.com/vllm-project/vllm/pull/25266); but the case of `num_speculated_tokens > 1` could benefit immediately by overlapping the drafter metadata prep with the target model's forward pass. NOTE: this would also mean `max_seq_len` would no-longer represent the true max but an upper-bound that could be off by `num_speculated_tokens`; this shouldn't be a problem given most backends (e.g. FA) use this simply for heuristics. We may want to consider renaming this to `seq_len_upper_bound` in-order to make it clear to backend developers this may not be exact. NOTE: `num_computed_tokens_cpu` would also need to be avoided but a similar approach could be used considering its a derivative of `query_start_loc` and `seq_lens` ``` @dataclass class CommonAttentionMetadata: ... # WARNING: using this property will mean spec-decode with async-scheduling will not achieve # full overlap due to Host<>GPU sync @property def num_computed_tokens_cpu(self): # Potentially log warning here to encourage developers to avoid this property return seq_lens_cpu - (query_start_loc_cpu[1:] - query_start_loc_cpu[0:]) ``` cc @benchislett @WoosukKwon @MatthewBonanni", "url": "https://github.com/vllm-project/vllm/issues/29134", "createdAt": "2025-11-21T00:48:02Z", "updatedAt": "2026-08-13T02:15:59Z", "timestamp": "2026-08-13T02:15:59Z", "metrics": { "reactions": 4, "comments": 6 }, "labels": [ "performance", "stale" ], "author": "LucasWilkinson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:29245", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Usage]: 启动 qwen3 vl 超级超级超级慢，sglang 启动很快，可能的原因是什么？", "text": "### Your current environment 连执行 python collect_env.py 都很慢，环境是直接 uv 安装的 ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.2 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 4.1.2 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.9.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.10.134-19.100.al8.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.9.86 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA L20Y GPU 1: NVIDIA L20Y GPU 2: NVIDIA L20Y GPU 3: NVIDIA L20Y GPU 4: NVIDIA L20Y GPU 5: NVIDIA L20Y GPU 6: NVIDIA L20Y GPU 7: NVIDIA L20Y Nvidia driver version : 570.148.08 cuDNN version : Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.10.2 HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8468V CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 70% CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 192 MiB (96 instances) L3 cache: 195 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.5.2 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.16.0 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.3.0 [pip3] nvidia-ml-py==13.580.82 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.3.20 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.9.0 [pip3] torchaudio==2.9.0 [pip3] torchvision==0.24.0 [pip3] transformers==4.57.1 [pip3] triton==3.5.0 [conda] flashinfer-python 0.5.2 pypi_0 pypi [conda] galore-torch 1.0 pypi_0 pypi [conda] nccl-xray 2.26.2rc2+xray1.1.0 pypi_0 pypi [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi [conda] nvidia-cuda-cccl-cu12 12.9.27 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cuda-nvcc-cu12 12.9.86 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.16.0 pypi_0 pypi [conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi [conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi [conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.2.1 pypi_0 pypi [conda] nvidia-ml-py 12.575.51 pypi_0 pypi [conda] nvidia-nccl-cu12 2.27.3 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi [conda] pyzmq 27.0.0 pypi_0 pypi [conda] torch 2.8.0 pypi_0 pypi [conda] torch-memory-saver 0.0.9 pypi_0 pypi [conda] torchao 0.9.0 pypi_0 pypi [conda] torchaudio 2.8.0 pypi_0 pypi [conda] torchdata 0.11.0 pypi_0 pypi [conda] torchmetrics 1.8.0 pypi_0 pypi [conda] torchvision 0.23.0 pypi_0 pypi [conda] transformer-engine-torch 2.5.0 pypi_0 pypi [conda] transformers 4.57.1 pypi_0 pypi [conda] triton 3.4.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 PIX NODE NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 NODE PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 NODE NODE PIX NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 NODE NODE NODE PIX SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS PIX NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS NODE PIX NODE NODE 48-95,144-191 1 N/A GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS NODE NODE PIX NODE 48-95,144-191 1 N/A GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=6,7,0,1,2,3,4,5 NCCL_IB_TC=160 CUBLAS_VERSION=12.9.1.4 NVIDIA_REQUIRE_CUDA=cuda>=9.0 NCCL_VERSION=2.27.3 NCCL_NET_GDR_LEVEL=2 NVIDIA_DRIVER_CAPABILITIES=compute,utility,video NCCL_DEBUG=INFO CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0 NVIDIA_PRODUCT_NAME=CUDA NCCL_IB_GID_INDEX=3 CUDA_VERSION=12.9.1.010 CUBLASMP_VERSION=0.4.0.789 CUDNN_FRONTEND_VERSION=1.12.0 CUDNN_VERSION=9.10.2.21 LD_LIBRARY_PATH=/usr/local/cuda/compat/lib.real:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_DRIVER_VERSION=575.57.08 PYTORCH_BUILD_VERSION=2.7.0 NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS= NVIDIA_PYTORCH_VERSION=25.06 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ``` 可以排除一些 IO 的问题，因为 sglang 启动很正常 ``` (trl) root@qs-44560-984753-ai-304574-default0-0:/env/trl# env CUDA_VISIBLE_DEVICES=6,7 vllm serve /path/qwen3vl-4b-1113/checkpoint-84000 --dtype auto --max-model-len 8192 -tp 2 --trust-remote-code DEBUG 11-23 04:33:31 [plugins/__init__.py:32] No plugins for group vllm.platform_plugins found. DEBUG 11-23 04:33:31 [platforms/__init__.py:36] Checking if TPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu' DEBUG 11-23 04:33:32 [platforms/__init__.py:61] Checking if CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:84] Confirmed CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:112] Checking if ROCm platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi' DEBUG 11-23 04:33:32 [platforms/__init__.py:133] Checking if XPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:153] XPU platform is not available because: No module named 'intel_extension_for_pytorch' DEBUG 11-23 04:33:32 [platforms/__init__.py:160] Checking if CPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:61] Checking if CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:84] Confirmed CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:225] Automatically detected platform cuda. DEBUG 11-23 04:34:23 [utils/flashinfer.py:45] flashinfer-cubin package was not found DEBUG 11-23 04:34:50 [entrypoints/utils.py:175] Setting VLLM_WORKER_MULTIPROC_METHOD to 'spawn' DEBUG 11-23 04:34:52 [plugins/__init__.py:40] Available plugins for group vllm.general_plugins: DEBUG 11-23 04:34:52 [plugins/__init__.py:42] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver DEBUG 11-23 04:34:52 [plugins/__init__.py:45] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. INFO 11-23 04:34:52 [config/scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=161589) INFO 11-23 04:34:52 [entrypoints/openai/api_server.py:1977] vLLM API server version 0.11.2 (APIServer pid=161589) INFO 11-23 04:34:52 [entrypoints/utils.py:253] non-default args: {'model_tag': '/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000', 'model': '/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000', 'trust_remote_code': True, 'max_model_len': 8192, 'tensor_parallel_size': 2} (APIServer pid=161589) INFO 11-23 04:34:52 [engine/arg_utils.py:592] HF_HUB_OFFLINE is True, replace model_id [/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000] to model_path [/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000] (APIServer pid=161589) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=161589) DEBUG 11-23 04:34:52 [model_executor/models/registry.py:652] Loaded model info for class vllm.model_executor.models.qwen3_vl.Qwen3VLForConditionalGeneration from cache (APIServer pid=161589) DEBUG 11-23 04:34:52 [logging_utils/log_time.py:29] Registry inspect model class: Elapsed time 0.0225311 secs (APIServer pid=161589) INFO 11-23 04:34:52 [config/model.py:631] Resolved architecture: Qwen3VLForConditionalGeneration (APIServer pid=161589) INFO 11-23 04:34:52 [config/model.py:1745] Using max model len 8192 (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:1958] Enabling chunked prefill by default (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:1976] Enabling prefix caching by default (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:2028] Defaulting max_num_batched_tokens to 8192 for OPENAI_API_SERVER usage context. (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:2038] Defaulting max_num_seqs to 1024 for OPENAI_API_SERVER usage context. (APIServer pid=161589) DEBUG 11-23 04:34:57 [config/parallel.py:589] Defaulting to use mp for distributed inference (APIServer pid=161589) INFO 11-23 04:34:57 [config/scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=8192. (APIServer pid=161589) DEBUG 11-23 04:34:59 [plugins/__init__.py:32] No plugins for group vllm.stat_logger_plugins found. (APIServer pid=161589) DEBUG 11-23 04:35:02 [plugins/io_processors/__init__.py:33] No IOProcessor plugins requested by the model (APIServer pid=161589) DEBUG 11-23 04:35:12 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:22 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:32 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:42 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. ``` ### How would you like to use vllm I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29245", "createdAt": "2025-11-22T20:41:27Z", "updatedAt": "2026-08-13T02:15:57Z", "timestamp": "2026-08-13T02:15:57Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [ "usage", "stale" ], "author": "hucorz", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:29341", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: sleep level 2 causes gibberish outputs", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 20.04.6 LTS (x86_64) GCC version : (Ubuntu 10.5.0-1ubuntu1~20.04) 10.5.0 Clang version : Could not collect CMake version : version 3.27.7 Libc version : glibc-2.31 ============================== PyTorch Info ============================== PyTorch version : 2.9.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime) Python platform : Linux-5.15.0-1048-aws-x86_64-with-glibc2.31 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.1.105 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H100 80GB HBM3 Nvidia driver version : 575.57.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 48 bits physical, 48 bits virtual CPU(s): 96 On-line CPU(s) list: 0-95 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7R13 Processor Stepping: 1 CPU MHz: 3491.190 BogoMIPS: 5299.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 3 MiB L1i cache: 3 MiB L2 cache: 48 MiB L3 cache: 384 MiB NUMA node0 CPU(s): 0-47 NUMA node1 CPU(s): 48-95 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.5.2 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.16.0 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.3.0 [pip3] nvidia-ml-py==13.580.82 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.3.20 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.9.0 [pip3] torchaudio==2.9.0 [pip3] torchvision==0.24.0 [pip3] transformers==4.57.2 [pip3] triton==3.5.0 [conda] flashinfer-python 0.5.2 pypi_0 pypi [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.16.0 pypi_0 pypi [conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi [conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi [conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.3.0 pypi_0 pypi [conda] nvidia-ml-py 13.580.82 pypi_0 pypi [conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi [conda] nvidia-nvshmem-cu12 3.3.20 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi [conda] pyzmq 27.1.0 py312hcf8288c_1 [conda] torch 2.9.0 pypi_0 pypi [conda] torchaudio 2.9.0 pypi_0 pypi [conda] torchvision 0.24.0 pypi_0 pypi [conda] transformers 4.57.2 pypi_0 pypi [conda] triton 3.5.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-10 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib::/opt/amazon/openmpi/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/efa/lib VLLM_LOGGING_LEVEL=ERROR PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
### 🐛 Describe the bug When using the sleep mode level 2, the model produces gibberish completions: ```python from vllm import LLM llm = LLM(model=\"openai/gpt-oss-20b\", enable_sleep_mode=True) llm.sleep(level=2) llm.wake_up() prompts = [[{\"role\": \"user\", \"content\": \"Where is the Machu Picchu located?\"}]] outputs = llm.chat(prompts) print(repr(outputs[0].outputs[0].text)) ``` ``` 'ocado \\'\" chemical optimal UriWord Beef nwanyị Mehmet性质Usuarioಡೆಯ profanity դեպիSleep Columbia' ``` **notes**: - with sleep level = 1, there is no such issue: ``` 'analysisUser asks location. Need to answer: Machu Picchu located' ``` - the same issue occurs with version 0.10.2, 0.11.0, 0.11.1 and 0.11.2 - the same issue occurs with transformers 4.57.0 and 5.0.0.dev0 - the same issue occurs with models Qwen3 and GPT-OSS, so I guess it affects all models. - `model_impl=\"transformers\"` doesn't solve the issue ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29341", "createdAt": "2025-11-24T20:31:46Z", "updatedAt": "2026-08-13T08:21:44Z", "timestamp": "2026-08-13T08:21:44Z", "metrics": { "reactions": 0, "comments": 15 }, "labels": [ "bug", "stale" ], "author": "qgallouedec", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:29405", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Qwen3 Omni Transcriptions", "text": "### 🚀 The feature, motivation and pitch Would love to see this model included in the supported as a Transcription model. According to the docs it looks like only 4 different models are supported as of now. https://docs.vllm.ai/en/latest/models/supported_models/#transcription ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29405", "createdAt": "2025-11-25T12:35:16Z", "updatedAt": "2026-08-13T02:15:52Z", "timestamp": "2026-08-13T02:15:52Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [ "good first issue", "feature request", "stale" ], "author": "justinlime", "state": "closed", "assignees": [ "mu-hashmi" ] }, { "id": "github:vllm-project/vllm:issue:29688", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "Document Ray Serve LLM incompatibility with vLLM v1 engine for PD disaggregation", "text": "## Summary vLLM v1 engine with `RayDistributedExecutor` is incompatible with Ray Serve LLM's `build_pd_openai_app` due to nested placement group conflicts. Users attempting PD (prefill/decode) disaggregation with Ray Serve LLM encounter silent failures. This should be documented to guide users toward working deployment patterns. ## Environment - **vLLM**: 0.10.0+ (v1 engine) - **Ray**: 2.43.0 - **Hardware**: AWS p5.48xlarge (8x H100 80GB) - **NIXL**: 0.7.1 ## Problem When deploying vLLM with Ray Serve LLM's PD disaggregation API: ```python from ray.serve.llm import build_pd_openai_app, LLMConfig app = build_pd_openai_app({ \"prefill_config\": prefill_llm_config, \"decode_config\": decode_llm_config, }) ``` The deployment fails with: ``` RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} ``` ## Root Cause 1. Ray Serve LLM pre-allocates GPUs in a placement group 2. vLLM v1 detects Ray environment and uses `RayDistributedExecutor` 3. `RayDistributedExecutor` attempts to create its own placement group 4. Inner placement group fails because GPUs are already reserved This creates a fundamental conflict between the two resource allocation strategies. ## Working Deployment Pattern Standalone vLLM deployment works correctly: ```bash # Prefill instance (kv_producer) python -m vllm.entrypoints.openai.api_server \\ --model Qwen/Qwen2-7B-Instruct \\ --port 8100 \\ --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"]}}' # Decode instance (kv_consumer) python -m vllm.entrypoints.openai.api_server \\ --model Qwen/Qwen2-7B-Instruct \\ --port 8200 \\ --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"]}}' ``` Both UCX and Libfabric (EFA) backends work correctly with this approach. ## Documentation Request Please add a note to the NixlConnector documentation (https://docs.vllm.ai/en/stable/features/nixl_connector_usage.html) indicating: 1. Ray Serve LLM's `build_pd_openai_app` is currently incompatible with vLLM v1 engine 2. Standalone vLLM deployment is the recommended approach for PD disaggregation 3. Users need to implement their own routing layer to coordinate prefill→decode flow ## Verified Working Configuration | Component | Version | Status | |-----------|---------|--------| | vLLM standalone | 0.10.0+ | ✅ Working | | NixlConnector + UCX | 0.7.1 | ✅ Working | | NixlConnector + Libfabric | 0.7.1 | ✅ Working | | Ray Serve LLM wrapper | 2.43.0 | ❌ Fails | ## Related Issues - Ray issue filed: ray-project/ray#59064 - #27055 - Libfabric garbage output (fixed in NIXL 0.7.1) ## Additional Context Full test suite and reproduction steps: https://github.com/dmvevents/ray-testing", "url": "https://github.com/vllm-project/vllm/issues/29688", "createdAt": "2025-11-28T16:11:35Z", "updatedAt": "2026-08-13T02:15:50Z", "timestamp": "2026-08-13T02:15:50Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [ "ray", "stale" ], "author": "dmvevents", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:30016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor)", "text": "# TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor) ## Summary When deploying vLLM with `tensor_parallel_size > 1` on Ray Serve, use the **multiprocessing executor** (`distributed_executor_backend=\"mp\"`) instead of the Ray executor. This avoids placement group context issues with vLLM v1's subprocess architecture. ## Problem Description Attempting to use `tensor_parallel_size > 1` with Ray Serve and the Ray executor (`distributed_executor_backend=\"ray\"`) results in worker initialization failures: ```python # This FAILS with TP > 1 on Ray Serve: @serve.deployment(ray_actor_options={\"num_gpus\": 2}) class VLLMDeployment: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, distributed_executor_backend=\"ray\", # ❌ Fails ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) ``` **Error symptoms:** - Workers cannot find or access Ray's placement group - \"Placement group not found\" or similar initialization errors - Timeout waiting for workers to initialize ## Root Cause vLLM v1 architecture: 1. Ray Serve creates an actor with a placement group 2. vLLM spawns `EngineCore` as a **subprocess** 3. The subprocess **loses Ray's placement group context** (stored in thread-local storage) 4. When `EngineCore` tries to spawn Ray worker actors for TP, they cannot access the placement group 5. Worker initialization fails This is an architectural interaction between: - Ray's placement group context (thread-local) - vLLM v1's subprocess-based EngineCore - Ray Serve's placement group creation ## ✅ Solution: Multiprocessing Executor **Use the multiprocessing executor instead:** ```python @serve.deployment( name=\"vllm-tp2\", num_replicas=1, ray_actor_options={\"num_gpus\": 2}, ) @serve.ingress(app) class VLLMDeployment: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, distributed_executor_backend=\"mp\", # ✅ Use multiprocessing! trust_remote_code=True, enforce_eager=True, gpu_memory_utilization=0.4, ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) ``` ### Why This Works 1. **Ray Serve allocates GPUs** - `ray_actor_options={\"num_gpus\": 2}` gives the actor 2 GPUs 2. **vLLM inherits GPU visibility** - Actor has `CUDA_VISIBLE_DEVICES=0,1` 3. **Multiprocessing spawns workers** - vLLM creates worker processes using Python multiprocessing 4. **Workers inherit GPU environment** - Each worker process gets access to the GPUs 5. **Ray assigns different GPUs** - Each worker gets a different GPU via Ray's internal allocation 6. **NCCL enables communication** - Workers coordinate via NCCL for tensor parallelism **No placement groups needed** - Everything stays within the Ray Serve actor's resource allocation. ## Complete Working Example ```python #!/usr/bin/env python3 import ray from ray import serve from vllm.engine.async_llm_engine import AsyncLLMEngine from vllm import AsyncEngineArgs, SamplingParams from fastapi import FastAPI from fastapi.responses import JSONResponse ray.init(address=\"auto\") app = FastAPI() @serve.deployment( name=\"vllm-tp2\", num_replicas=1, ray_actor_options={\"num_gpus\": 2}, max_ongoing_requests=5, ) @serve.ingress(app) class VLLMWithTP2: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, trust_remote_code=True, distributed_executor_backend=\"mp\", # KEY: multiprocessing enforce_eager=True, gpu_memory_utilization=0.4, max_model_len=2048, ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) @app.post(\"/v1/chat/completions\") async def chat(self, request: dict): messages = request.get(\"messages\", []) max_tokens = min(request.get(\"max_tokens\", 50), 100) prompt = \"\\n\".join([ f\"{msg.get('role', 'user')}: {msg.get('content', '')}\" for msg in messages ]) + \"\\nassistant:\" sampling_params = SamplingParams(temperature=0.7, max_tokens=max_tokens) request_id = f\"req_{int(time.time()*1000)}\" results_generator = self.engine.generate(prompt, sampling_params, request_id) final_output = None async for output in results_generator: final_output = output text = final_output.outputs[0].text if final_output else \"\" return JSONResponse({ \"id\": request_id, \"model\": \"Qwen/Qwen2-7B-Instruct\", \"tp_size\": 2, \"executor\": \"multiprocessing\", \"choices\": [{ \"message\": {\"role\": \"assistant\", \"content\": text.strip()}, \"finish_reason\": \"stop\", }], }) @app.get(\"/health\") async def health(self): return {\"status\": \"healthy\", \"tp_size\": 2, \"executor\": \"multiprocessing\"} # Deploy serve.run(VLLMWithTP2.bind(), name=\"vllm-tp2\", route_prefix=\"/\") ``` ## Test Results **Deployment logs:** ``` Worker_TP0 pid=4701 (GPU 0) Worker_TP1 pid=4702 (GPU 1) NCCL version 2.27.3+cuda12.9 comm rank 0 nRanks 2 localRanks 2 Channel 00-23/24 configured Loading checkpoint shards: 100% | 4/4 Application ready at http://127.0.0.1:8000/ ``` **Inference test:** ```bash $ curl -X POST http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{\"messages\":[{\"role\":\"user\",\"content\":\"What is 2+2?\"}]}' { \"id\": \"req_1764813981407\", \"model\": \"Qwen/Qwen2-7B-Instruct\", \"tp_size\": 2, \"executor\": \"multiprocessing\", \"choices\": [{ \"message\": { \"role\": \"assistant\", \"content\": \"2 + 2 equals 4.\" } }] } ``` ✅ **Status:** Working perfectly with TP=2 ## Architecture Comparison ### Ray Executor (Fails with Ray Serve) ``` Ray Serve Actor └─> Placement Group (2 GPUs) └─> EngineCore Subprocess ⚠️ (loses PG context) └─> Ray Worker Actor 0 ❌ (can't find PG) └─> Ray Worker Actor 1 ❌ (can't find PG) ``` ### Multiprocessing Executor (Works) ``` Ray Serve Actor (num_gpus=2) └─> CUDA_VISIBLE_DEVICES=0,1 └─> EngineCore Subprocess (inherits CUDA vars) └─> Worker Process 0 (GPU 0) ✅ └─> Worker Process 1 (GPU 1) ✅ └─> NCCL Communication ✅ ``` ## When to Use Each Executor ### Use Multiprocessing Executor (`\"mp\"`) When: - ✅ Deploying with Ray Serve - ✅ Single-node TP (2-8 GPUs on one node) - ✅ Want simple, reliable deployment - ✅ Don't need multi-node TP ### Use Ray Executor (`\"ray\"`) When: - Multi-node TP is required (TP across nodes) - **NOT deploying with Ray Serve** (use standalone vLLM) - Need Ray's advanced scheduling features ## Limitations - **Single node only** - Multiprocessing executor works within one node - **Multi-node TP** - Would require Ray executor (but not compatible with Ray Serve) - **Recommended TP values** - 2-8 GPUs on single node For multi-node scenarios, use standalone vLLM deployment instead of Ray Serve. ## Environment - **vLLM:** v0.11.0 - **Ray:** 2.40+ - **Ray Serve:** Enabled - **GPUs:** NVIDIA H100 (AWS p5.48xlarge) - **NCCL:** 2.27.3+cuda12.9 - **CUDA:** 12.6 ## Additional Resources - **Full implementation:** https://github.com/dmvevents/ray-testing/tree/feature/tp-greater-than-1-support - **Comprehensive analysis:** https://github.com/dmvevents/ray-testing/blob/feature/tp-greater-than-1-support/TP2_SUCCESS_SUMMARY.md - **Deployment scripts:** https://github.com/dmvevents/ray-testing/blob/feature/tp-greater-than-1-support/deploy_tp2_mp_executor.py ## Related Issues This documents a **working solution** for TP > 1 with Ray Serve. No code changes needed - just use the multiprocessing executor. If multi-node TP with Ray Serve is needed in the future, that would require architectural changes to preserve placement group context across subprocess boundaries. --- **TL;DR:** Use `distributed_executor_backend=\"mp\"` for TP > 1 with Ray Serve. It works perfectly.", "url": "https://github.com/vllm-project/vllm/issues/30016", "createdAt": "2025-12-04T02:36:27Z", "updatedAt": "2026-08-13T02:15:46Z", "timestamp": "2026-08-13T02:15:46Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "ray", "stale" ], "author": "dmvevents", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:36222", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Usage]: MoE flatten_tp_size should not unconditionally include dp_size — DP loses its original semantics for MoE layers", "text": "## Summary In FusedMoEParallelConfig.flatten_tp_across_dp_and_pcp (vllm/model_executor/layers/fused_moe/config.py:979-988), when EP is not enabled, dp_size is unconditionally folded into the flattened TP size: ``` flatten_tp_size = dp_size * pcp_size * tp_size flatten_tp_rank = dp_rank * pcp_size * tp_size + pcp_rank * tp_size + tp_rank ``` This means that for MoE layers, DP no longer behaves as \"data parallelism\" (independent replicas processing different data). Instead, it becomes an additional weight-sharding dimension — effectively turning DP ranks into extra TP ranks for MoE weights only. I believe this conflates two distinct parallelism concepts and leads to practical deployment issues. Concrete failure case When deploying Qwen3.5-35B-A3B-FP8 (which has intermediate_size=512 for MoE experts): TP=8, DP=1: intermediate_size_per_partition = 512 / 8 = 64, which is smaller than the FP8 block quantization size (128). This causes a quantization alignment error. TP=4, DP=2 (hoping to avoid the above issue): The MoE layer computes flatten_tp_size = 2 * 1 * 4 = 8, so intermediate_size_per_partition = 512 / 8 = 64 — the exact same problem. DP=2 does not help at all because DP is silently absorbed into TP for MoE layers. There is currently no way to deploy this model on 8 GPUs with TP=4 and have MoE layers actually use tp_size=4 (giving intermediate_size_per_partition = 512 / 4 = 128), because the flatten logic always forces MoE to shard across all devices. Semantic concern For non-MoE layers (Attention, Dense MLP), DP ranks hold identical weight replicas and process different batches independently. But for MoE layers (when EP is disabled), DP ranks hold different weight slices and must communicate to produce correct results. This means: - \"DP\" has fundamentally different meanings for MoE vs. non-MoE layers within the same model, which is confusing. - Users cannot independently control the MoE sharding strategy — it is implicitly determined by dp_size, which was chosen for throughput/batch scheduling reasons, not for MoE weight distribution. - The tight coupling prevents valid deployment configurations (as shown above). ## Proposal I suggest introducing a separate, explicit parameter to control MoE-layer parallelism independently of DP, rather than unconditionally folding dp_size into flatten_tp_size. ### How would you like to use vllm I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/36222", "createdAt": "2026-03-06T07:02:21Z", "updatedAt": "2026-08-13T02:14:41Z", "timestamp": "2026-08-13T02:14:41Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "usage", "stale" ], "author": "gerayking", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:36315", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers'", "text": "### Your current environment vllm 0.17.0 lastest. ### 🐛 Describe the bug CUDA_VISIBLE_DEVICES=1,2 vllm serve /home/ub3960x/model_data/Qwen3.5-9B \\ --tensor-parallel-size 2 \\ --gpu-memory-utilization 0.80 \\ --port 30005 \\ --served-model-name qwen3.5-9b \\ --max-model-len 16384 \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser hermes \\ --trust-remote-code (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] █ █ █▄ ▄█ (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.17.0 (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] █▄█▀ █ █ █ █ model /home/ub3960x/model_data/Qwen3.5-9B (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:238] non-default args: {'model_tag': '/home/ub3960x/model_data/Qwen3.5-9B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'port': 30005, 'model': '/home/ub3960x/model_data/Qwen3.5-9B', 'trust_remote_code': True, 'max_model_len': 16384, 'served_model_name': ['qwen3.5-9b'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.8} (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) INFO 03-07 14:45:02 [model.py:531] Resolved architecture: Qwen3ForCausalLM (APIServer pid=31946) INFO 03-07 14:45:02 [model.py:1554] Using max model len 16384 (APIServer pid=31946) INFO 03-07 14:45:02 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=31946) INFO 03-07 14:45:02 [vllm.py:747] Asynchronous scheduling is enabled. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (EngineCore_DP0 pid=32058) INFO 03-07 14:45:08 [core.py:101] Initializing a V1 LLM engine (v0.17.0) with config: model='/home/ub3960x/model_data/Qwen3.5-9B', speculative_config=None, tokenizer='/home/ub3960x/model_data/Qwen3.5-9B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.5-9b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore_DP0 pid=32058) WARNING 03-07 14:45:08 [multiproc_executor.py:945] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore_DP0 pid=32058) INFO 03-07 14:45:08 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.1.64 (local), world_size=2, local_world_size=2 (Worker pid=32161) INFO 03-07 14:45:14 [parallel_state.py:1393] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:42561 backend=nccl (Worker pid=32162) INFO 03-07 14:45:14 [parallel_state.py:1393] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:42561 backend=nccl (Worker pid=32161) :1325: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=32161) :1325: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=32161) INFO 03-07 14:45:14 [pynccl.py:111] vLLM is using nccl==2.27.5 (Worker pid=32162) :1325: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=32162) :1325: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=32161) WARNING 03-07 14:45:15 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=32162) WARNING 03-07 14:45:15 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=32161) WARNING 03-07 14:45:15 [custom_all_reduce.py:165] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly. (Worker pid=32162) WARNING 03-07 14:45:15 [custom_all_reduce.py:165] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly. (Worker pid=32162) INFO 03-07 14:45:15 [parallel_state.py:1715] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A (Worker pid=32161) INFO 03-07 14:45:15 [parallel_state.py:1715] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (Worker pid=32161) INFO 03-07 14:45:15 [base.py:106] Offloader set to NoopOffloader (Worker pid=32162) INFO 03-07 14:45:15 [base.py:106] Offloader set to NoopOffloader (Worker pid=32161) (Worker_TP0 pid=32161) INFO 03-07 14:45:15 [gpu_model_runner.py:4255] Starting to load model /home/ub3960x/model_data/Qwen3.5-9B... (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] WorkerProc failed to start. (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] Traceback (most recent call last): (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 771, in worker_main (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] worker = WorkerProc(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 597, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.worker.load_model() (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py\", line 337, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model_runner.load_model(load_dummy_weights=dummy_weights) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4271, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = model_loader.load_model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=self.vllm_config, model_config=self.model_config (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/base_loader.py\", line 54, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = initialize_model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, model_config=model_config, prefix=prefix (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/utils.py\", line 56, in initialize_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = model_class(vllm_config=vllm_config, prefix=prefix) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 287, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = Qwen3Model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=maybe_prefix(prefix, \"model\") (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 256, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] super().__init__( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=prefix, decoder_layer_type=Qwen3DecoderLayer (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen2.py\", line 368, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] assert config.max_window_layers == config.num_hidden_layers, ( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/transformers/configuration_utils.py\", line 207, in __getattribute__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return super().__getattribute__(key) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers' (Worker pid=32162) (Worker_TP1 pid=32162) INFO 03-07 14:45:16 [multiproc_executor.py:749] Parent process exited, terminating worker (Worker pid=32161) (Worker_TP0 pid=32161) INFO 03-07 14:45:16 [multiproc_executor.py:749] Parent process exited, terminating worker (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] WorkerProc failed to start. (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] Traceback (most recent call last): (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 771, in worker_main (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] worker = WorkerProc(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 597, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.worker.load_model() (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py\", line 337, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model_runner.load_model(load_dummy_weights=dummy_weights) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4271, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = model_loader.load_model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=self.vllm_config, model_config=self.model_config (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/base_loader.py\", line 54, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = initialize_model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, model_config=model_config, prefix=prefix (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/utils.py\", line 56, in initialize_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = model_class(vllm_config=vllm_config, prefix=prefix) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 287, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = Qwen3Model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=maybe_prefix(prefix, \"model\") (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 256, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] super().__init__( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=prefix, decoder_layer_type=Qwen3DecoderLayer (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen2.py\", line 368, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] assert config.max_window_layers == config.num_hidden_layers, ( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/transformers/configuration_utils.py\", line 207, in __getattribute__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return super().__getattribute__(key) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers' [rank0]:[W307 14:45:17.261277764 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] EngineCore failed to start. (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] Traceback (most recent call last): (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1090, in run_engine_core (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] return func(*args, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 834, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] super().__init__( (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] vllm_config, (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ...<3 lines>... (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] internal_dp_balancing, (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 110, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self.model_executor = executor_class(vllm_config) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 100, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] super().__init__(vllm_config) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] return func(*args, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/abstract.py\", line 103, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self._init_executor() (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~~~~^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 180, in _init_executor (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self.workers = WorkerProc.wait_for_ready(unready_workers) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 697, in wait_for_ready (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] raise e from None (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. (EngineCore_DP0 pid=32058) Process EngineCore_DP0: (EngineCore_DP0 pid=32058) Traceback (most recent call last): (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/multiprocessing/process.py\", line 313, in _bootstrap (EngineCore_DP0 pid=32058) self.run() (EngineCore_DP0 pid=32058) ~~~~~~~~^^ (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/multiprocessing/process.py\", line 108, in run (EngineCore_DP0 pid=32058) self._target(*self._args, **self._kwargs) (EngineCore_DP0 pid=32058) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1104, in run_engine_core (EngineCore_DP0 pid=32058) raise e (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1090, in run_engine_core (EngineCore_DP0 pid=32058) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/py ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/36315", "createdAt": "2026-03-07T07:37:14Z", "updatedAt": "2026-08-13T02:14:38Z", "timestamp": "2026-08-13T02:14:38Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "skfeng36", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:37035", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: cudaErrorIllegalAddress in gdn_attn.py:237 when using qwen3_next_mtp with num_speculative_tokens=5 under load", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu129 Is debug build : False CUDA used to build PyTorch : 12.9 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.9.86 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 Nvidia driver version : 595.79 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: AuthenticAMD Model name: AMD Ryzen Threadripper 3990X 64-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 1 Stepping: 0 BogoMIPS: 5789.20 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip rdpid Virtualization: AMD-V Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 2 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 32 MiB (64 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-127 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.6 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.9.1.4 [pip3] nvidia-cuda-cupti-cu12==12.9.79 [pip3] nvidia-cuda-nvrtc-cu12==12.9.86 [pip3] nvidia-cuda-runtime-cu12==12.9.79 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft-cu12==11.4.1.4 [pip3] nvidia-cufile-cu12==1.14.1.1 [pip3] nvidia-curand-cu12==10.3.10.19 [pip3] nvidia-cusolver-cu12==11.7.5.82 [pip3] nvidia-cusparse-cu12==12.5.10.65 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.4.1 [pip3] nvidia-cutlass-dsl-libs-base==4.4.1 [pip3] nvidia-ml-py==13.590.48 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.9.86 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.9.79 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+cu129 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.10.0+cu129 [pip3] torchvision==0.25.0+cu129 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.17.1rc1.dev126+gbc2c0c86e (git sha: bc2c0c86e) vLLM Build Flags: CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_CUDAGRAPH_GC=1 VLLM_USAGE_SOURCE=production-docker-image TORCHINDUCTOR_CACHE_DIR=/root/.cache/torch/inductor VLLM_USE_FLASHINFER_SAMPLER=1 CUDA_VERSION=12.9.1 VLLM_ENABLE_CUDA_COMPATIBILITY=0 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
- **vLLM version**: `v0.17.1rc1.dev126+gbc2c0c86e` (nightly, `vllm/vllm-openai:nightly`) - **Model**: `cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4` - **Quantization**: `compressed-tensors` (AWQ BF16 INT4) - **Hardware**: 2× NVIDIA GPU (tensor-parallel-size=2), WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2) - **Attention backend**: FlashInfer (`--attention-backend FLASHINFER`) - **Python**: 3.12 - **PyTorch**: inferred from nightly image ### 🐛 Describe the bug ## Summary vLLM crashes with `CUDA error: an illegal memory access was encountered` in `gdn_attn.py:237` (`spec_state_indices_tensor = block_table_tensor[...]`) when running `qwen3_next_mtp` speculative decoding with `num_speculative_tokens=5` and FlashInfer attention backend under concurrent load. Both TP workers (TP0 and TP1) crash with the same error. The crash occurs consistently once the request queue fills (~500 waiting requests) and KV cache usage exceeds ~63%. The crash reproduces reliably across multiple runs (observed twice within a single session, ~7 minutes apart). ## Suspected Root Cause FlashInfer PR [#2679](https://github.com/flashinfer-ai/flashinfer/pull/2679) (\"feat(gdn): add BF16 state kernel with MTP support beyond T>4\") is open and unmerged as of this report. It specifically adds GDN MTP kernel support for **T>4 speculative tokens**. With `num_speculative_tokens=5` (T=5), the existing GDN kernel likely has an OOB access when building `spec_state_indices_tensor` from the block table, which manifests as `cudaErrorIllegalAddress`. Related open issue: [vllm#36613](https://github.com/vllm-project/vllm/issues/36613) (same error class, different model/config). ## Reproduction ### Docker Compose (vLLM service definition) ```yaml vllm: image: vllm/vllm-openai:nightly container_name: vllm ipc: host ports: - \"8000:8000\" volumes: - vllm-models:/root/.cache/huggingface - vllm-compile:/root/.cache/vllm - vllm-flashinfer:/root/.cache/flashinfer - vllm-torch:/root/.cache/torch - vllm-triton:/root/.triton environment: - VLLM_WORKER_MULTIPROC_METHOD=spawn - VLLM_SLEEP_WHEN_IDLE=1 - VLLM_ENABLE_CUDAGRAPH_GC=1 - VLLM_USE_FLASHINFER_SAMPLER=1 - RAY_memory_monitor_refresh_ms=0 - TORCHINDUCTOR_CACHE_DIR=/root/.cache/torch/inductor deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] command: > -O3 --model cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4 --served-model-name local --quantization compressed-tensors --max-model-len=100000 --max-num-seqs=8 --block-size=32 --max-num-batched-tokens=2048 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --attention-backend FLASHINFER --tensor-parallel-size=2 --gpu-memory-utilization=0.90 --trust-remote-code --disable-custom-all-reduce --speculative-config '{\"method\":\"qwen3_next_mtp\",\"num_speculative_tokens\":5}' ``` ### Benchmark command (triggers the crash under load) ```bash docker run --rm -it \\ --ipc=host --network=host \\ -v \"./guidellm:/results:rw\" \\ ghcr.io/vllm-project/guidellm:latest -- benchmark \\ --target \"http://127.0.0.1:8000\" \\ --profile sweep \\ --max-seconds 30 \\ --warmup 0.1 \\ --cooldown 0.1 \\ --max-errors 5 \\ --detect-saturation \\ --data \"abisee/cnn_dailymail\" \\ --data-args '{\"name\": \"3.0.0\"}' \\ --data-column-mapper '{\"text_column\":\"article\"}' ``` The crash occurs once the queue saturates (~504 waiting requests, KV cache ~63–66%). ## Stack Trace (Crash 1 — Worker_TP0, full Python traceback available) ``` (Worker_TP0 pid=409) ERROR 03-14 03:28:43 [multiproc_executor.py:932] Traceback (most recent call last): File \".../vllm/v1/worker/gpu_model_runner.py\", line 3254, in synchronize_input_prep yield File \".../vllm/v1/worker/gpu_model_runner.py\", line 3723, in execute_model self._build_attention_metadata( File \".../vllm/v1/worker/gpu_model_runner.py\", line 2102, in _build_attention_metadata _build_attn_group_metadata(kv_cache_gid, attn_gid, cm) File \".../vllm/v1/worker/gpu_model_runner.py\", line 2053, in _build_attn_group_metadata attn_metadata_i = builder.build( File \".../vllm/v1/attention/backends/gdn_attn.py\", line 237, in build spec_state_indices_tensor = block_table_tensor[ ^^^^^^^^^^^^^^^^^^^ torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` Both TP0 and TP1 workers raise the same error at `gdn_attn.py:237`. The secondary traceback (after exception propagation) surfaces at: ``` File \".../vllm/v1/worker/gpu_model_runner.py\", line 3256, in synchronize_input_prep self.prepare_inputs_event.record() torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` Followed by: ``` (EngineCore pid=210) ERROR [multiproc_executor.py:273] Worker proc VllmWorker-0 died unexpectedly, shutting down executor. RuntimeError: cancelled vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` ## Crash 2 (7 minutes later, same session) Identical error, but this time `VllmWorker-1` dies first and the Python traceback is replaced by a C++ `terminate called after throwing an instance of 'c10::AcceleratorError'`. Conditions: 8 running, 504 waiting, KV cache 65.4%. ## Scheduler State at Crash Both crashes share the same scheduler snapshot: - `num_running_reqs=8`, `num_waiting_reqs=504` - `kv_cache_usage≈0.63–0.65` - All 8 requests had `scheduled_spec_decode_tokens: [-1, -1, -1, -1, -1]` (5 spec tokens, all rejected/pending) - `step_counter=0` — crash on the very first step after restart ## Confirmed Bisection | `num_speculative_tokens` | Result | |---|---| | 5 | **Crashes** — `cudaErrorIllegalAddress` in `gdn_attn.py:237` under load | | 4 | **Stable** — no crash observed under identical load | This confirms the T>4 boundary as the root cause. The GDN FlashInfer kernel does not correctly support 5+ speculative tokens, consistent with flashinfer PR [#2679](https://github.com/flashinfer-ai/flashinfer/pull/2679) which adds that support but is not yet merged. ## Workaround Set `num_speculative_tokens` to 4 or fewer. Alternatively, disable MTP entirely by removing `--speculative-config`. ## Attached Logs - [bug_report_crash1.log](https://pastebin.com/gq5kkWiD) — full Worker_TP0/TP1 output for crash 1 (Python traceback, C++ frames) - [bug_report_crash2.log](https://pastebin.com/iA5DVgAG) — crash 2 output (C++ terminate, no Python traceback) ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37035", "createdAt": "2026-03-14T04:10:52Z", "updatedAt": "2026-08-13T02:14:24Z", "timestamp": "2026-08-13T02:14:24Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "unstale" ], "author": "Quentin-M", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:37271", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: In_proj_ba of GDN in Qwen3Next use MergeColumnParallelLinear may cause accuracy decrease?", "text": "### Your current environment
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug I found that in_proj_ba is a MergeColumnParallelLinear now, and it was a ColumnParallelLinear before. It leads to a different way of weight loading. And it will cause accuracy decrease on gsm8k dataset. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37271", "createdAt": "2026-03-17T08:39:37Z", "updatedAt": "2026-08-13T02:14:12Z", "timestamp": "2026-08-13T02:14:12Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "stale" ], "author": "SunnyLee151064", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:37304", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Language Models Test (Extended Generation) test_models[False-False-5-32-bigcode/starcoder2-3b] test issue", "text": "### Your current environment main ### 🐛 Describe the bug I got the test to fail under the following two situations: 1. We're trying to upgrade PyTorch from 2.10 to 2.11. The test succeeds in PyTorch 2.10, but fails in PyTorch 2.11 ([logs](https://buildkite.com/vllm/ci/builds/56028#019ce7ae-eb9b-41be-90ae-310763ec6438)) 2. In PyTorch 2.10, if I add [--enforce-eager](https://github.com/vllm-project/vllm/blob/3717a4dd475e6a936df0c84b043743310368e766/tests/models/language/generation/test_common.py#L178), the test fails. The error message looks the same in both cases. ([logs](https://buildkite.com/vllm/ci/builds/56028#019ce7ae-eb9b-41be-90ae-310763ec6438)) ``` [2026-03-13T16:18:07Z] E AssertionError: Test1: -- [2026-03-13T16:18:07Z] E Matched tokens: [222, 40, 494, 447, 9009, 98, 828, 366] [2026-03-13T16:18:07Z] E hf: '\\n# + [markdown] id=\"5f7o229v00-L\"\\n# **1950** - ' {59: -4.088154315948486, 58: -4.088154315948486, 56: -4.213154315948486, 55: -4.213154315948486, 54: -4.213154315948486} [2026-03-13T16:18:07Z] E vllm: '\\n# + [markdown] id=\"68284477\"\\n# **1950**\\n#\\n# - ' {59: Logprob(logprob=-4.071040153503418, rank=1, decoded_token='6'), 56: Logprob(logprob=-4.196040153503418, rank=2, decoded_token='3'), 54: Logprob(logprob=-4.196040153503418, rank=3, decoded_token='1'), 57: Logprob(logprob=-4.196040153503418, rank=4, decoded_token='4'), 55: Logprob(logprob=-4.196040153503418, rank=5, decoded_token='2')} ``` Here's a [link to the test](https://github.com/vllm-project/vllm/blob/3717a4dd475e6a936df0c84b043743310368e766/tests/models/language/generation/test_common.py#L198-L203) I'm trying to figure out if we should block the PyTorch 2.11 release on this test. Given datapoint (2): the test fails when adding --enforce-eager, I think there is something wrong with the test and that we shouldn't block on this. I don't have very much context over this test though. @hmellor I see you poked around at this file recently for Gemma and Gemma2. Do you have any thoughts here? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37304", "createdAt": "2026-03-17T14:47:03Z", "updatedAt": "2026-08-13T08:22:28Z", "timestamp": "2026-08-13T08:22:28Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "bug", "stale" ], "author": "zou3519", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:37736", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[CI Failure]: Gemma3 OOMs with transformers backend", "text": "### Test group mi250_1: Multi-Modal Models (Standard) 2: qwen3 + gemma ### Describe the failing test This is not exactly a test failure, but it has been recommended to investigate further the OOMing event of Gemma3, which is a 4B model. The intuition here is that the fake tensor that is used for profiling is large enough that exceeds the 64 GB of MI250 GPUs. However, it has been suggested that this is still weird. ### 📝 History of failing test https://github.com/vllm-project/vllm/pull/37610#issuecomment-4102286515 ### CC List. @DarkLight1337 (for transparency)", "url": "https://github.com/vllm-project/vllm/issues/37736", "createdAt": "2026-03-21T05:32:29Z", "updatedAt": "2026-08-13T17:16:19Z", "timestamp": "2026-08-13T17:16:19Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [ "rocm", "unstale", "ci-failure" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:38175", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Support ViT Full CUDA Graph (Tracker)", "text": "### Motivation. Multimodal large language models (e.g., Qwen3-VL, Qwen3.5, GLM-V, Kimi K2.5) rely on a Vision Transformer (ViT) encoder to process visual inputs before feeding them into the language model backbone. In production serving scenarios, the ViT forward pass involves launching a large number of small CUDA kernels — including patch embedding, layer normalization, multi-head self-attention, and MLP projections — each of which incurs non-trivial kernel launch overhead on the host side. Currently, vLLM supports CUDA graph capture for the decoder (LLM) portion of the model, which has proven effective at reducing kernel launch costs and improving throughput. However, the ViT encoder is still executed eagerly, meaning every forward pass re-launches all kernels from scratch. Extending full CUDA graph support to the ViT encoder would allow the entire encoder forward pass to be captured and replayed as a single graph, eliminating per-kernel launch overhead and enabling more consistent, low-latency inference for multimodal models. ### Proposed Change. **Model Integration:** - [x] https://github.com/vllm-project/vllm/pull/35963 @b-mu - [x] https://github.com/vllm-project/vllm/pull/41736 @johncalesp - [x] https://github.com/vllm-project/vllm/pull/40830 @johncalesp - [x] https://github.com/vllm-project/vllm/pull/38061 @shen-shanshan - [x] https://github.com/vllm-project/vllm/pull/42151 @shen-shanshan - [x] https://github.com/vllm-project/vllm/pull/44720 @harsha20032020 - [x] https://github.com/vllm-project/vllm/pull/42224 @JisoLya - [x] https://github.com/vllm-project/vllm/pull/40576 @grYe99 - [x] https://github.com/vllm-project/vllm/pull/40660 @allgather - [x] https://github.com/vllm-project/vllm/pull/41759 @oguzhankir - [x] https://github.com/vllm-project/vllm/pull/41992 @oguzhankir - [x] https://github.com/vllm-project/vllm/pull/50929 @lk-chen - [ ] https://github.com/vllm-project/vllm/pull/42785 @YunzhuLu - [x] https://github.com/vllm-project/vllm/pull/43591 @JisoLya - [x] https://github.com/vllm-project/vllm/pull/46837 @anthonsu - [x] https://github.com/vllm-project/vllm/pull/43586 @shen-shanshan - [ ] https://github.com/vllm-project/vllm/pull/46005 @littlecircle0730 - [ ] https://github.com/vllm-project/vllm/pull/44394 @harsha20032020 - [x] https://github.com/vllm-project/vllm/pull/45254 @qyYue1389 - [ ] https://github.com/vllm-project/vllm/pull/51484 @gangula-karthik > [!NOTE] > **Integration Workflow:** > > 1. Implement ViT CUDA graph interface for the model referring to Qwen3-VL. > 2. Do tests: ut/e2e/benchmark/... > 3. Update supported model list in the [doc](https://docs.vllm.ai/en/latest/design/cuda_graphs_multimodal/#model-integration-via-supportsencodercudagraph). > 4. Add this model to CI test. **Bugfix / Improvement:** - [x] https://github.com/vllm-project/vllm/pull/38040 - [x] https://github.com/vllm-project/vllm/pull/38116 - [x] https://github.com/vllm-project/vllm/pull/40445 - [x] https://github.com/vllm-project/vllm/pull/40580 - [x] https://github.com/vllm-project/vllm/pull/41234 - [x] https://github.com/vllm-project/vllm/pull/41714 - [x] https://github.com/vllm-project/vllm/pull/42288 - [x] https://github.com/vllm-project/vllm/pull/42796 - [x] https://github.com/vllm-project/vllm/pull/43082 - [x] https://github.com/vllm-project/vllm/pull/43321 - [ ] https://github.com/vllm-project/vllm/pull/43403 - [x] https://github.com/vllm-project/vllm/pull/44484 - [x] https://github.com/vllm-project/vllm/pull/45654 - [ ] https://github.com/vllm-project/vllm/pull/45886 - [x] https://github.com/vllm-project/vllm/pull/46034 - [x] https://github.com/vllm-project/vllm/pull/49934 - [x] https://github.com/vllm-project/vllm/pull/51461 **Testing Coverage:** - [x] https://github.com/vllm-project/vllm/pull/40780 **Documentation:** - [x] https://github.com/vllm-project/vllm/pull/37914 - [x] https://github.com/vllm-project/vllm/pull/40355 - [x] https://github.com/vllm-project/vllm/pull/44388 ### Feedback Period. _No response_ ### CC List. @ywang96 @Isotr0py @wangshangsam ### Any Other Things. _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38175", "createdAt": "2026-03-26T02:22:01Z", "updatedAt": "2026-08-13T07:36:09Z", "timestamp": "2026-08-13T07:36:09Z", "metrics": { "reactions": 1, "comments": 27 }, "labels": [ "help wanted", "RFC", "multi-modality", "kimi" ], "author": "shen-shanshan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:38988", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Qwen 3.5 27B Prefix Caching", "text": "### Proposal to improve performance _No response_ ### Report of performance regression There have been discussions if Qwen 3.5 (27B) supports prefix caching in vLLM. - https://www.reddit.com/r/LocalLLaMA/comments/1rplb3r/vllm_prefix_caching_cannot_be_used_with_qwen_35/ - #36010 I would appreciate information on this issue. ### Misc discussion on performance _No response_ ### Your current environment (if you think it is necessary) ```text The output of `python collect_env.py` ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38988", "createdAt": "2026-04-04T17:00:25Z", "updatedAt": "2026-08-13T02:13:59Z", "timestamp": "2026-08-13T02:13:59Z", "metrics": { "reactions": 4, "comments": 7 }, "labels": [ "performance", "stale" ], "author": "NilsHellwig", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:38994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "Qwen-3.5 9B often producing repetitive/garbled output with Intel Backend", "text": "### Your current environment This was the output from running python collect_env.py (running in docker image) ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : version 4.3.1 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+xpu Is debug build : False CUDA used to build PyTorch : None ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.17.0-20-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : False CUDA runtime version : No CUDA CUDA_MODULE_LOADING set to : N/A GPU models and configuration : No CUDA Nvidia driver version : No CUDA cuDNN version : No CUDA HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. Model name: AMD Ryzen Threadripper PRO 3945WX 12-Cores BIOS Model name: AMD Ryzen Threadripper PRO 3945WX 12-Cores Unknown CPU @ 4.0GHz BIOS CPU family: 107 CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU(s) scaling MHz: 66% CPU max MHz: 4427.7739 CPU min MHz: 560.1400 BogoMIPS: 8000.53 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es Virtualization: AMD-V L1d cache: 384 KiB (12 instances) L1i cache: 384 KiB (12 instances) L2 cache: 6 MiB (12 instances) L3 cache: 64 MiB (4 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-23 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flake8==7.3.0 [pip3] mypy==1.20.0 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] pyzmq==27.1.0 [pip3] sentence-transformers==5.3.0 [pip3] torch==2.10.0+xpu [pip3] torchaudio==2.11.0+xpu [pip3] torchvision==0.25.0+xpu [pip3] transformers==4.57.6 [pip3] triton-xpu==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.19.1rc1.dev1+gfa9e68022 (git sha: fa9e68022) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_TARGET_DEVICE=xpu LD_LIBRARY_PATH=/opt/intel/oneapi/ccl/2021.15/lib/:/opt/intel/oneapi/tcm/1.4/lib:/opt/intel/oneapi/umf/1.0/lib:/opt/intel/oneapi/tbb/2022.3/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/0.16/lib:/opt/intel/oneapi/mpi/2021.17/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.17/lib:/opt/intel/oneapi/mkl/2025.3/lib:/opt/intel/oneapi/dnnl/2025.3/lib:/opt/intel/oneapi/debugger/2025.3/opt/debugger/lib:/opt/intel/oneapi/compiler/2025.3/opt/compiler/lib:/opt/intel/oneapi/compiler/2025.3/lib:/opt/intel/oneapi/ccl/2021.15/lib/:/tmp/ucx_install/lib:/opt/intel/oneapi/tcm/1.4/lib:/opt/intel/oneapi/umf/1.0/lib:/opt/intel/oneapi/tbb/2022.3/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/0.16/lib:/opt/intel/oneapi/mpi/2021.17/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.17/lib:/opt/intel/oneapi/mkl/2025.3/lib:/opt/intel/oneapi/dnnl/2025.3/lib:/opt/intel/oneapi/debugger/2025.3/opt/debugger/lib:/opt/intel/oneapi/compiler/2025.3/opt/compiler/lib:/opt/intel/oneapi/compiler/2025.3/lib:/opt/intel/oneapi/ccl/2021.17/lib/:/usr/local/lib/ PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root Additionally I am running on 2xB70 Battlemage GPU's. ### 🐛 Describe the bug TLDR: I am running Qwen-3.5 9B on Intel Backend - and its often producing long thinking loops that sometimes degenerate into garbled output. Sometimes it works with a very high number of output tokens but this seems a bit arbitrary. I triple checked my input parameters - but if you see any errors in my setup you are welcome to mention it. I am starting the image and server with these settings: docker run -d --restart=always \\ --name vllm \\ --net=bridge \\ -p 8000:8000 \\ --group-add=video \\ --ipc=host \\ --privileged \\ --device /dev/dri:/dev/dri \\ -v /dev/dri/by-path:/dev/dri/by-path \\ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \\ -v /home/ejer/llm/local_models/Qwen-3.5-9B-original:/app/model:ro \\ --entrypoint /bin/bash \\ vllm-intel-03-04-2026 \\ -lc 'source /opt/intel/oneapi/setvars.sh --force && \\ vllm serve \\ --model /app/model \\ --host 0.0.0.0 --port 8000 \\ --served-model-name Qwen-3.5-9B \\ --no-enable-prefix-caching \\ --enable-chunked-prefill \\ --tensor-parallel-size 2 \\ --reasoning-parser qwen3 \\ --max-model-len 16384 \\ --enforce-eager \\ --disable-custom-all-reduce \\ --gpu-memory-utilization 0.9 \\ --max-num-seqs 128 \\ --attention-backend TRITON_ATTN \\ --dtype bfloat16 \\ --trust-remote-code' In the first example it reaches an answer but after a very long thinking loop (as if the presence penalty and repetition penalty are not taking effect). curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 5000, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.0, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Topic: Denmark.\\n * Format: One sentence.\\n * Content: Fun fact.\\n * Constraint: \\\"Give me a one-sentence fun fact about Denmark.\\\"\\n\\n2. **Brainstorming Facts about Denmark:**\\n * It has the lowest suicide rate in Europe (actually not true anymore, need to check).\\n * It's famous for hygge (coziness).\\n * It was a country of refugees during WWII (Auschwitz? No, Danish saved Jews by lying).\\n * The flag is officially the oldest national flag still in use (Dannebrog).\\n * They invented the first vending machine for newspapers.\\n * It has no nuclear weapons.\\n * It has more than 50 bicycles per 100 people (maybe too specific/changed).\\n * Famous for Lego.\\n * King Vagn? No, King Christian X had a toothache?\\n * Fact: The longest bridge is actually just a connection between Copenhagen and Zealand, not crossing water?\\n * Fact: Danish people don't have last names traditionally in old times?\\n * Fact: The Danish word \\\"hygge\\\" describes comfortable moments.\\n * Fact: Denmark was once home to the first public toilet.\\n * Fact: Their parliament is called Folketinget.\\n * Fact: It's a unitary monarchy.\\n * Fact: They drink tea in glass cups instead of mugs.\\n * Fact: It rains less often in Denmark? No.\\n * Fact: The Royal Guard wears red coats because that's what they look like in cartoons? No.\\n * Fact: Denmark is a landlocked? No, coastal.\\n * Fact: It has many islands (Danish islands).\\n * Fact: Copenhagen airport is very efficient?\\n * Fact: The Danes are known for their long nap time?\\n * Wait, let's pick a really solid one.\\n * Idea 1: \\\"The word 'hygge', which means cozy, was added to the Oxford English Dictionary in 2016.\\\" -> Good, trendy.\\n * Idea 2: \\\"Denmark was once ruled by the Vikings.\\\" -> Too obvious.\\n * Idea 3: \\\"The Danish flag is the oldest national flag still in existence.\\\" -> This is controversial. Is it the *oldest* flag or just one of them? The Jastarn flag is older. But it is widely cited as such.\\n * Idea 4: \\\"Denmark has the highest number of windmills per capita.\\\" -> True-ish.\\n * Idea 5: \\\"They have mandatory cycling?\\\" -> Mandatory bike lane?\\n * Idea 6: \\\"It rained less in Denmark recently\\\".\\n * Let's go with Hygge. \\\"The Danish concept of 'hygge' was so popularized that the word itself was added to the Oxford English Dictionary in 2016.\\\" -> This is a good fun fact.\\n * How about: \\\"Denmark has the world's largest percentage of households with bicycles.\\\" -> Maybe not accurate enough.\\n * Better fact: \\\"Denmark has more cows than people.\\\" -> That's Iceland or Sweden. Denmark has more sheep? No.\\n * Better fact: \\\"It has the most LEGO.\\\"\\n * How about this: \\\"In Denmark, Christmas trees must be decorated before sunrise on December 24th.\\\"\\n * Let's check \\\"Denmark is known for being a safe country.\\\"\\n * Actually, how about: \\\"Denmark holds the Guinness World Record for the smallest nation without any coastline.\\\" -> No, that's Monaco.\\n * Okay, let's focus on something culturally iconic.\\n * Fact: The word \\\"Hygge\\\" is in the dictionary.\\n * Fact: \\\"The Danish word 'smørrebrød' translates to buttered bread.\\\" -> Smørrebrød = open-faced sandwich.\\n * Fact: \\\"Denmark has more bridges than roads.\\\" -> Maybe not.\\n * Fact: \\\"It is impossible to walk from one end of Denmark to another.\\\" -> No.\\n * What about: \\\"Copenhagen has the longest nameplate in the world.\\\"\\n * Let's search for a verified fun fact.\\n * \\\"In Denmark, you can order coffee 'with creamer' only if you ask specifically.\\\"\\n * \\\"There are more islands than mainland areas.\\\"\\n * \\\"The capital city of Copenhagen is not an island.\\\" -> Yes, it's on Zeland.\\n * Fact: \\\"The Danish language has two different past tense verbs.\\\"\\n * Let's stick to the \\\"Hygge\\\" one.\\n * Actually, a very common one is: \\\"Denmark is a landless nation that owns no land.\\\" -> No.\\n * \\\"Every year, millions of people come to see the Great Dane.\\\" -> No.\\n * \\\"It has a flag that flies at sea.\\\"\\n * Let's try: \\\"Denmark has the official color code of its flag listed in the United Nations registry of countries.\\\" -> No.\\n * How about: \\\"Denmark has the smallest population density of all European nations.\\\" -> No.\\n * Fact: \\\"There are more than 50 islands in the Danish archipelago.\\\"\\n * Fact: \\\"Denmark is a flat country.\\\"\\n * Okay, I recall something about the Danish royal family having a cat named...\\n * \\\"The King of Denmark has never eaten sushi.\\\" -> No.\\n * \\\"The Queen of Denmark...\\\"\\n * Wait, I found this fact online earlier: \\\"Denmark was once called the 'land of happiness'.\\\" -> Not really.\\n * \\\"Denmark was part of Sweden for a while.\\\"\\n * Okay, here is a solid one: \\\"The word 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Or: \\\"Denmark has more cows than people.\\\" (No)\\n * \\\"The longest tunnel is in Odense.\\\"\\n * How about: \\\"Denmark has over 400 islands.\\\" -> There are 443 islands off the coast.\\n * Let's refine: \\\"Denmark consists of over 400 distinct islands within its territory.\\\" -> Yes, 429 islands total including mainland? No.\\n * \\\"It is the safest country in the world according to some ranking.\\\"\\n * \\\"The word 'hygge' is used to describe a feeling of comfort.\\\"\\n * Okay, I'll stick with that.\\n * Another option: \\\"In Denmark, people celebrate Valentine's Day on February 14th but also St. Valentine's day.\\\"\\n * Let's choose the \\\"Oldest Flag\\\" fact. It's widely claimed. \\\"The Danish flag is considered the world's oldest continuously used national flag.\\\" -> This is debated (Scandinavian flags existed earlier), but usually accepted as a trivia answer. But wait, \\\"the word hygge was adopted into the English language\\\" is stronger.\\n * Let's check \\\"The longest nameplate in the world\\\" - No.\\n * \\\"In Denmark, the word 'hush' is forbidden.\\\" -> No.\\n * \\\"Denmark has no king.\\\" -> It does.\\n * \\\"Queen Margrethe II is a feminist.\\\"\\n * Okay, let's craft a sentence about \\\"Hygge\\\".\\n * \\\"The concept of 'hygge' – meaning coziness and community comfort – was added to the Oxford English Dictionary in 2016, highlighting its cultural significance.\\\" -> A bit dry.\\n * How about: \\\"Denmark is famous for the term 'hygge,' which literally means cozy.\\\" -> Simple.\\n * Wait, I recall something about a Danish law requiring bicycles to stop when stopped.\\n * \\\"Denmark has the world's longest flagpole.\\\" -> No.\\n * \\\"The Danish government spends millions on a bicycle highway.\\\"\\n * Okay, let's go with: \\\"The Danish word for happiness, 'lykke,' is often replaced by 'hygge.'\\\" -> No.\\n * Let's make it punchy. \\\"Denmark is best known for the concept of 'hygge,' which refers to creating cozy, comfortable environments and experiences.\\\"\\n * Better: \\\"The Danish concept of 'hygge'—meaning coziness—was officially added to the Oxford English Dictionary in 2016.\\\"\\n * Even better: \\\"The Danish culture celebrates 'hygge' (coziness) as a vital part of daily life, and the word even made its way into the Oxford English Dictionary in 2016.\\\"\\n\\n * Check accuracy: Did \\\"hygge\\\" get added? Yes.\\n * Okay, let's draft: \\\"The Danish concept of 'hygge,' which describes cozy social interactions and well-being, was recently added to the Oxford English Dictionary.\\\" -> Accurate.\\n * How about: \\\"Denmark was once home to a Viking king who had a beard painted yellow.\\\"\\n * How about: \\\"Denmark uses the Euro currency.\\\" -> No, DKK.\\n * \\\"Denmark has the lowest crime rate in Europe.\\\"\\n * \\\"The Danes eat herring every Thursday.\\\" -> No.\\n * \\\"Denmark has more bicycles than cars.\\\" -> False.\\n * \\\"Denmark's population is mostly related to the Norsemen.\\\" -> True.\\n * \\\"Denmark has no mountains taller than 10 meters.\\\" -> True.\\n * \\\"All buildings in Denmark must face north-east.\\\" -> No.\\n * \\\"The Danish language is difficult.\\\"\\n * Okay, I think the \\\"Oxford English Dictionary 2016\\\" fact is the most robust \\\"fun fact\\\" style fact.\\n\\n * Alternative: \\\"Denmark has the world's largest collection of fairytales.\\\" -> No.\\n * \\\"It's famous for Tivoli Gardens opening night in 1843.\\\"\\n * Let's combine: \\\"Denmark is famous for adding the Danish word 'hygge' meaning cozy to the Oxford English Dictionary.\\\"\\n * Refinement: \\\"The concept of 'hygge' meaning coziness was added to the Oxford English Dictionary in 2016.\\\"\\n * How about: \\\"Denmark is known for being the country where the concept of 'hygge' was officially added to the Oxford English Dictionary.\\\"\\n * Or maybe: \\\"Denmark has the world's largest number of islands.\\\" -> No, Greece is larger.\\n * \\\"Denmark is flat.\\\"\\n * \\\"It's impossible to see the sun set on both sides of the equator there.\\\" -> No.\\n * \\\"The Queen of Denmark is known as the Queen Mother.\\\" -> No.\\n * \\\"Denmark has 16,000 lakes.\\\"\\n * Okay, I'll write about the dictionary entry.\\n * Draft: \\\"The Danish concept of 'hygge,' which represents coziness and contentment, was added to the Oxford English Dictionary in 2016.\\\"\\n * Can I make it shorter? \\\"The Danish concept of coziness, 'hygge,' entered the Oxford English Dictionary in 2016.\\\"\\n\\n * Another idea: \\\"Denmark has more bridges than tunnels.\\\" -> False.\\n * \\\"Denmark has 70% of its electricity from wind power.\\\" -> Maybe false.\\n * \\\"Denmark exports more energy.\\\"\\n * Okay, sticking to OED.\\n * Wait, another fun fact: \\\"Denmark has the shortest distance to the moon.\\\" -> No.\\n * \\\"It has the fastest train.\\\" -> No.\\n * \\\"It has a lot of castles.\\\"\\n * \\\"The Danish monarchy has been around since 1300s.\\\" -> No.\\n * \\\"The word 'hygge' was born in Denmark.\\\" -> No.\\n * Okay, I think the OED one is fine.\\n\\n * Let's double check facts.\\n * Wikipedia says: \\\"In 2016, the word 'hygge' was added to the Oxford English Dictionary.\\\"\\n * Okay, let's write it down.\\n\\n * Sentence: \\\"The Danish concept of coziness, known as 'hygge,' was added to the Oxford English Dictionary in 2016.\\\"\\n * Better flow: \\\"The Danish concept of 'hygge,' meaning coziness and contentment, was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n3. **Final Polish:** Ensure it meets \\\"one-sentence\\\" constraint strictly.\\n\\n * Proposed sentence: \\\"The Danish concept of 'hygge,' which refers to coziness and comfort, was added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Check alternative interesting fact: \\\"Denmark has the highest average income in Europe.\\\" -> Debatable.\\n * \\\"Denmark is famous for its Lego sets.\\\"\\n * \\\"The Danish word for 'love' is 'forelsket'.\\\"\\n * \\\"Denmark has the smallest island in the world?\\\" -> No.\\n * \\\"It is a small country.\\\"\\n * Okay, the \\\"Oxford English Dictionary\\\" one seems perfect.\\n\\n * Wait, is it the Danish concept or just \\\"hygge\\\"? The word itself might not be in there in 2016?\\n * Checking sources mentally: Many articles claim \\\"hygge\\\" entered in 2016. Some say it was in 2017. Let's assume the prompt doesn't require citation, just general knowledge.\\n * I'll stick with it.\\n\\n * Wait, did they add 'hyggelig'?\\n * Okay.\\n\\n * \\\"Denmark's capital city has a park built by a man who was killed in a car crash.\\\" -> No.\\n * \\\"It was voted the happiest country in 2003.\\\" -> Gallup World Poll.\\n * \\\"Denmark has 143 castles.\\\"\\n * Okay, final selection.\\n\\n * Draft: \\\"The Danish concept of 'hygge,' which describes a feeling of coziness and well-being, was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Let's check if there is a better one.\\n * \\\"Denmark is a flat country.\\\" -> Too simple.\\n * \\\"Denmark has a high standard of living.\\\"\\n * \\\"It has the world's first newspaper vending machine.\\\" -> Maybe.\\n * \\\"Denmark has the world's first traffic lights.\\\" -> No.\\n * \\\"The first traffic light was in Copenhagen.\\\"\\n * Okay, \\\"hygge\\\" feels safer.\\n\\n * Sentence: \\\"The Danish concept of 'hygge,' which means cozy and relaxed, was added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Or maybe the flag one is more visually descriptive.\\n * \\\"The Danish flag is the oldest national flag still in continuous use.\\\" -> \\\"continuous use\\\" implies it hasn't changed design significantly.\\n * I will stick with the dictionary one.\\n\\n * Wait, is it \\\"Oxford English Dictionary\\\" or just \\\"English Dictionary\\\"? Oxford.\\n * Sentence structure: \\\"In Denmark, the Danish word 'hygge,' which means cozy, was added to the Oxford English Dictionary in 2016.\\\"\\n * Correction: \\\"Hygge\\\" is the word.\\n * \\\"The word 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Is it Danish? Yes.\\n * Does it make sense as a fact? Yes.\\n\\n * Alternative: \\\"Denmark has more bicycles per person than cars.\\\" -> False.\\n * \\\"The Danish capital has a lot of bridges.\\\"\\n * Okay.\\n\\n * Wait, another popular fun fact is: \\\"The first person to live in a car.\\\"\\n * No.\\n * \\\"It's impossible to see the sun set on the equator in Denmark.\\\" -> No.\\n * \\\"It has no mountain higher than 1 meter.\\\" -> False, 126 meters max.\\n * Okay, the dictionary one is good.\\n\\n * Final choice: \\\"The Danish concept of coziness, known as 'hygge,' was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Let's refine for punchiness.\\n * \\\"Denmark's unique concept of 'hygge'—referring to cozy social comfort—was added to the Oxford English Dictionary in 2016.\\\"\\n * Okay.\\n\\n * Wait, is it the country or the concept? Both.\\n * Maybe \\\"The Danish concept of 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Add \\\"in Denmark\\\"? -> \\\"In Denmark, the word 'hygge,' which means cozy, was added to the Oxford English Dictionary in 2016.\\\" -> Repetitive.\\n * Okay.\\n * Let's go with that.\\n\\n * Final sentence: \\\"The Danish concept of 'hygge,' meaning cozy and comfortable, was added to the Oxford English Dictionary in 2016.\\\"\\n * Check constraints again. One sentence.\\n * Done.\\n\\n4. **Wait, wait.**\\n * Are there other fun facts?\\n * \\\"The longest bridge is a footbridge.\\\"\\n * \\\"The shortest road.\\\"\\n * \\\"It has a statue of a queen.\\\" -> No.\\n * Okay, let's check \\\"Denmark is the land of happiness.\\\" -> \\\"The happiest place on earth.\\\"\\n * \\\"The Danes have the shortest haircuts.\\\" -> No.\\n * Okay, sticking with OED fact.\\n * Wait, the user asked for a *fun* fact.\\n * \\\"Denmark was once ruled by a king who liked cats.\\\" -> No.\\n * Okay, \\\"The word 'hygge' entered the Oxford English Dictionary in 2016.\\\"\\n\\n * Okay.\\n\", \"content\": \"\\n\\nThe Danish concept of 'hygge,' which refers to coziness and well-being, was officially added to the Oxford English Dictionary in 2016.\" In the next example I run with slightly higher repetition_penalty - and here output degenerates rapidly: curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 2048, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.2, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Topic: Denmark (country).\\n * Constraint: One sentence only.\\n * Tone: Fun fact.\\n\\n2. **Brainstorming Facts about Denmark:**\\n * Lego is Danish? Yes, but maybe too cliché. \\\"It was founded by Hans Christian Andersen?\\\" No, wrong name. It's known for windmills and hygge. Flat landscape with lots of beaches on some islands vs mountains elsewhere in Europe... Wait that doesn't sound right either. Actually most countries are low-lying due to sea level rise fears or something like Copenhagen has high tides sometimes called 'High Tide'. Or perhaps it rains every day except Sunday - this might be a joke rather than true.. Let me check facts.\\\"Denmark invented LEGO\\\" -> The toy company started as wooden models made up from various parts which they could put together\\\". But wait there were also more specific items. What kind would make someone smile then laugh out loud at their own stupidity when I said no such things exist?\\\". Ah yes! Here comes an idea; How many people speak English perfectly well here compared other Europeans because so much time spent watching TV programs broadcasted worldwide where everyone speaks multiple languages easily without learning any new words first!\\\" That isn't quite accurate though since language barrier does still exists even within Scandinavia itself alone today according studies show otherwise now actually!\\\". Okay let’s focus back on actual history related stuff regarding famous inventions happening around world before moving forward towards modern times later again next.\\\". Hmm okay good enough information provided above should suffice if written properly using simple vocabulary terms plus concise expression styles applied throughout writing process overall making sense easy reading possible anytime anywhere whenever needed soonest tomorrow morning afternoon evening night all hours round clock everyday forever until eternity ends someday sometime neveragain evermore whatsoever whateverwhereverwheneverwhywhenwhatifhowaboutthisandthatthensoyesnoormaybebutnotnorthernsouthernmidwesterneasterncentralinnerouterupperlowermiddlehighlowshortlongfar nearbig smallheavy lightdarkbrightdullclearcloudyfairstarkrainysunnywindygustyfoggyshoweryhailingsnowfallfreezingmeltingboilingcondensingevaporatingsublimationprecipitationtranspirationphotosynthesiscellularrespirationdigestionabsorptionmetabolismhomeostasisreproductiongrowthdifferentiationmaturationapoptosismitochondriaorganicinorganicacidbaseoxidizereducentelectronegativityconductivityspecificheatlatentheatspecificenergykineticpotentialmechanicalchemicalthermalnucleargravitationalhydrostaticbarometricatmosphericweatherclimateenvironmentecologybiodiversityspeciesdiversityecosystemhabitatnichepopulationcommunityinteractioncompetitionpredationsymbiosisparasitismmutualismandcommensalismcooperationcollaborativeconsensusdemocracyliberalauthoritarianautocraticanarchisticfascisttotalitariumcommunistsocialismanarcho-capitalismeconomicinequalitypovertywealthresourceallocationdistributionconsumptionprofitmarketdemandsupplypriceinterestratevalueexchangeutilityscarcitysatisfactionneedwantsdesirepreferenceschoicesdecisionsplanningbudgetfinancesinvestmentsavingsassetsliabilitiesdebtcreditriskguaranteewarrantylifecycleownershipmaintenancerepairreplacementretirementpensioninsurancebenefitscompensationrewardrecognitionappreciationsuccessachievementaccommodationcomfortluxuryconvenienceeaseaccessibilityavailabilityresponsivenessservicecustomerclientsupportfeedbackloopimprovementefficiencyproductivityperformanceoutputinputthroughputconversionraterevenuecostspricingstrategymarketsharebrandawarenessimageperceptionpositioningattractivelocalglobalnationwideinternationalworldwidemulticulturalcrossculturalinterracialinterfaithintrareligiousethnocentricxenophobicracialnationalreligioussecularaestheticbeautyartcreativestyleformdesignstylefunctionalsignificancemeaningpurposeintentmessagecommunicationlanguageexpressionvoicearticulationtonepitchvolumeclaritydistortionnoisebackgroundambienceacousticsonicsoundaudiovisualsensorysynesthesiachromaticcolorfulnesshuebrightnesssaturationcontrastshadingtexturelightshadowreflectanceopacitytranslucencydensitymassvolumeweighthumiditytemperaturepressureforcevelocityaccelerationmomentuminertiafrictionviscosityelasticmodulusstrengthdurabilityhardnessexactitudetransparencyimpuritiesdefectsflawsweakpointsfailuresbreakagescratchesdentchipsdingsbruiseabrasionscorrosiorrusttarnishwearouterosivegrindingpolishingcuttingslicingcarvingengravingdrillingboringreamingtappingthreadinggasketsealingvalvesleakagepipeslinesductssprinklerirrigationwaterworksfiltratesedimentstorageaccumulateloadbearingcompressionexpansioncontractionflexuretorsioneccentrictangentialradialnormalcurvatureangledirectionorientationinclinationdeclinationlatitudelongitudealtitudeelevationheightdepthleveldistanceproximityclosenessnearbyremoteisolatedremoteseparateconnectedlinkedbondedsimilaridenticalsametypekindnatureessencesubstancecompositionstructureorganizationarrangementpatternorderchaosdisorgandomesmessynescrambledmixedupconfusedjumbledentangledknottedtwistedcrookedstraightforwarddirectindirectambiguousunclearobfuscatelegraphiccryptographicephermerencesectionsparagrapbspageschaptersshelfsectiontopicdivisionpartsegmentportionfractionquantitynumbercountmeasuremetricstandardunitssystemmeasurementweighthingscalingdimensionalgeometricaltopologicalspatialtemporalchronologicaleventsequentialtimelineperiodintervaltimefrequencydurationlengthwidththicknessareaavolumeamountweightcapacitycapabilitypossiblerequirementexpectancyhopeanticipationpredictionforecastprojectionspeculationguessestimatejudgmentopinionassumptionbeliefconfidencecertaintyprobabilityoddslikelihoodchanceriskgamblebetwinninglossbreakingmakinglosingpayingchargingbillingfeepremiumsurchargefeeassessmenttaxlevydutytariffimportexportcustomsdraftbillstatuteregulationsordinancemarxlawofeconomicdevelopmentlawsregulatorylegallegislationconstitutionjurisdictioncourtsjudgejusticecrimefelonyoffensepunishmentsentencepenaltyfineforfeitforbiddenillegalillicitunauthorizedundesirabledangerousunsafeimmovablemobilemovabletransferportabledispersedistributedstoreduploadeddownloadsharedpublicprivatepersonalproprietaryintellectualpropertycopyrighttrademarkservice trademark service trade secret proprietary intellectual property copyright patent design application invention creation development innovation technology science research education health care social services public policy government administration bureaucracy governance democracy liberty equality justice freedom human rights universal values international norms global standards local rules national laws regional agreements intergovernmental cooperation multilateral collaboration bilateral partnership tri-lateral alliance coalition treaty pact accord protocol understanding memorandum agreement commitment declaration statement position view opinion belief thought feeling emotion attitude perspective angle direction aspect factor element component part segment portion fraction quantity number count measure metric standard unit system measurement weighing scaling dimensional geometrical topological spatial temporal chronological sequential timeline period interval duration length width thickness area volume amount weight capacity capability possibility requirement expectation hope anticipation prediction forecast projection speculation guess estimate judgment assumption condition necessity impossibility constraint limitation restriction rule regulation law principle axiom theorem postulate hypothesis conjecture theory proposition assertion claim argument debate discussion conversation dialogue monologue narrative story tale legend myth folklore tradition custom culture heritage legacy ancestry descent bloodline genetic inheritance familial lineage genealogical tree family pedigree ancestors descendants progeny offspring children grandchildren great-grandchildren etceterasiblingspartnersfriendsrelativesneighborsextendedfamilynetworksocialsupportcirclesrelationshipsconnectionslinksbridgestiesetieabroadlocaldomesticforeignoverseacontinentalhemisphereplanetsolarsystemgalaxyuniversmultiverseeverythingallnothingnonexistnullvacuumvoidemptinesssilencequietstillnesstopausebreatherrestslacksettlementresolutionclosurecompletionterminationendingfinishdonepastpresentfuturehistorymemoryrecallforgetforgivenoignoreneglectdismissrejectacceptapproveadoptchooseselectpicktakeholdgethavekeepownlosegivegrantdonatenextnextstepfollowingbeforeafterduringwhileuntilsincefromtointoontouptoadatabooktextdocumentfilepaperpageprintwordletteremailnotememoentryrecordlogdatabaseinformationdataknowledgewisedomainspecificationsrequirementsstandardsrulesguidelinespolicyproceduremanualinstructionguidebookhelpdeskassistanceconsultantexpertadviceaidcomfortcarehealthwellbeingfitnessactivityexerciselearningtrainingteachingeducationschoolcollegedeskclassroomlaboratoryequipmentmachinerytoolinstrumentgeardeviceappliancedigitalanaloguehybriddesktopworkstationservercomputerprocessorchipmicroelectronicssemiconductorphotolithographyetchingoxygenizingnitrogenpurgingargonshieldingasphygasaircompressorradiowavesmicrowavesmillimeterswaveinfaredultravioletvisiblelightspectrumradiolightelectriccurrentvoltammeterskilometersvoltageluxlumensohmsampsjoulescoulombssecondsminuteshoursdaysweeksmonthsyearsdecennialsseptenaryoctennialquinquagenarianvigintiarytri-centurycircularquadricularenglandswedenbelgiumgermanyfrancenetherlandsdenmarksaxtoniaitaliashriyenafreezelandmondoitaliaeuromonacoathenaolympusmontpelierspainnylonsteelwoodbrickglassstonecementplasticonlyerubberfibermetalceramicmineralsoilwatervineyardcornfieldforestmarshmeadowwetlandsgrasslaandleisurezoneurbanareasbuildingconstructionhousingfacilityinstallationfactoryplantmachinevehicletransportshipboattrainbusmetrotramcartvantruckmotorcycleautomobilemotoauto-carjeetcarsvehicletypetypesmodelsmakebrandsmanufacturerproducercreatorfabricatorinventormakercraftsmanbuilderartisanartistwritercomposeractorperformerpresentertalentstarcelebritystudiosceneplayfilmcinemaauditoriumtheaterconcertorchestraoperamusichallsoundsystemsplaylisttracklistlibrarycollectionmusicgenretypecategorylabelpublishereditorjournalmagazineradiochannelsstationsbroadcastsignaltelecommunicationsmediainternetwebonlinevirtualcyberspaceblockchainbitcoinethereumaltcoinscryptoassetdigitalcoinfiattokenledgerwallettransactioncontractsmartcontractscodealgorithmprotocolframeworklayerstacktechnodearchitectureengineeringstructurescomponentspartsmodulesfunctionsoperationsmethodsprocessestrategiesplansprogramsprojectsinitiativescampaignsstoriesideasthemesconceptsthoughtmodelsprincipledesignsbusinessenterprisesorganizationsindustriessectorsfieldsdomainsnicheexpertise\", \"content\": null } With running docker server with --tensor-parallel-size 1 \\ (without \"--enforce-eager\") I get this garbled output: curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 2048, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.2, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\", \"content\": null } ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38994", "createdAt": "2026-04-04T19:51:48Z", "updatedAt": "2026-08-13T02:13:57Z", "timestamp": "2026-08-13T02:13:57Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "AlexanderValentini", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:40756", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MTP speculative decoding crash with illegal memory access on long sequences (Qwen3.6-27B-FP8, v0.19.1)", "text": "### Your current environment
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug Description When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on long requests. Environment vLLM version: 0.19.1 Model: Qwen3.6-27B-FP8 TP size: 4, fp8 quantization, prefix caching + chunked prefill enabled Speculative config: method=mtp, model=same as target, num_spec_tokens=5 Symptoms The crash occurs after the request has accumulated ~26 k total tokens and has generated >1200 output tokens. Just before the crash, spec metrics become abnormal: accepted tokens equal drafted tokens, acceptance rate jumps to 100%, and the scheduled draft tokens are all -1. All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)). Excerpt from logs SpecDecoding metrics: Mean acceptance length: 6.00, ..., Avg Draft acceptance rate: 100.0% scheduled_spec_decode_tokens={...: [-1, -1, -1, -1, -1]} ... torch.AcceleratorError: CUDA error: an illegal memory access was encountered File \"gpu_model_runner.py\", line 1706, in _prepare_input_ids prev_common_req_indices_tensor = torch.tensor( To reproduce Serve Qwen3.6-27B-FP8 with MTP speculative decoding using --num-speculative-tokens 5. Send a conversation that grows to 25k+ tokens total. The crash typically happens after 1000+ tokens have been generated. Expected behavior Generation should continue without invalid draft tokens or illegal memory access. Additional notes GPU memory usage was low (~5.4% KV cache), so it’s not an OOM issue. The problem is reproducible; disabling speculative decoding avoids the crash. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/40756", "createdAt": "2026-04-24T02:28:03Z", "updatedAt": "2026-08-13T06:08:26Z", "timestamp": "2026-08-13T06:08:26Z", "metrics": { "reactions": 14, "comments": 36 }, "labels": [ "bug" ], "author": "SongXiaoMao", "state": "open", "assignees": [ "z1ying" ] }, { "id": "github:vllm-project/vllm:issue:41623", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Decode Context Parallelism (`--decode-context-parallel-size`) output drift and gibberish in v0.21.0 and latest nightly", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.15.134+release+2.10.0r8-amd64-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA A100-SXM4-80GB GPU 1: NVIDIA A100-SXM4-80GB GPU 2: NVIDIA A100-SXM4-80GB GPU 3: NVIDIA A100-SXM4-80GB GPU 4: NVIDIA A100-SXM4-80GB GPU 5: NVIDIA A100-SXM4-80GB GPU 6: NVIDIA A100-SXM4-80GB GPU 7: NVIDIA A100-SXM4-80GB Nvidia driver version : 590.48.01 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: AuthenticAMD Model name: AMD EPYC 7742 64-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 2 Stepping: 0 Frequency boost: enabled CPU max MHz: 2250.0000 CPU min MHz: 1500.0000 BogoMIPS: 4500.26 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es Virtualization: AMD-V L1d cache: 4 MiB (128 instances) L1i cache: 4 MiB (128 instances) L2 cache: 64 MiB (128 instances) L3 cache: 512 MiB (32 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-63 NUMA node1 CPU(s): 64-127 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled Vulnerability Spec rstack overflow: Mitigation; SMT disabled Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.595.45 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchvision==0.26.0+cu130 [pip3] transformers==5.7.0 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.1rc1.dev91+ga749a33d8 (git sha: a749a33d8) vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 NODE NODE 64-127 1 N/A GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 NODE NODE 64-127 1 N/A GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 NODE NODE 64-127 1 N/A GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X NODE NODE 64-127 1 N/A NIC0 SYS SYS SYS SYS NODE NODE NODE NODE X PIX NIC1 SYS SYS SYS SYS NODE NODE NODE NODE PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=GPU-f810eb6d-56e1-1937-6319-dc03bb5381c2,GPU-d6e4c7e1-a414-78e0-4c9e-7d3d1a6f97a5,GPU-972f9bbe-6f43-b12f-2ad3-ca8c2d44328a,GPU-238b6428-8034-d89d-ff91-d9b7d0dc6cbd,GPU-5c2d9c0d-7b82-9b66-9a0b-30238270fac3,GPU-bad1cbfa-554c-22c7-ccb8-e24d4a26883e,GPU-4e92be98-9b5d-af97-0e44-1a50cc7971ea,GPU-b22f4dec-4abb-6dcb-c1aa-52a2a705758e NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 VLLM_ENABLE_CUDA_COMPATIBILITY=0 VLLM_ENGINE_READY_TIMEOUT_S=86400 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug GPU: 8x A100 `vllm serve moonshotai/Kimi-K2.6 --port 5000 --host 0.0.0.0 --download-dir /workspace/.cache/huggingface/hub --api-server-count 8 --tensor-parallel-size 8 --decode-context-parallel-size 8 --trust-remote-code --enable-chunked-prefill --enable-prefix-caching --max-num-seqs 32 --gpu-memory-utilization 0.95 --max-model-len 262144 --enable-auto-tool-choice --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --mm-processor-cache-gb 8 --mm-processor-cache-type shm` Decode Context Parallelism (DCP, `--decode-context-parallel-size`) produces unrelated gibberish output in latest nightly. This is a regression. Happens somewhere between `a749a33d8d05acdd3ab346bd3f0c6b5c9c80474f` (works well) `01d4d1ad375dc5854779c593eee093bcebb0cada` (gibberish output). Only when `--decode-context-parallel-size` is set, Kimi-K2.6 leads to completely unrelated gibberish. Problematic commit is between the above commits. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/41623", "createdAt": "2026-05-04T10:57:26Z", "updatedAt": "2026-08-13T17:08:51Z", "timestamp": "2026-08-13T17:08:51Z", "metrics": { "reactions": 0, "comments": 20 }, "labels": [ "bug" ], "author": "ehfd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:41814", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "NixlConnector hardcodes backends=[\"UCX\"] default; no env-var override path; LIBFABRIC/EFA operators must discover kv_connector_extra_config.backends from source", "text": "### Your current environment - vLLM 0.17.1 (bundled in `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0`) - Dynamo 1.1.0 runtime (for disaggregated prefill/decode serving) - NIXL `nixl_cu12` 1.0.1 with LIBFABRIC + UCX plugins both present on disk - AWS EFA hardware (SRD transport, libfabric provider `efa`) - 2× P5.48xlarge H100 HyperPod nodes - `libplugin_LIBFABRIC.so` and `libplugin_UCX.so` both available at `/opt/dynamo/venv/lib/python3.12/site-packages/.nixl_cu12.mesonpy.libs/plugins/` ### 🐛 Describe the bug `NixlConnector` hardcodes `backends=[\"UCX\"]` as the default in `vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py:1023`. There is no environment-variable fallback for backend selection, and this default is not documented. Operators running vLLM + NIXL on EFA (where UCX can't establish cross-node handshakes, but libfabric works) have no way to switch backends short of reading the source. The NIXL library itself supports multiple backends — the limitation is entirely in vLLM's default. ```python # nixl_connector.py:1022-1024 self.nixl_backends = vllm_config.kv_transfer_config.get_from_extra_config( \"backends\", [\"UCX\"] ) ``` Setting `NIXL_BACKEND=LIBFABRIC` or `VLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC` as environment variables does **nothing** — neither string appears in `nixl_connector.py`. Operators routinely discover this by tailing logs and seeing: ``` NIXL INFO _api.py:361 Backend UCX was instantiated ... NIXL transfer failure: handshake_failed ``` ### 🛠️ How to reproduce 1. Deploy disaggregated vLLM with `NixlConnector` on AWS EFA (or any non-RDMA-over-Ethernet fabric where UCX can't handshake). 2. Set `--kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'`. 3. Set env `NIXL_BACKEND=LIBFABRIC` and `FI_PROVIDER=efa` on both workers. 4. Observe: `NIXL _api.py:361 Backend UCX was instantiated` followed by cross-node `handshake_failed`. ### ✅ Workaround The only way to override the default today is via `kv_connector_extra_config.backends`: ```bash --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"backends\":[\"LIBFABRIC\"]}}' ``` Once the workaround is applied, LIBFABRIC instantiates and EFA handshakes succeed. ### 💡 Suggested fix Either: 1. **Read `NIXL_BACKEND` / `VLLM_NIXL_BACKEND` from env as a fallback** when `kv_connector_extra_config.backends` is absent — consistent with how NCCL + other vLLM networking knobs are exposed. 2. **Document the `kv_connector_extra_config.backends` path** prominently in `docs/source/models/kv_transfer.md` (or equivalent). Currently the JSON schema is discoverable only by reading connector source. 3. **Surface a startup WARN** when `NIXL_BACKEND` env is set but is being ignored because the connector is bypassing it — catches the common operator mistake. ### Evidence Reproducible from commit `` on our PR branch. Verified against `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0`. Cross-references: - Downstream blocker: [aws-samples/awsome-inference#72](https://github.com/aws-samples/awsome-inference/pull/72) — this limitation was the real root cause mis-diagnosed as a Dynamo operator issue in the related `ai-dynamo/dynamo#9200`. cc whomever owns the `NixlConnector` module.", "url": "https://github.com/vllm-project/vllm/issues/41814", "timestamp": "2026-08-12T16:48:07Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "stale" ], "author": "dmvevents", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:41963", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[ROCm/MI325X] DeepSeek-V4-Flash: Triton fp8_mqa_logits kernel requires 96KB shared memory, MI325X limit is 64KB", "text": "## Environment - **Hardware**: AMD MI325X × 8 (255.98 GiB HBM each) - **vLLM version**: 0.20.2rc1.dev90+g7a576e2c7 (rocm/vllm-dev:nightly, 2026-05-06) - **Model**: deepseek-ai/DeepSeek-V4-Flash - **Config**: TP=8, EP=8, `--kv-cache-dtype fp8`, `--enforce-eager` ## Bug At inference time, the first request fails with a Triton shared memory OOM: ``` triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 98304, Hardware limit: 65536. Reducing block sizes or `num_stages` may help. ``` Call stack: ``` rocm_aiter_mla_sparse.py:538 → rocm_fp8_mqa_logits aiter/ops/triton/attention/fp8_mqa_logits.py:53 → _fp8_mqa_logits_kernel ``` ## Root Cause The Triton kernel `_fp8_mqa_logits_kernel` in `aiter/ops/triton/attention/fp8_mqa_logits.py` is configured with block sizes that require **96 KB** of LDS (Local Data Share / shared memory). The AMD MI325X GPU supports a maximum of **64 KB** per workgroup. The PR #40871 was developed and tested exclusively on **MI355X**, which appears to support the larger shared memory configuration required by this kernel. ## Impact DeepSeek-V4-Flash is completely non-functional for inference on MI325X. The model loads correctly but crashes on the first forward pass. ## Suggested Fix Add a hardware-capability check before launching the kernel, and reduce `BLOCK_SIZE` or `num_stages` for GPUs with ≤64 KB shared memory. The fix should be in `aiter` (the AITER library used by vLLM ROCm). ## Related - PR #40871 \"Add AMD support for DeepSeek V4\" (merged 2026-05-05) — tested on MI355X only - Issue #41961: `mul_cuda` not implemented for `Float8_e8m0fnu` (MI325X) - Issue #41962: OOM in `rocm_dequantize_blocked_k_cache` + CUDAGraph incompatibility", "url": "https://github.com/vllm-project/vllm/issues/41963", "createdAt": "2026-05-07T14:01:05Z", "updatedAt": "2026-08-13T03:06:53Z", "timestamp": "2026-08-13T03:06:53Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "rocm", "stale" ], "author": "angelopaolosg", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:42508", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Inconsistent speculative decoding acceptance metrics between vLLM and SpecForge on Qwen3-32B baselines", "text": "### Proposal to improve performance _No response_ ### Report of performance regression _No response_ ### Misc discussion on performance I am benchmarking speculative decoding baselines in vLLM and observed inconsistent acceptance metrics compared with SpecForge. I would like to understand whether this difference is expected due to different metric definitions, verification logic, benchmark prompt formatting, or whether it may indicate a configuration mismatch or a potential issue in the vLLM speculative decoding benchmark path. The target model is Qwen3-32B. I tested two speculative decoding baselines: Qwen3-32B with Eagle3-Qwen3-32B-zh, and Qwen3-32B with Qwen3-0.6B as a standalone draft model. In both cases, I set num_speculative_tokens to 3, used max concurrency / batch size 16, greedy decoding with temperature=0 and top_p=1.0, output length 1024, and the same full GSM8K test set with 1319 samples for both vLLM and SpecForge.My intention was to compare chain-style speculative decoding baselines, not tree-style candidate expansion. For the EAGLE3 baseline, I launched vLLM with the following command: VLLM_CACHE_ROOT=/root/.cache/vllm_eagle3_argmax_gpu0 \\ SPEC_DECODE_MODE=baseline \\ CUDA_VISIBLE_DEVICES=0 vllm serve /root/autodl-tmp/Qwen3-32B \\ --served-model-name Qwen3-32B \\ --host 0.0.0.0 \\ --port 8000 \\ --trust-remote-code \\ --dtype bfloat16 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.85 \\ --max-model-len 4096 \\ --max-num-seqs 16 \\ --enable-chunked-prefill \\ --speculative-config '{ \"method\": \"eagle3\", \"model\": \"/root/autodl-tmp/Eagle3-Qwen3-32B-zh\", \"draft_tensor_parallel_size\": 1, \"num_speculative_tokens\": 3 }' The vLLM benchmark command was: vllm bench serve \\ --backend openai \\ --base-url http://127.0.0.1:8000 \\ --endpoint /v1/completions \\ --model Qwen3-32B \\ --tokenizer /root/autodl-tmp/Qwen3-32B \\ --dataset-name custom \\ --dataset-path /root/autodl-tmp/gsm8k_vllm_custom_mt1024.jsonl \\ --custom-output-len 1024 \\ --num-prompts 1319 \\ --request-rate inf \\ --max-concurrency 16 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,95,99 \\ --save-result \\ --save-detailed \\ --result-dir /root/vllm/results \\ --result-filename qwen32b_eagle3_argmax_chain3_gsm8k_custom_np1319_mt1024_bs16_temp0_topp1_gpu0_vllmbench.json \\ --extra-body '{\"temperature\":0,\"top_p\":1.0}' \"Image\" The vLLM result shows a mean acceptance rate of 55.08% and a mean accepted length of 2.65. For the same EAGLE3 baseline, I used SpecForge with the following command: CUDA_VISIBLE_DEVICES=0 python3 bench_eagle3.py \\ --model-path /root/autodl-tmp/Qwen3-32B \\ --speculative-draft-model-path /root/autodl-tmp/Eagle3-Qwen3-32B-zh \\ --port 30000 \\ --host http://127.0.0.1 \\ --trust-remote-code \\ --tp-size 1 \\ --attention-backend fa3 \\ --mem-fraction-static 0.7 \\ --config-list 16,3,1,4 \\ --benchmark-list gsm8k \\ --output-dir ./test \\ --dtype bfloat16 \"Image\" The SpecForge result shows a mean accepted length of 2.32, and the derived mean acceptance rate is around 44%. Therefore, for the EAGLE3 baseline, vLLM reports about 10 percentage points higher acceptance rate than SpecForge. For the standalone draft model baseline, I launched vLLM with the following command: VLLM_CACHE_ROOT=/root/.cache/vllm_qwen0p6b_argmax_gpu0 \\ SPEC_DECODE_MODE=baseline \\ CUDA_VISIBLE_DEVICES=0 vllm serve /root/autodl-tmp/Qwen3-32B \\ --served-model-name Qwen3-32B \\ --host 0.0.0.0 \\ --port 8000 \\ --trust-remote-code \\ --dtype bfloat16 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.85 \\ --max-model-len 4096 \\ --max-num-seqs 16 \\ --enable-chunked-prefill \\ --speculative-config '{ \"method\": \"draft_model\", \"model\": \"/root/autodl-tmp/Qwen3-0.6B\", \"draft_tensor_parallel_size\": 1, \"num_speculative_tokens\": 3 }' The vLLM benchmark command was: vllm bench serve \\ --backend openai \\ --base-url http://127.0.0.1:8000 \\ --endpoint /v1/completions \\ --model Qwen3-32B \\ --tokenizer /root/autodl-tmp/Qwen3-32B \\ --dataset-name custom \\ --dataset-path /root/autodl-tmp/gsm8k_vllm_custom_mt1024.jsonl \\ --custom-output-len 1024 \\ --num-prompts 1319 \\ --request-rate inf \\ --max-concurrency 16 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,95,99 \\ --save-result \\ --save-detailed \\ --result-dir /root/vllm/results \\ --result-filename qwen32b_qwen0p6b_ar_argmax_chain3_gsm8k_custom_np1319_mt1024_bs16_temp0_topp1_gpu0_vllmbench.json \\ --extra-body '{\"temperature\":0,\"top_p\":1.0}' \"Image\" The vLLM result shows a mean acceptance rate of 66.57% and a mean accepted length of 3.00. For the same standalone draft model baseline, I used SpecForge with the following command: CUDA_VISIBLE_DEVICES=0 python3 bench_speculative.py \\ --model-path /root/autodl-tmp/Qwen3-32B \\ --speculative-algorithm STANDALONE \\ --speculative-draft-model-path /root/autodl-tmp/Qwen3-0.6B \\ --port 30000 \\ --host http://127.0.0.1 \\ --trust-remote-code \\ --tp-size 1 \\ --attention-backend fa3 \\ --mem-fraction-static 0.7 \\ --config-list 16,3,1,4 \\ --benchmark-list gsm8k \\ --output-dir ./test \\ --dtype bfloat16 \"Image\" The SpecForge result shows a mean accepted length of 3.17, and the derived mean acceptance rate is around 72%. Therefore, for the standalone draft model baseline, vLLM reports about 6 percentage points lower acceptance rate than SpecForge. The direction of the difference is not consistent. For EAGLE3, vLLM reports higher acceptance than SpecForge. For the standalone draft model baseline, vLLM reports lower acceptance than SpecForge. I am mainly trying to understand whether the above difference is expected, or whether it suggests a configuration mismatch or a possible issue in the speculative decoding benchmark path. ### Your current environment (if you think it is necessary) ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by Anaconda, Inc. | (main, Mar 19 2026, 20:20:58) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-5.15.0-171-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H20 GPU 1: NVIDIA H20 GPU 2: NVIDIA H20 GPU 3: NVIDIA H20 GPU 4: NVIDIA H20 GPU 5: NVIDIA H20 GPU 6: NVIDIA H20 GPU 7: NVIDIA H20 Nvidia driver version : 580.105.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 224 On-line CPU(s) list: 0-223 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8480+ CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 56 Socket(s): 2 Stepping: 8 CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 5.3 MiB (112 instances) L1i cache: 3.5 MiB (112 instances) L2 cache: 224 MiB (112 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-55,112-167 NUMA node1 CPU(s): 56-111,168-223 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.590.48 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchvision==0.26.0 [pip3] transformers==5.6.2 [pip3] triton==3.6.0 [conda] flashinfer-python 0.6.8.post1 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.0.3 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.18.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.4.2 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.4.2 pypi_0 pypi [conda] nvidia-ml-py 13.590.48 pypi_0 pypi [conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] torch 2.11.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchvision 0.26.0 pypi_0 pypi [conda] transformers 5.6.2 pypi_0 pypi [conda] triton 3.6.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.2rc1.dev42+g98661fe01 (git sha: 98661fe01) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYS SYS 0-55,112-167 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE 56-111,168-223 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE NODE 56-111,168-223 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE PIX 56-111,168-223 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE 56-111,168-223 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS NODE X PIX NODE SYS SYS NIC2 NODE NODE NODE NODE SYS SYS SYS SYS NODE PIX X NODE SYS SYS NIC3 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE X SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NIC5 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 ============================== Environment Variables ============================== PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_test001 ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/42508", "createdAt": "2026-05-13T08:50:11Z", "updatedAt": "2026-08-13T13:28:25Z", "timestamp": "2026-08-13T13:28:25Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "performance" ], "author": "Mterzzl", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:42932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vLLM wheel version mismatch", "text": "### Your current environment vLLM wheel that is getting installed is different from that of the branch used. Example Docker is generated using the vllm branch - 0.19.0 , https://github.com/vllm-project/vllm/tree/releases/v0.19.0 But while building the wheel/docker, it generates the wheel which has version 0.19.1 pip3 list -l | grep vllm vllm 0.19.1.dev0+g2a69949bd.d20260417.rocm713 Same applies for vLLM branch - 0.19.1 as well vLLM branch used --- https://github.com/vllm-project/vllm/tree/releases/v0.19.1 Wheel that is generated has the version as pip3 list -l | grep vllm vllm 0.19.2.dev3+g24efb8904.d20260514.rocm713 ### 🐛 Describe the bug There are no crashes or faults, This mismatch should not happen ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/42932", "createdAt": "2026-05-18T06:18:15Z", "updatedAt": "2026-08-13T17:07:51Z", "timestamp": "2026-08-13T17:07:51Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug" ], "author": "jvadakat", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:46589", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Segfault encountered in qwen3.5-27B long video (>2G)inference", "text": "### Your current environment
The output of python collect_env.py ```text Your output of `python collect_env.py` here 2026-06-24 07:11:18 (110 B/s) - 'collect_env.py' saved [35090/35090] Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (aarch64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 15.0.7 CMake version : version 4.3.2 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cpu Is debug build : False CUDA used to build PyTorch : None ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.11.15 (main, May 13 2026, 07:56:17) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.10.0-216.0.0.115.oe2203sp4.aarch64-aarch64-with-glibc2.35 ============================== CPU Info ============================== Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: HiSilicon BIOS Vendor ID: HiSilicon Model name: Kunpeng-920 BIOS Model name: HUAWEI Kunpeng 920 5250 Model: 0 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 4 Stepping: 0x1 BogoMIPS: 200.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs L1d cache: 12 MiB (192 instances) L1i cache: 12 MiB (192 instances) L2 cache: 96 MiB (192 instances) L3 cache: 192 MiB (8 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-23 NUMA node1 CPU(s): 24-47 NUMA node2 CPU(s): 48-71 NUMA node3 CPU(s): 72-95 NUMA node4 CPU(s): 96-119 NUMA node5 CPU(s): 120-143 NUMA node6 CPU(s): 144-167 NUMA node7 CPU(s): 168-191 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] numpy==1.26.4 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+cpu [pip3] torch_npu==2.10.0 [pip3] torchaudio==2.10.0+cpu [pip3] torchvision==0.25.0+cpu [pip3] transformers==5.5.3 [pip3] triton==3.5.0 [pip3] triton_ascend==3.2.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/Ascend/cann-9.0.0/lib64:/usr/local/Ascend/cann-9.0.0/lib64/plugin/opskernel:/usr/local/Ascend/cann-9.0.0/lib64/plugin/nnengine:/usr/local/Ascend/cann-9.0.0/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/python3.11.15/lib::/usr/local/lib OMP_NUM_THREADS=1 TORCH_DEVICE_BACKEND_AUTOLOAD=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
### 🐛 Describe the bug log: ``` (APIServer pid=89271) INFO 06-24 02:35:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% !!!!!!! Segfault encountered !!!!!!! File \"\", line 0, in avio_seek File \"\", line 0, in avformat_open_input File \"Objects/typeobject.c\", line 1103, in type_call File \"Objects/call.c\", line 214, in _PyObject_MakeTpCall File \"Python/ceval.c\", line 4769, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 59, in method_vectorcall File \"Objects/call.c\", line 257, in _PyVectorcall_Call File \"Objects/call.c\", line 328, in _PyObject_Call File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 89, in method_vectorcall File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 67, in method_vectorcall File \"./Modules/_threadmodule.c\", line 1124, in thread_run File \"Python/thread_pthread.h\", line 241, in pythread_wrapper File \"\", line 0, in 0xffffffffffffffff ``` ``` vllm serve /home/model_weights/Qwen3.5-27B \\ --served-model-name \"qwen3.5\" \\ --host 0.0.0.0 \\ --port 8888 \\ --data-parallel-size 1 \\ --tensor-parallel-size 8 \\ --max-num-batched-tokens 16384 \\ --max-num-seqs 128 \\ --gpu-memory-utilization 0.92 \\ --compilation-config '{\"cudagraph_capture_sizes\":[1,6,12,18,24,30,36,42,48,50,54,72,78,84,90,96,102,108,144,192], \"cudagraph_mode\":\"FULL_DECODE_ONLY\"}' \\ --speculative_config '{\"method\": \"qwen3_5_mtp\", \"num_speculative_tokens\": 5, \"enforce_eager\": true}' \\ --trust-remote-code \\ --async-scheduling \\ --allowed-local-media-path / \\ --mm-processor-cache-gb 0 \\ ``` long video: https://modelscope.cn/datasets/Byters/HourVideo/tree/master/videos 01236d91-23a2-49be-8f9f-d25cd3e126f6.mp4 test curl: ``` video_path=${1:-'/xxx/xxx.mp4'} curl http://localhost:8888/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d \"{ \\\"model\\\": \\\"qwen3.5\\\", \\\"messages\\\": [ {\\\"role\\\": \\\"system\\\", \\\"content\\\": \\\"You are a helpful assistant.\\\"}, {\\\"role\\\": \\\"user\\\", \\\"content\\\": [ {\\\"type\\\": \\\"video_url\\\", \\\"video_url\\\": {\\\"url\\\": \\\"file://${video_path}\\\"}}, {\\\"type\\\": \\\"text\\\", \\\"text\\\": \\\"Summarize the video content in one sentence.\\\"} ]} ] }\" ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/46589", "createdAt": "2026-06-24T07:36:42Z", "updatedAt": "2026-08-13T08:30:54Z", "timestamp": "2026-08-13T08:30:54Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug" ], "author": "shuixiaoer", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:46967", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]:[New Model] Gemma4UnifiedForConditionalGeneration (google/gemma-4-12B-it)", "text": "### The model to consider. `google/gemma-4-12B-it` — `model_type: gemma4_unified`, architecture `Gemma4UnifiedForConditionalGeneration` (Google's unified text+vision+audio \"omni\" Gemma 4). Public repo; ships no remote code (no `auto_map` / modeling `.py`). ### The closest model vLLM already supports. vLLM registers `Gemma4ForCausalLM` and `Gemma4ForConditionalGeneration`, but **not** the `Unified` variant. The supported-models docs list only `Gemma4ForCausalLM`. ### What happens today. Serving fails at model load, before any request: ValueError: The checkpoint you are trying to load has model type gemma4_unified but Transformers does not recognize this architecture. Verified in-image: - `Gemma4UnifiedForConditionalGeneration` is **absent** from `ModelRegistry.get_supported_archs()`. - `gemma4_unified` is **absent** from `transformers` `CONFIG_MAPPING_NAMES`. - No remote code in the repo, so `--trust-remote-code` does not help. ### Request. Add `Gemma4UnifiedForConditionalGeneration` to vLLM's model registry, and document the minimum `transformers` version that provides the `gemma4_unified` config.", "url": "https://github.com/vllm-project/vllm/issues/46967", "createdAt": "2026-06-28T17:16:35Z", "updatedAt": "2026-08-13T15:00:20Z", "timestamp": "2026-08-13T15:00:20Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "feature request" ], "author": "adityasingh0510", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:47277", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Qwen3.5 native MTP can be slower than no-MTP CUDA graph baseline despite good acceptance", "text": "## Summary We are benchmarking a Qwen3.5 OCR checkpoint with native MTP support in vLLM. The model has a lightweight native MTP branch: - `mtp_num_hidden_layers=1` - `mtp_use_dedicated_embeddings=false` In our OCR workload, MTP acceptance is not terrible. MTP1 reaches around `82%-88%` acceptance, and MTP2 can reach mean accepted length above `2.3`. However, end-to-end throughput in graph mode is still below the best no-MTP vLLM baseline. The main issue seems to be that the vLLM spec decode execution path adds enough per-step overhead to erase the theoretical benefit from fewer decode steps. This is particularly visible with `num_speculative_tokens=2`, where rejection sampling, metadata preparation, slot mapping, and `num_computed_tokens` updates become large overheads. We would like to understand whether this is expected for native Qwen MTP today, whether `num_speculative_tokens > 1` currently requires repeated forward/processing through the same MTP layer, and whether there are planned optimizations for the graph-mode spec decode path. ## Environment - vLLM: `0.23.0` - PyTorch: `2.11.0+cu130` - GPU: NVIDIA L20X - Tensor parallel: `1` - CUDA graph mode: enabled, `enforce_eager=false` - Workload: multimodal OCR / document parsing, long outputs - Decode temperature: `0` Main generation config: ```text max_new_tokens=16384 max_model_len=32768 max_num_batched_tokens=32768 gpu_memory_utilization=0.8 max_num_seqs=None ``` We intentionally did not give MTP a larger `max_num_seqs` or give baseline a smaller one. Both paths use the same input subset and batching-related configuration. ## Speculative configs tested MTP1: ```json {\"method\":\"mtp\",\"num_speculative_tokens\":1,\"use_local_argmax_reduction\":true} ``` MTP2: ```json {\"method\":\"mtp\",\"num_speculative_tokens\":2,\"use_local_argmax_reduction\":true} ``` ## Results ### n64 graph-mode clean benchmark | mode | tok/s | req/s | bench sec | output tokens | finish | acceptance | mean accept length | per-position acceptance | | --- | ---: | ---: | ---: | ---: | --- | ---: | ---: | --- | | no-MTP baseline | 2246.73 | 1.381 | 46.35 | 104138 | `stop:64` | - | - | - | | MTP1 | 1926.41 | 1.181 | 54.21 | 104432 | `stop:64` | 82.83% | 1.828 | 82.83% | | MTP2 | 1821.76 | 1.119 | 57.17 | 104152 | `stop:64` | 66.77% | 2.335 | 78.21%, 55.34% | MTP1 is about `0.86x` of the no-MTP graph baseline, not a speedup. MTP2 has a higher mean accepted length, but still does not beat the best no-MTP graph baseline. The second speculative token acceptance is also much lower than the first token. ### n32 timing benchmark after local instrumentation | mode | tok/s | output tokens | finish | acceptance | mean accept length | | --- | ---: | ---: | --- | ---: | ---: | | no-MTP baseline | 1898.19 | 29017 | `stop:32` | - | - | | MTP1 before local Triton patch | 1824.31 | 29024 | `stop:32` | 88.06% | 1.881 | | MTP1 after local Triton patch | 1980.96 | 29016 | `stop:32` | 88.08% | 1.881 | | MTP2 after local Triton patch | 2172.36 | 28972 | `stop:32` | 76.03% | 2.521 | The n32 timing case is not our final speed claim; it is mostly for profiling. It suggests that some overhead can be reduced, but the larger n64 graph-mode benchmark still does not show a stable MTP speedup over the best no-MTP baseline. ## Timing hotspots With local timing instrumentation, MTP2 n32 shows the following cumulative costs: | item | cumulative time | | --- | ---: | | `gpu_draft_total` | 3017.8 ms | | `runner_prepare_inputs` | 2276.5 ms | | `reject_sample_kernel` | 1702.8 ms | | `gpu_target_forward` | 1357.8 ms | | `prepare_num_computed_gpu` | 1033.5 ms | | `prepare_num_computed_update_kernel` | 1016.9 ms | | `step35_loop_update_positions` | 668.5 ms | | `prepare_compute_slot_mapping` | 603.1 ms | Before our local patch, `prepare_num_computed_gpu` was one of the clearest overheads. In MTP1 n32, we observed: | item | baseline | MTP1 before patch | | --- | ---: | ---: | | `prepare_num_computed_gpu` | 6.4 ms | 626.7 ms | | `runner_prepare_inputs` | 529.8 ms | 1346.0 ms | | `gpu_target_forward` | 858.0 ms | 1243.5 ms | | `gpu_draft_total` | - | 531.5 ms | We locally replaced a `torch.compile(dynamic=True)` style `update_num_computed_tokens_for_batch_change` path with a small fused Triton kernel. This improved MTP1 n32 throughput from `1824.31 tok/s` to `1980.96 tok/s`, but it did not solve the full graph-mode overhead problem, especially for MTP2. ## Local experiments that were not safe enough to use We also tried more aggressive local fast paths: - a greedy rejection fast path for `temperature=0` - packed/allfast variants - faster attention metadata experiments - GPU token cache / active request gating variants Some of these either regressed performance or produced non-clean outputs such as `length:1` in offset tests, so we did not count them as valid benchmark results. ## Questions 1. Is it expected that Qwen native MTP in vLLM graph mode can be slower than no-MTP graph baseline even with MTP1 acceptance around `82%-88%`? 2. For `num_speculative_tokens > 1`, does vLLM currently run the same Qwen MTP layer multiple times / perform repeated per-token processing rather than producing multiple draft tokens in one low-overhead pass? 3. Are there known bottlenecks or planned optimizations for: - `reject_sample_kernel` - spec decode metadata preparation - slot mapping updates - `num_computed_tokens` correction/update - target/draft metadata reuse under CUDA graph 4. Is there a recommended benchmark setup for fairly comparing no-MTP graph baseline vs native MTP graph mode for long-output multimodal workloads? 5. Would the vLLM team be interested in a minimal repro using a public Qwen3.5 native-MTP checkpoint, if available, or local profiling traces from our private OCR checkpoint? ## Expected behavior For a native MTP model with `80%+` MTP1 acceptance and graph mode enabled, we would expect MTP1 to at least be competitive with no-MTP graph baseline, or for documentation to clarify that MTP may be slower in graph mode unless the acceptance rate and sequence/batch shape meet specific thresholds. For MTP2/MTP3, we would expect either: - a low-overhead multi-token draft path, or - guidance that `num_speculative_tokens > 1` may be slower for Qwen native MTP because it requires repeated MTP forward/processing. ## Actual behavior On this workload: - no-MTP graph baseline is still the fastest stable path. - MTP1 has good acceptance but is slower end to end. - MTP2 increases mean accepted length but introduces enough extra overhead that it still does not beat the best no-MTP graph baseline. - Local profiling points to rejection sampling, metadata preparation, slot mapping, and `num_computed_tokens` updates as major overheads. ## Additional context This benchmark is from a private OCR/document parsing workload, so some exact data/model paths are not public. We can prepare a smaller reproducer if a public Qwen native-MTP checkpoint is suitable. The most relevant local observation is that the bottleneck is not just draft quality. Even when acceptance is usable, the spec decode path can be too expensive relative to a strong CUDA graph no-MTP baseline.", "url": "https://github.com/vllm-project/vllm/issues/47277", "createdAt": "2026-07-01T09:23:25Z", "updatedAt": "2026-08-13T07:54:47Z", "timestamp": "2026-08-13T07:54:47Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [], "author": "JumpingRain", "state": "open", "assignees": [ "andakai" ] }, { "id": "github:vllm-project/vllm:issue:47522", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MLA chunked-context prefill crashes on sm80 with Marlin FP8: kv_c_normed cast to packed-int32 weight dtype (`unsupported \\`a\\` scalar_type`)", "text": "### Your current environment - vLLM 0.23.0 (official image), PyTorch 2.x, Triton, NCCL 2.28.9 - 12× NVIDIA A100-SXM4-80GB (compute capability **8.0**, no native FP8) across 3 nodes - Model: GLM-5.2-FP8 (fp8 weights → `MarlinFP8ScaledMMLinearKernel` / `MARLIN Fp8 MoE`), PP=3 × TP=4, `--enable-expert-parallel` - Attention: `TRITON_MLA` decode backend, `FLASH_ATTN` MLA prefill backend, bf16 KV cache - (We run with sparse attention disabled via a local patch because of #35021, but this bug is in the generic MLA chunked-context path and should reproduce with any fp8 MLA model on sm80, e.g. DeepSeek-V3/R1-FP8 on A100.) ### 🐛 Describe the bug On sm80 with fp8 weights (Marlin), any request that takes the **chunked-context prefill path** — i.e. a prefix-cache hit or a prompt longer than `max_num_batched_tokens` — crashes the worker with: ``` RuntimeError: unsupported `a` scalar_type ``` in `torch.ops._C.marlin_gemm`, called from `kv_b_proj` inside `MLACommonImpl._compute_prefill_context`. Requests that don't hit that path (short prompts, no shared prefix, plain decode) work fine, so the server passes startup, warmup, and simple smoke tests, then dies on the first prefix-cache hit in real traffic. **Root cause** — `vllm/model_executor/layers/attention/mla_attention.py`, in `_compute_prefill_context`: ```python _kv_b_proj_w_dtype = ( self.kv_b_proj.weight.dtype if hasattr(self.kv_b_proj, \"weight\") else self.kv_b_proj.params_dtype ) if ( use_fp8_prefill or _kv_b_proj_w_dtype != current_platform.fp8_dtype() ) and _kv_b_proj_w_dtype != torch.uint8: kv_c_normed = kv_c_normed.to(self.kv_b_proj.weight.dtype) ``` The intent is \"cast the gathered KV back to the layer's expected input dtype, except for quantized layouts\". But on sm80, `MarlinFP8ScaledMMLinearKernel.process_weights_after_loading` repacks the fp8 weight via `pack_fp8_to_int32` + `gptq_marlin_repack` and `replace_parameter(layer, \"weight\", marlin_qweight)` — so `kv_b_proj.weight.dtype` is **`torch.int32`** (packed), not `float8_e4m3fn`. The condition (`!= fp8_dtype()`, `!= uint8`) doesn't exclude it, so the bf16 activations are cast to `torch.int32` and `marlin_gemm` rejects them. On Hopper the weight stays fp8 so the cast is skipped and nothing breaks, which is presumably why this wasn't caught. ### Reproduction 1. Serve any fp8-quantized MLA model on an sm80 GPU (Marlin path), with prefix caching enabled (default). 2. Send one short request. It succeeds. 3. Send a second request sharing a token prefix with the first (or a single prompt longer than `max_num_batched_tokens`). 4. Worker crashes with the traceback below; engine core dies. ### Traceback (trimmed) ``` File \".../vllm/model_executor/layers/attention/mla_attention.py\", line 2300, in forward_mha context_output, context_lse = self._compute_prefill_context( File \".../vllm/model_executor/layers/attention/mla_attention.py\", line 2097, in _compute_prefill_context kv_nope = self.kv_b_proj(kv_c_normed)[0].view( File \".../vllm/model_executor/layers/linear.py\", line 582, in forward output_parallel = self.quant_method.apply(self, input_, bias) File \".../vllm/model_executor/layers/quantization/fp8.py\", line 474, in apply return self.fp8_linear.apply_weights(layer, x, bias) File \".../vllm/model_executor/kernels/linear/scaled_mm/marlin.py\", line 114, in apply_weights return apply_fp8_marlin_linear( File \".../vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py\", line 69, in apply_fp8_marlin_linear output = ops.marlin_gemm( RuntimeError: unsupported `a` scalar_type ``` ### Suggested fix Only perform the cast when the target dtype is a floating-point type — for packed integer weight layouts (Marlin int32, NVFP4 uint8, …) the quant method expects model-dtype input and quantizes internally, exactly like the normal (non-context) prefill path already does: ```python if ( (use_fp8_prefill or _kv_b_proj_w_dtype != current_platform.fp8_dtype()) and _kv_b_proj_w_dtype != torch.uint8 and _kv_b_proj_w_dtype.is_floating_point ): kv_c_normed = kv_c_normed.to(self.kv_b_proj.weight.dtype) ``` (The existing `!= torch.uint8` special case then becomes redundant.) We've been running this fix in production on 12×A100 and it resolves the crash: prefix-cache hits and 6.5K-token chunked prefills work, outputs are correct. Happy to send a PR if the approach looks right. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and checked the documentation and examples.", "url": "https://github.com/vllm-project/vllm/issues/47522", "createdAt": "2026-07-03T06:10:07Z", "updatedAt": "2026-08-13T14:19:27Z", "timestamp": "2026-08-13T14:19:27Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "biondogs", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:47691", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: `--data-parallel-start-rank 0` is silently treated as unset due to truthiness checks in `create_engine_config`", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... uv is set ============================== System Info ============================== OS : Linux Mint 22.1 (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-134-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.0.140 GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3050 Laptop GPU Nvidia driver version : 580.159.03 ============================== vLLM Info ============================== vLLM Version : 0.1.dev18413+g6569df6a3 (git sha: 6569df6a3) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled ```
Note: this bug is in pure Python config-derivation logic (`vllm/engine/arg_utils.py`) and is independent of GPU/driver/topology. It affects any multi-node **hybrid data-parallel load balancing** deployment (`docs/serving/data_parallel_deployment.md`, \"Hybrid Load Balancing\" section), regardless of hardware. ### 🐛 Describe the bug **Summary:** When a node explicitly passes `--data-parallel-start-rank 0`, `EngineArgs.create_engine_config` treats it identically to \"not specified,\" because the field is checked with Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None`. Since `0` is a perfectly valid and meaningful starting rank (the node owning the first slice of ranks), this silently corrupts the derived `data_parallel_hybrid_lb` flag for exactly that node. #### The two buggy lines `vllm/engine/arg_utils.py`, inside `create_engine_config`: ```python # line 1997 if self.data_parallel_start_rank and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True ``` ```python # line 2016 self.data_parallel_rank = ( self.data_parallel_start_rank or inferred_data_parallel_rank ) ``` In both cases, `self.data_parallel_start_rank == 0` is falsy, so: - Line 1997: the condition is `0 and not headless` → `False`, so `self.data_parallel_hybrid_lb` is **never inferred `True`** for this node, even though the deployment doc explicitly requires hybrid LB nodes to pass `--data-parallel-start-rank` so \"each node knows which ranks it owns\" (`docs/serving/data_parallel_deployment.md`, lines 87-91) — including the node that owns rank 0. - Line 2016: `0 or inferred_data_parallel_rank` evaluates to `inferred_data_parallel_rank`. This usually happens to still resolve to `0` in the common single-process-per-node case, but it means the explicit user-supplied value is discarded in favor of a derived one, which is fragile and wrong in intent even when the numeric result coincides. #### The correct pattern already exists 27 lines above, for the sibling field ```python # line 1969 — data_parallel_rank, handled correctly data_parallel_external_lb = ( self.data_parallel_external_lb or self.data_parallel_rank is not None ) ``` This is the established, correct convention in this exact function for exactly this class of field (`int | None`, where `0` is a valid value). The two other places in the codebase that read `data_parallel_start_rank` also use `is not None` correctly: - `vllm/entrypoints/cli/serve.py:86`: `is_hybrid_lb = args.data_parallel_start_rank is not None` - `vllm/entrypoints/openai/dp_supervisor.py:39-40`: `if start_rank is not None: return start_rank` Neither of those call sites compensates for the bug, though — `serve.py`'s `is_hybrid_lb` is a local variable used only for `--api-server-count` defaulting and LB-mode mutual-exclusivity checks; it is never written back to `args.data_parallel_hybrid_lb`. `dp_supervisor.py` is a separate code path (multi-port external LB) that never touches `data_parallel_hybrid_lb`. So `arg_utils.py:1997`/`2016` are the only place that derive `self.data_parallel_hybrid_lb` from this flag, and they get it wrong specifically for rank 0. #### Downstream impact chain The corrupted `data_parallel_hybrid_lb` flag propagates into `ParallelConfig.data_parallel_hybrid_lb` and is consumed in multiple places: 1. `vllm/config/parallel.py:536`, `ParallelConfig.local_engines_only`: ```python return self.data_parallel_external_lb or self.data_parallel_hybrid_lb ``` For the rank-0 node in a hybrid-LB deployment, this wrongly evaluates to `False` instead of `True`. 2. `vllm/v1/engine/utils.py:1244`, `wait_for_engine_startup`: ```python remote_should_be_headless = ( not parallel_config.data_parallel_hybrid_lb and not parallel_config.data_parallel_external_lb ) ``` Wrongly evaluates to `True` for that node, misclassifying the expected startup behavior of remote engines. 3. `vllm/entrypoints/cli/serve.py`, in the Rust-frontend startup path added by #45805 (merged 2026-06-17): ```python if parallel_config.local_engines_only: expected_engine_start_index = parallel_config.data_parallel_rank expected_engine_count = parallel_config.data_parallel_size_local else: expected_engine_start_index = 0 expected_engine_count = parallel_config.data_parallel_size ``` Because `local_engines_only` is wrongly `False` for the rank-0 node, the Rust frontend computes it as owning **all** global engines (`0`, `data_parallel_size`) instead of just its local slice (`data_parallel_rank`, `data_parallel_size_local`). None of these downstream consumers compensate for or work around the truthiness bug — they all trust `data_parallel_hybrid_lb` as computed in `arg_utils.py`. #### Steps to reproduce Set up a 2-node hybrid load-balancing deployment per `docs/serving/data_parallel_deployment.md` (\"Hybrid Load Balancing\"): ```bash # Node 0 (owns global ranks 0-1, i.e. the first slice) — non-headless, exposes its own API endpoint vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \\ --data-parallel-start-rank 0 \\ --data-parallel-address 192.0.2.1 --data-parallel-rpc-port 13345 # Node 1 (owns global ranks 2-3) vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 \\ --data-parallel-start-rank 2 \\ --data-parallel-address 192.0.2.1 --data-parallel-rpc-port 13345 ``` Node 0 (rank 0) ends up misclassified as not participating in hybrid LB while every other node in the same deployment is classified correctly — see the minimal reproduction below for the exact config values this produces. #### Minimal reproduction (no cluster required) The same underlying bug can be triggered directly against `EngineArgs.create_engine_config`, without standing up an actual multi-node deployment: ```python from vllm.engine.arg_utils import EngineArgs from vllm.usage.usage_lib import UsageContext engine_args = EngineArgs( model=\"facebook/opt-125m\", data_parallel_size=4, data_parallel_size_local=2, data_parallel_start_rank=0, ) vllm_config = engine_args.create_engine_config(UsageContext.OPENAI_API_SERVER) print(\"data_parallel_hybrid_lb:\", vllm_config.parallel_config.data_parallel_hybrid_lb) print(\"data_parallel_rank:\", vllm_config.parallel_config.data_parallel_rank) ``` Actual output on current `main` (commit `6569df6a3`): ``` data_parallel_hybrid_lb: False data_parallel_rank: 0 ``` Expected output (and what this snippet produces once `data_parallel_start_rank` is checked with `is not None` instead of truthiness): ``` data_parallel_hybrid_lb: True data_parallel_rank: 0 ``` ### Proposed fix Use `is not None` instead of truthiness at both sites, matching the existing convention used for the sibling field `data_parallel_rank` (line 1969) and the other two call sites of this same field (`serve.py:86`, `dp_supervisor.py:40`): ```python # line 1997 if self.data_parallel_start_rank is not None and not headless: self.data_parallel_hybrid_lb = True ``` ```python # line 2016 self.data_parallel_rank = ( self.data_parallel_start_rank if self.data_parallel_start_rank is not None else inferred_data_parallel_rank ) ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47691", "createdAt": "2026-07-06T05:39:54Z", "updatedAt": "2026-08-13T15:12:28Z", "timestamp": "2026-08-13T15:12:28Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "syedalijaseem", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:47761", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm 0.23.0 and 0.24.0 - Qwen3.6-35B-A3B-FP8 - Fails generating code- \"400 Unterminated string starting at\"", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.12.0-160000.35-default-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA A100-PCIE-40GB GPU 1: NVIDIA A100-PCIE-40GB Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 Stepping: 7 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities L1d cache: 1.3 MiB (40 instances) L1i cache: 1.3 MiB (40 instances) L2 cache: 40 MiB (40 instances) L3 cache: 55 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-9,40-49 NUMA node1 CPU(s): 10-19,50-59 NUMA node2 CPU(s): 20-29,60-69 NUMA node3 CPU(s): 30-39,70-79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Mitigation; TSX disabled Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.12 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cccl==13.3.3.3.1 [pip3] nvidia-cuda-crt==13.3.33 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.2.78 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cuda-tileiras==13.2.78 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.25.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.5.2 [pip3] nvidia-cutlass-dsl-libs-base==4.5.2 [pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.2.78 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.7.10.post20260531 [pip3] torch==2.11.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchvision==0.26.0+cu130 [pip3] transformers==5.12.1 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.24.0 vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX 20-29,60-69 2 N/A GPU1 PIX X 20-29,60-69 2 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility NCCL_DEBUG=INFO VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 NCCL_COMM_ID=127.0.0.1:12345 VLLM_ENABLE_CUDA_COMPATIBILITY=0 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 VLLM_NO_USAGE_STATS=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug When i request to the model to write some code more complex everything looks good at first, but during the code generation an exception occurs \"400 Unterminated string starting at: line 1 column 14 (char 13)\" ```exception [Extension Host] [@continuedev] \u001b[31merror\u001b[39m: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} logToConsole @ log.ts:117 console.ts:139 [Extension Host] [@continuedev] error: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} log @ console.ts:139 localProcessExtensionHost.ts:317 Extension Host localProcessExtensionHost.ts:318 [@continuedev] error: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} ``` At first i was thinking it was related to this issue [39055] (https://github.com/vllm-project/vllm/pull/39055) , but after updating to the last version i am still getting the same error. i am also sharing my docker-composer.yaml to make sure that this problem is not a configuration problem: ```yaml vllm-instruct: image: vllm/vllm-openai:v0.24.0 container_name: vllm-Qwen3.6-35B-A3B-FP8 restart: unless-stopped command: [ \"Qwen/Qwen3.6-35B-A3B-FP8\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--tensor-parallel-size\", \"2\", \"--gpu-memory-utilization\", \"0.90\", \"--max-model-len\", \"32768\", \"--max-num-batched-tokens\", \"32768\", \"--max-num-seqs\", \"64\", \"--enable-prefix-caching\", \"--kv-cache-dtype\", \"fp8\", \"--enable-chunked-prefill\", #\"--disable-custom-all-reduce\", \"--attention-backend\", \"FLASHINFER\", \"--enable-auto-tool-choice\", \"--tool-call-parser\", \"qwen3_coder\", \"--reasoning-parser\", \"qwen3\", ] ports: - \"8001:8000\" env_file: - .env environment: - VLLM_API_KEY=${VLLM_API_KEY} - HF_TOKEN=${HF_TOKEN} - VLLM_NO_USAGE_STATS=1 - HF_HUB_DISABLE_TELEMETRY=1 - NVIDIA_VISIBLE_DEVICES=all - NCCL_DEBUG=INFO - NCCL_COMM_ID=127.0.0.1:12345 - GLOO_SOCKET_IFNAME=lo - VLLM_MARLIN_USE_ATOMIC_ADD=1 ipc: host shm_size: '32gb' volumes: - hf_cache:/root/.cache/huggingface - vllm_cache:/root/.cache/vllm deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] security_opt: - no-new-privileges:true healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8000/health\"] interval: 30s timeout: 10s retries: 3 start_period: 240s (...) ``` Can you please help me? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47761", "createdAt": "2026-07-06T16:42:07Z", "updatedAt": "2026-08-12T17:46:25Z", "timestamp": "2026-08-12T17:46:25Z", "metrics": { "reactions": 0, "comments": 21 }, "labels": [ "bug" ], "author": "rnogueira-tech", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:47839", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Packed Variable Length Speculative Decoding", "text": "### Motivation. In speculative decoding, we can save verification cost by selectively verifying a different number of draft tokens per request in each step. There are various techniques of selecting and tuning which / how many tokens to submit for verification, but all share a common benefit: reducing verification overhead with per-request variable-length query lengths in a batch. ### Proposed Change. We will support verification input batches with heterogeneous query lengths for each decode request. We will separate the concerns of total verification token budget and allocation of that budget among the draft tokens. The scheduler chooses a host-visible total token budget, and the worker distributes that budget across requests entirely on-device, avoiding serialization of the query lengths to/from the host. This allows us to maintain asynchronous scheduling and CUDA graph compatibility. This will involve changes to the scheduler, model runner input preparation, cuda graph management, and speculator-scheduler implementations (for the actual logic to decide which tokens to propose). I propose to build off of the [batch-level Dynamic Speculative Decoding](https://github.com/vllm-project/vllm/pull/45953) support (which already has full-graph compatibility) in ModelRunnerV2. ### Feedback Period. _No response_ ### CC List. @TheEpicDolphin @WoosukKwon @LucasWilkinson @mgoin @ekagra-ranjan @simon-mo @zyongye ### Any Other Things. ## Challenges ### Varlen kernel support Attention kernels take as input the number of *requests* in addition to the query offset pointers. For decoding, many attention kernels omit the latter and simply assume either 1 (autoregressive) or N (uniform speculative) query lengths per request. Many, but not all, attention kernels already support variable length queries per request: - FlashAttention Decode (Hopper/Blackwell GQA): supports varlen queries via `flash_attn_varlen_func` arguments `max_seqlen_q` and `cu_seqlens_q`. (Definitely FA3, not sure about varlen full-graph for FA4) - FlashInfer TRTLLM-Gen Decode (Blackwell GQA): supports varlen queries via `max_q_len` and `cum_seq_lens_q`, but not plugged into vLLM yet - FlashInfer TRTLLM-Gen MLA Decode (Blackwell MLA): no support - FlashAttention MLA: Currently uniform-query CG compatibility, but supports varlen Q, might be compatible - DeepSeek V4 Indexed attention: Underlying Sparse MLA kernel supports varlen, but seems to need some plumbing to fully support varlen - MiniMax M3 Sparse attention: no support - FlashMLA (Hopper MLA): no support - GDN: seems possible with some extra work on the metadata builder - Mamba/Mamba2: seems possible with a bit of plumbing Thus we will have to gate support to only the attention backends which have varlen support implemented, and it'll take quite a bit of follow-up work to enable this for most backends. ### CUDA Graph Compatibility We currently have support for FULL CUDA Graphs with different query lengths for different batch sizes. This is done by keying the graphs by `num_reqs` _and_ `num_tokens` _and_ `uniform_token_count`. We can extend this: for each `num_reqs` (binned to powers of 2 / multiples of 8 / etc.) from `1` up to `max_num_seqs`, we record a separate set of graphs for each `num_query_tokens_per_request` from `1` (no spec tokens) up to (1 + max_num_speculative_tokens). To unlock true dynamism, we break the connection between `num_query_tokens_per_request` and `max_q_len`: we always permit up to `max_num_speculative_tokens` for a single request, a pass `max_q_len=max_num_speculative_tokens` in every attention kernel invocation, and thus allow some requests to have fewer than `num_query_tokens_per_request` and some to have more. This means that the scheduler effectively is only deciding `mean_num_query_tokens_per_request` (and will likely use heuristics from the workers to make that decision, see below). Then all we need to do is leverage `cu_seqlens_q` to have variable query start locations within the block, mark the remaining tokens in the suffix with `is_padding`, and pad `num_reqs` to `num_reqs_padded` by appending to `cu_seqlens_q` with 0-query-length requests which are ignored by the kernel. Thus, the attention kernel is, for a given `num_reqs_padded` and `num_tokens = int(mean_num_query_tokens_per_request * num_reqs_unpadded)`, always invoked with inputs of consistent shape and address. Note, that we can support non-integer `mean_num_query_tokens_per_request` as long as we record more graphs for `num_tokens` values in between `num_reqs * i` and `num_reqs * (i + 1)`, though they will not be perfectly uniform; we will have to support some extra padding or handle this specifically when doing capture. ### Supporting mixed batches Supporting mixed batches with FULL CUDA Graphs is appealing as host-overhead in short prefills is a real concern, but this is out-of-scope for this RFC. For PIECEWISE graphs, this is simpler. The main remaining consideration is compatibility with the `is_padding` optimization, which allows some kernels (e.g. MoE kernels) to skip work on graph-padding tokens. With variable query lengths in the decode, we may end up with some decode padding (to match the number of actual query tokens with the number of expected query tokens) in addition to graph padding. This should be avoided, either by packing the decodes tightly in the batch or reordering prefills and extends to the front of the batch, to share the query padding with the graph's total-token-count padding. It's unclear to me if it will be easy to implement the former, as the total number of used query tokens need not be known on the host, but either solution seems feasible here. The easiest short-term solution seems to be to **enforce no padding in the query block**. Since we always have more draft tokens than we will need to admit in our budget, we can just fill all allocated slots with draft tokens even if our performance model might indicate it's favorable to skip them. Even though they might be skippable via `is_padding`, there are still some attention and constant overheads to be paid, so the price shouldn't be too extreme here. ### Supporting Data Parallelism Some coordination across workers will be necessary to synchronize which graph is being executed, i.e. which (`num_reqs_padded` x `num_tokens_padded`) pair is active. Seems like this should fit naturally with our existing DP sync, but I'm not an expert in this area. ### Scheduler-Speculator awareness In vLLM, our scheduler lives in a separate process; yet I propose to let it decide `mean_num_query_tokens_per_request` when scheduling. In order to have the scheduler make decisions about how many total query tokens to permit, it will need some information from the model runner. I propose to extend the Speculator interface to allow the scheduler to \"query\" for \"hints\" such as how many query tokens should be issued on average per request. It is important to maintain asynchronous scheduling support here; in the DSpark report, they suggest using information delayed by 2 engine iterations in order to schedule the next step. This seems very reasonable, as acceptance rates over batches generally have significant temporal locality over multiple steps (reasoning v.s. final response, conversation domain, blocks of code/math/formatted output, etc). This would probably be easier to implement if the scheduler was replicated into each worker, so we don't need to cross an IPC boundary, but this is what we have today. I'm not certain that the scheduler needs to know about this at all. Conceptually I feel like it's the role of the scheduler to be choosing how many tokens to put in the batch, but besides allocating KV slots and selecting which requests to run (both of which are pretty agnostic to the variable-length decode tokens, the scheduler can just allocate the slots optimistically), I'm not sure why the current scheduler would need to be involved. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47839", "createdAt": "2026-07-07T08:04:55Z", "updatedAt": "2026-08-12T19:27:50Z", "timestamp": "2026-08-12T19:27:50Z", "metrics": { "reactions": 10, "comments": 2 }, "labels": [ "RFC", "dflash" ], "author": "benchislett", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:48197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: StructuredOutputManager x Speculative Decoding Refactor", "text": "### Motivation. Simplify the contract between `Scheduler`, grammar backend, and `StructuredOutputManager`, especially when speculative decoding is used. `StructuredOutputManager` exists because constrained decoding often doesn't kick in until a model has finished reasoning. With speculative decoding, grammar-invalid draft tokens need to be rejected, draft tokens need to temporarily advance grammar state to generate bitmasks, rejected draft tokens should not advance grammar state. Reasoning end boundary can also fall in the middle of the draft token window, so only draft tokens after said boundary should advance grammar state. `Scheduler` and the `StructuredOutputManager` need to do a careful dance to make this is all correct. This leads to some bugs such as [#34650](https://github.com/vllm-project/vllm/issues/34650) -- only recently fixed thanks to [#44297](https://github.com/vllm-project/vllm/pull/44297). ``` Per request inside a scheduler step 1. StructuredOutputManager.should_advance(request) # Should we constrain some draft tokens? Grammar.validate_tokens(draft tokens) # Reject grammar invalid draft tokens 2. StructuredOutputManager.grammar_bitmask(...) # Generate bitmasks for the draft tokens 3. Verifier runs 4. StructuredOutputManager.should_advance(request) # Should we constrain some draft token? StructuredOutputManager.trim_reasoning_for_advance(verified tokens) # Trim the draft tokens that are not constrained Grammar.accept_tokens(verified tokens) # Advance grammar state ``` In fact, speculative decoding x constrained decoding remains fragile / subtly incorrect: 1. ~~`StructuredOutputManager.should_advance(request)` treats STRUCTURAL_TAG vs other constraint types differently. It defers non STRUCTURAL_TAG constraints. [#44297](https://github.com/vllm-project/vllm/pull/44297) only fix the strict tool calling case (which uses STRUCTURAL_TAG); [#34650](https://github.com/vllm-project/vllm/issues/34650) is still unfixed for JSON, REGEX, and other constraint types.~~ Fixed by https://github.com/vllm-project/vllm/pull/44993 2. ~~`StructuredOutputManager.should_advance(request)` also doesn't answer \"should we constrain some draft tokens?\" reliably. It incorrectly infers the draft token window from `num_output_placeholders`. ~~ Fixed by https://github.com/vllm-project/vllm/pull/44993 3. `Grammar.validate_tokens(draft_tokens)` only works on a best effort basis -- it doesn't validate any draft token unless all draft tokens should be constrained. We rely on `StructuredOutputManager.grammar_bitmask(...)` and rejection sampling to discard grammer-invalid tokens. ### Proposed Change. Let `StructuredOutputManager` handle the token validation and grammar advancement. Remove: - `StructuredOutputManager.should_advance(request)` - `StructuredOutputManager.trim_reasoning_for_advance(verified tokens)` Add: - `StructuredOutputManager.validate_tokens(draft tokens)` - `StructuredOutputManager.accept_tokens(verified tokens)` ``` Per request inside a scheduler step 1. StructuredOutputManager.validate_tokens(draft tokens) 2. StructuredOutputManager.grammar_bitmask(...) 3. Verifier runs 4. StructuredOutputManager.accept_tokens(verified tokens) ``` Internally, `validate_tokens()`, `grammar_bitmask()`, and `accept_tokens()` all use the same `_get_constraint_start()` helper to determine whether and where draft tokens should be constrained and grammar state advanced. Concretly, `_get_constraint_start()` computes the first token that should be grammar-constrained. - `validate_tokens()` preserves any unconstrained prefix and validates only the constrained suffix. - `grammar_bitmask()` emits unconstrained bitmaks before the boundary and constrained bitmasks after it. - `accept_tokens()` advances the grammar state only on the constrained suffix only This gives a simpler mental model for the system. The scheduler is responsible for request lifecycle and token plumbing. `StructuredOutputManager` is responsible for draft token validation, bitmask generation, and grammar backends. ~100 insertions, ~200 deletions of changes excluding test updates. Would fix issues like: #34650 and https://github.com/vllm-project/vllm/issues/43388 ### CC List. @njhill @mgoin @chaunceyjiang @benchislett @arpera ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/48197", "createdAt": "2026-07-10T02:13:01Z", "updatedAt": "2026-08-13T14:22:20Z", "timestamp": "2026-08-13T14:22:20Z", "metrics": { "reactions": 1, "comments": 5 }, "labels": [ "RFC" ], "author": "yzong-rh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:48435", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug/Perf]: hybrid-SWA prefix caching collapses to zero for ALL requests in multi-session round-robin at ~25% pool occupancy (Gemma-4-31B; eager-freed SWA tails recycled tail-first)", "text": "## Summary For hybrid-SWA models (tested: Gemma-4-31B, 10 full-attention / 50 sliding-window(1024) layers), **cross-request prefix-cache reuse collapses to exactly zero for every request in a multi-session round-robin workload once the combined working set exceeds a sharp threshold far below pool capacity** (~34–38k tokens vs a 153k-token pool — ~25% occupancy). Reuse is total-or-nothing, never partial. Reproduced on v0.24.0 and v0.25.0. A GDN-hybrid control (Qwen3.5-27B, align mode) on the same versions/hardware shows no such collapse, and a full-attention-only workload pattern (single session re-fired) gets 99.96% hits at 64k — the failure is specific to the SWA path under cross-session interleaving. ## Environment - vLLM v0.24.0 and v0.25.0 (`vllm/vllm-openai` images), 1× RTX 5090 32GB, driver 580.x - `vllm serve google/gemma-4-31B-it-qat-w4a16-ct --tool-call-parser gemma4 --enable-auto-tool-choice --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 --gpu-memory-utilization 0.93` - Log: `GPU KV cache size: 153,374 tokens` ## Reproduction N independent chat sessions (\"seats\") with distinct prompts, fired round-robin; each seat re-sent with an identical prefix + a short new suffix each round (append-only, the agentic-session shape). Measure warm-round TTFT vs cold. | interleaved shape (measured prompt tokens) | combined | warm TTFT | |---|---|---| | single 64k seat, immediate re-fire | 64k | 1.3 s (99.96% hit) | | 2 × 17.1k | 34.3k | 0.7–1.2 s — resident | | 2 × 19.3k | 38.6k | **7.3 s = cold. zero reuse** | | 2 × 22k | 44k | cold every round | | 64k + 3 × 5k | 79k | **ALL seats cold — the large seat destroys even the 5k seats' reuse** | | 58k + 4 × 20k | 152k | cold every round | Zero **partial** hits in the collapsed regime (`prefix_cache_hits_total` stays flat), not degraded hit rates. Threshold is insensitive to per-seat size, seat count, and chunked-prefill size; reproduced at 25% pool occupancy. ## Analysis (from v0.25.0 source) The collapse appears to be an emergent interaction of four individually-reasonable behaviors: 1. A request's cache hit is the **intersection** across KV-cache groups; for the SWA group, `SlidingWindowManager.find_longest_cache_hit` requires `cdiv(window−1, block_size)` **contiguous** cached blocks at the tail — all-or-nothing. 2. SWA groups only ever hash boundary-tail blocks (`reachable_block_mask` sparse-hit semantics) — so there is nothing partial to hit. 3. Out-of-window SWA blocks are freed **eagerly mid-request** into the shared free queue. 4. `FreeKVCacheBlockQueue` evicts **tail blocks first** (right for full-attention shared-prefix workloads, but the tail-window blocks are precisely the only blocks an SWA hit needs). Under round-robin, other sessions' allocations recycle a session's SWA tail blocks between its turns; losing one zeroes the whole request's hit via (1), full-attention blocks included. Below the threshold, allocation demand is satisfied without touching hashed tails; above it, every seat's tails are recycled every rotation — hence the sharp cliff. ## Workaround that confirms the mechanism `OffloadingConnector` with a CPU tier (`--kv-transfer-config ... cpu_bytes_to_use=48GiB+`) fully restores residency: the same 152k five-session shape runs warm (TTFT 0.4–0.6 s vs 27 s cold) with sub-second turns. `--disable-hybrid-kv-cache-manager` is not viable at this scale (uniform pool drops below one 64k request on 32GB). Fresh-serve tier bisect: 32 GiB insufficient, 48 GiB sufficient for the 152k shape. ## Ask Either (a) SWA-aware eviction priority — don't evict the only blocks that can serve an SWA-group hit ahead of blocks that can't, or (b) an explicit block-pinning mechanism (#23083 would cover this). Happy to provide the battery script / raw per-round TTFT JSONs, and to re-run shapes on request. *Disclosure per the project's AI-contributions norms: this investigation and report were AI-assisted (Claude). All measurements are from real runs on rented hardware, the source analysis was human-directed, and the account owner stands behind the report and will respond to follow-ups.*", "url": "https://github.com/vllm-project/vllm/issues/48435", "timestamp": "2026-08-12T16:57:03Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [], "author": "claudematttest-dev", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:48494", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] num_speculative_tokens_per_batch_size + MTP speculator fails full CUDA graph decode capture (InputBatch.make_dummy assert)", "text": "### Environment - vLLM v0.25.0 (official docker image), single H100 NVL 94GB (SM90) - Target: `prithivMLmods/gemma-4-31B-it-qat-FP8` (compressed-tensors FP8) - Drafter: `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` (Gemma4 MTP, 4 layers: 3 sliding + 1 full attention) - `--kv-cache-dtype fp8 --async-scheduling --attention-backend TRITON_ATTN` (backend forced due to a separate SM90/SWA selection issue, filed separately) - `VLLM_USE_V2_MODEL_RUNNER=1` (per the new hint in the downgrade warning, to enable DSD × full CUDA graphs from #45953) ### Repro ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /models \\ --speculative-config '{\"model\":\"/drafter\",\"method\":\"mtp\",\"num_speculative_tokens\":3, \"num_speculative_tokens_per_batch_size\":[[1,64,3],[65,128,1],[129,512,0]]}' \\ --kv-cache-dtype fp8 --async-scheduling --attention-backend TRITON_ATTN ``` The env opt-in follows the hint added in 0.25's downgrade warning: *\"Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs.\"* Without it, DSD falls back to PIECEWISE and does not reach this capture path. Startup proceeds through weight load, torch.compile, target-model graph capture (`Capturing CUDA graphs (FULL): 150/150` OK), speculator prefill capture OK, then crashes at speculator **decode** FULL capture: ``` File \".../vllm/v1/worker/gpu/spec_decode/autoregressive/cudagraph_utils.py\", line 83, in create_forward_fn attn_state = prepare_inputs_to_capture( File \".../vllm/v1/worker/gpu/cudagraph_utils.py\", line 604, in prepare_inputs_to_capture input_batch = InputBatch.make_dummy(num_reqs, num_tokens, input_buffers) File \".../vllm/v1/worker/gpu/input_batch.py\", line 109, in make_dummy assert 0 < num_reqs <= num_tokens AssertionError ``` ### Isolation matrix (3 configurations, same stack otherwise) | speculative-config | Result | |---|---| | DSD table `[[1,64,3],[65,128,1],[129,512,0]]` | crash at ~1% of speculator decode capture | | DSD table `[[1,64,3],[65,512,1]]` (no K=0 tier) | crash immediately at speculator decode capture | | static `num_speculative_tokens: 3` (no table) | **capture completes** (`Graph capturing finished`) | → The **presence of the batch-size table** is the trigger; K=0 tiers are not required. ### Candidate root cause (hypothesis, file:line trail) `vllm/v1/worker/gpu/cudagraph_utils.py:205-225` reconstructs per-descriptor decode query lens as `num_new_sampled_tokens_per_step = decode_query_len − num_speculative_tokens`, an invariant written for the **target** manager (`decode_query_len = K + newly sampled`). The autoregressive speculator's capture manager (`spec_decode/autoregressive/cudagraph_utils.py:59-105`) inherits this base, but MTP drafting has different decode-query semantics (1 token/seq/step), so the derived `num_tokens` collapses to ≈K while `num_reqs` is the capture batch size — matching the observed failure boundary (K=3 table fails from bs=4, K=1 table from bs=2: `num_reqs > num_tokens`). EAGLE presumably passes because its execution shape satisfies the invariant. ### Notes - The DSD docs state the feature is \"only tested with Eagle and Eagle-3\"; this report adds the MTP data point. MTP + DSD works on v0.24.0 (runtime K switching across tiers verified with spec-decode counters), but v0.24.0 forces `cudagraph_mode=PIECEWISE` for DSD, so #45953's full-CG path is the reason to move to 0.25 — and this capture bug currently blocks that move for MTP. - Perf context (v0.24.0, informational, related to #47277): at client concurrency 256 we see DSD mode incur a 12–25% throughput penalty vs no-spec (oscillating run-to-run) with TTFT p50 inflating ~6×, **even with an all-K=0 table producing near-zero draft tokens**. We isolated it away from the cudagraph downgrade (forcing `cudagraph_mode=PIECEWISE` on a no-spec config costs ≈0% at the same concurrency) — the overhead appears to live in the DSD/spec-decode path itself at high batch, matching the \"per-step overhead erases the benefit\" pattern reported in #47277 on a different stack (Qwen3.5 native MTP / L20X). We can post the isolation trace on that thread once this capture blocker is resolved and we can re-run the comparison on 0.25. - Workaround today: static `num_speculative_tokens` (on 0.25 the full-CG capture passes; note startup then needs extra gpu-memory-utilization margin — we hit a warmup OOM at 0.95 on a 94GB H100), or DSD on 0.24 with the PIECEWISE cost.", "url": "https://github.com/vllm-project/vllm/issues/48494", "createdAt": "2026-07-13T13:32:16Z", "updatedAt": "2026-08-13T07:57:29Z", "timestamp": "2026-08-13T07:57:29Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:48495", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] Drafter attention backend auto-selection picks FlashInfer on SM90 for a sliding-window drafter, hitting the guard crash", "text": "### Environment Same as #48494 (v0.25.0, single H100 NVL 94GB SM90, Gemma4 MTP drafter with 3 sliding-attention layers + 1 full-attention layer), **without** `--attention-backend`. ### Behavior - Target model backend selection: `Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN']` — fine (heterogeneous head dims exclude others). - Drafter selection: `Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN']` — then engine init crashes: ``` File \".../vllm/v1/attention/backends/flashinfer.py\", line 757, in __init__ raise NotImplementedError( NotImplementedError: FlashInfer backend on SM90 currently crashes with sliding-window attention layers. Use the default attention backend. ``` ### Expected The auto-selector should exclude FlashInfer for a draft model containing sliding-window layers on SM90, instead of selecting it and failing at `create_metadata_builders`. The constraint is already documented in-code — the guard at `flashinfer.py:757` cites `flashinfer-ai/flashinfer#3578`; the selector just doesn't consult it. This is not Gemma-specific: any drafter with sliding-window layers on SM90 will hit the same path. ### Workaround `--attention-backend TRITON_ATTN` (forces both target and drafter). Note for migrators: the `VLLM_ATTENTION_BACKEND` env var is removed in 0.25 (logged as *Unknown vLLM environment variable*) — the CLI flag is the only path now.", "url": "https://github.com/vllm-project/vllm/issues/48495", "createdAt": "2026-07-13T13:32:33Z", "updatedAt": "2026-08-13T07:57:31Z", "timestamp": "2026-08-13T07:57:31Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:48627", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Context-length-aware speculative token scheduling — extending num_speculative_tokens_per_batch_size with a context-length axis", "text": "## Summary `num_speculative_tokens_per_batch_size` schedules the speculative depth K by batch size. We propose extending each entry with an optional, backward-compatible **context-length range**, so the runtime picks K from a `(batch, ctx)` table instead of a batch-only table: ```jsonc // today (unchanged, still valid): \"num_speculative_tokens_per_batch_size\": [[1,64,3],[65,128,1],[129,512,0]] // proposed (5-tuple entries; 3-tuple entries mean \"all context lengths\"): \"num_speculative_tokens_per_batch_size\": [ [1, 64, 0, 32768, 3], [65, 256, 0, 768, 0], // short-context high batch: spec off [65, 256, 769, 32768, 3], // long-context high batch: spec back on [257, 512, 0, 32768, 0] ] ``` *(Both tables are illustrative of the API shape, not prescriptive; K values are deployment-specific — see §Evidence for the measured Gemma-4-31B / H100 numbers this shape is drawn from.)* The scheduler change is one lookup dimension; per-K CUDA graphs and buffers are unaffected (they are keyed by K, not by table shape), so the extension is resource-neutral. ## Motivation **The batch-only rule \"speculation stops paying at high batch\" is a short-context artifact, not a general property.** We measured this on Gemma-4-31B (FP8, hybrid sliding+global attention) with its MTP drafter on a single H100 NVL 94GB, vLLM 0.23/0.24, greedy, fixed output length, 120s steady-state windows with Prometheus counter cross-checks. **1. Short-context crossover (the motivation for today's batch table) — reproduced:** | client concurrency | K=0 (tok/s) | best K (tok/s) | gain | |---|---|---|---| | 30 | 1,400 | 2,500 (K=3) | 1.79× | | 60 | 2,200 | 3,000 (K=3) | 1.36× | | 128 | 3,413 | 3,413 | 1.00× (converged) | **2. Same high batch, growing context (prefix-cache-hit regime, ~98% APC hit, prefill amortized) — the gain returns and grows:** | decode-time ctx (tok) | K=0 (tok/s) | K=3 (tok/s) | gain | K=0 TPOT | K=3 TPOT | |---|---|---|---|---|---| | ~460 | 3,107 | 3,640 | **1.17×** | 81.0 ms | 62.9 ms | | ~970 | 2,320 | 2,897 | **1.25×** | 109.3 ms | 77.8 ms | | ~1,990 | 2,110 | 2,915 | **1.38×** | 119.9 ms | 78.0 ms | | ~4,096 | 1,768 | 2,397 | 1.36× [1] | 137.8 ms | 96.9 ms [2] | (client concurrency 256 fixed; at concurrency 192 the ctx≈2k cell gives 1.45×.) [1] The 4k throughput ratio is a 3rd-run warm value; APC cache accumulation nudged it 1.28→1.32→1.36 run-to-run, so 1.36× is a conservative floor. The TPOT ratio (~1.4×, stable across 3 runs) is the more robust estimate. [2] 4k is a single point; we characterize it as *onset of decline*, not a decline curve — mapping ctx > 2k needs more points. The mechanism is visible in the TPOT column: as ctx doubles (970 → 1,990), **K=3 TPOT stays flat (77.8 → 78.0 ms) while K=0 TPOT keeps climbing (109 → 120 ms)**. Long-context decode is memory-bandwidth-bound on the per-step KV read; verifying K drafted tokens amortizes that read across K+1 tokens. The saving grows with ctx — but not without bound. **The gain peaks around ctx ≈ 1.5–2k (1.38× throughput / 1.54× TPOT) and begins to recede by 4k (1.36× / ~1.42×), as K=3's own per-step cost starts rising past 2k (its flat 78 ms breaks to 97 ms), narrowing the gap to K=0.** (We did not decompose that rise; candidates include the drafter's own long-context decode cost — SWA window growth, draft-layer FLOPs — and target-side KV growth.) The practical reading: speculation's sweet spot is the mid-to-long band (a few hundred to ~2k tokens); in the very-long regime the amortization gain converges to a ceiling rather than growing indefinitely. This is a stronger claim than monotone growth — it is bounded, mechanistic, and does not invite the \"then why not always speculate at long ctx\" objection. **Consequence: optimal K is a function of `(batch, ctx)` and is not separable.** A batch-only table forces one K per batch tier: - If the tier says K=0 (tuned for short-context saturation), long-context traffic at that batch loses a measured 1.2–1.45×. - If the tier says K>0 (tuned for long-context), short-context traffic at that batch pays verify overhead for no gain (and TTFT tail inflation). This matters most for exactly the workloads the ecosystem is optimizing for — agentic / RAG / multi-turn traffic with long, prefix-shared contexts at high concurrency. Related discussion where a member raised the \"MTP may only help for small batch size\" rule and we posted this datapoint: #47277. A secondary observation reinforces that even the *batch* axis is currently scheduled coarsely: on the same stack, short-context K=3 stays optimal further up the batch range than a typical hand-tuned table assumes — at 110 scheduled requests K=3 (3,300 tok/s) still beats K=1 (3,185) with a healthy TTFT p99 (796 ms), only converging to K=0 at the ~128 crossover. A static per-batch table with a conservative middle tier (e.g. dropping to K=1 at 65) leaves throughput on the table. This is orthogonal to the ctx axis but points the same way: the optimal-K surface is finer than a coarse batch-only table captures, along both axes. ## Proposed Change **1. Schema (`vllm/config/speculative.py`, `v1/spec_decode/dynamic/utils.py`):** accept 5-tuple entries `(bs_lo, bs_hi, ctx_lo, ctx_hi, K)` alongside today's 3-tuples (interpreted as `ctx_lo=0, ctx_hi=max_model_len`). Validation extends the existing rules: inclusive ranges, bs coverage from 1, non-overlapping, and per-bs-range full ctx coverage (rectangular grid). **2. Runtime lookup (`v1/core/sched/scheduler.py`):** today the scheduler does `dynamic_sd_lookup[len(num_scheduled_tokens)]`. We extend the dense lookup to two dimensions: `dense[B][ctx_bucket]`, where the batch's context representative is the **p50 of decode-time sequence lengths of the scheduled requests** — information the scheduler already holds, so the new signal costs nothing. `SchedulerOutput.num_spec_tokens_to_schedule` is unchanged (still a scalar per step). **3. Resource neutrality:** CUDA graphs and runtime buffers are keyed by the K values appearing in the schedule, not by the number of table cells (this is already how the per-K capture works). A 2D table with palette {0,1,3} captures exactly the same graphs as a 1D table with the same palette. **The context axis is resource-free.** **4. Docs:** two clarifications we found necessary in practice: - The table index is the **per-step scheduled request count**, not client concurrency; it fluctuates through admission ramps, so tier boundaries near a workload's steady running level leak (~2% draft volume in our measurement). Guidance: place boundaries outside the running-distribution tail. - Because the amortization gain saturates around ~2k (see Motivation), ctx buckets need not be fine-grained in the long tail: a single bucket covering \"≳2k\" is sufficient in our data, keeping the table small. - K=0 tiers do not currently make speculation free (drafter obligations persist; see the overhead reports in #47277 and our isolation summary there). In our measurement the penalty concentrates in TTFT (prefill/scheduling: p50 0.28s → 2.0s) rather than TPOT (+6.7%), consistent with the input-preparation hotspots reported in #47277. That is an orthogonal issue, but users sizing K=0 tiers for high-batch production should know it. ## Evidence & reproducibility Full tables, the measurement protocol (120s windows / 30s warmup / generated-token deltas / spec-counter cross-checks), harness, and an engine-agnostic reference controller (declarative `(B, ctx)` table + acceptance-rate correction + overload override) are public: **https://github.com/seongyun1104/depthchart**. MTP × DSD runtime tier switching on 0.24.0 was verified with spec-decode counters (c30 → K=3 with drafts/step ≈ 3.0; c400 → K=0 with zero drafts); the DSD docs currently note testing with Eagle/E3 only, so this doubles as an MTP datapoint (capture-path bug on 0.25 filed as #48494, backend-selection issue as #48495). ## Alternatives considered - **Acceptance-driven adaptation only** (SGLang `--speculative-adaptive` style): reacts to drafter quality but not to the regime. Two structural limits: at K=0 the acceptance signal vanishes and requires periodic probing to escape; and acceptance does not encode the ctx-dependent verify economics at all (our AR was flat 86/66/50 per position across c=30→128 while the gain moved from 1.79× to 1.00×). A `(B, ctx)` table's inputs never vanish; acceptance works better as a *correction* layer on top. - **Per-sequence K**: strictly more expressive, but requires variable-K verify batching and straggler control (cf. DSDE's per-sequence SL with a cap). The per-batch 2D table is the minimal change that captures the measured effect; per-sequence can layer on later. - **Entropy/complexity signals** (HeteroSpec-style): orthogonal — they estimate *acceptance*, not the *verify-side KV-read economics* that the ctx axis captures. (Note: HeteroSpec's *contextual* refers to language / semantic complexity of the input, not context length; the two signals target different phenomena and are compatible layers rather than substitutes.) ## Composition with concurrent adaptive-verification work An adaptive-verification design is being iterated in the DSpark track — #48692 (open, actively iterated; per its author on 2026-07-15, *\"Iterated design should land somewhere between this PR and #47808\"*). Its body is explicit that this layer does not schedule its own verification budget: it consumes the user-provided `num_speculative_tokens_per_batch_size` as the per-batch-size pool of draft tokens to verify. This RFC extends exactly that surface — existing 3-tuple entries normalize to a single full-range ctx bucket, so consumers that read the schedule (including `SpeculativeConfig.uses_dynamic_speculative_decoding()` gating already in main) see no interface change. The two designs are then strictly compositional layers: the DSpark track varies **per-request K within a budget cell** (spend more of the budget where confidence is high); this RFC defines **the per-(bs, ctx) budget cell itself**. ## Policy ladder The `(bs, ctx)` table this RFC adds is the base rung of a three-rung policy ladder for choosing K; the higher rungs remain open surface for follow-up work and other tracks. 1. **Declarative table (this RFC)** — a zero-lag prior over K derived from deployment characterization. Tier-boundary transitions incur no learning delay because the lookup is a pure function of `(bs, ctx)`; the moment the batch composition crosses a boundary, the new K applies. This is the adaptation-latency argument for keeping a table as the base layer even under adaptive strategies: online estimators (per-bucket AR, hybrid cost/acceptance models) have inherent settling time on regime shifts, whereas the table's response is O(1) lookup. 2. **Per-bucket acceptance correction** — table K modified by the running acceptance rate observed in each `(bs, ctx)` cell. Handles drafter quality drift that a static table cannot; DSDE and SGLang `--speculative-adaptive` are single-axis instances of this rung. 3. **Hybrid cost-model × acceptance** — a full verify-cost model composed with per-bucket acceptance to select K online (cf. the DSpark confidence-scheduled verification track in #47808). Requires the base table as its bootstrap prior — the cost model has to know where to look for signal before it can improve on the prior. This RFC defines rung 1. #48692 / #47808 iterate rung 3 and (as noted in the Composition section above) consume the schedule from rung 1 as the per-batch-size budget pool for their per-request budget scheduling. Rung 2 is the natural intermediate; a first-class implementation is out of scope here but the schedule surface accommodates it (`num_speculative_tokens` override at request time on top of the table lookup). ## Prior art & scope honesty Batch-size-conditioned K is established (this feature; SGLang ships batch-tiered candidate sets by default). \"Context-aware speculative decoding\" (CASD, 2024) names a different technique — drafting by retrieval *from* the context. ~~We have not found prior art that uses **context length as a first-class scheduling axis for K**, nor measurements of the `(batch, ctx)` interaction; we would welcome pointers if they exist.~~ **Update (2026-07-27):** The core observation — that optimal K should depend on sequence length under memory-bound decode — was formalized by **MagicDec (Sadhukhan et al., arXiv:2408.11049, 2024)**: *\"Leveraging our observation that there is a sequence length threshold above which decoding becomes memory bound, we propose choosing the amount of speculation as a function of the sequence length (longer sequence length → more speculated tokens).\"* MagicDec's instantiation was self-speculation with StreamingLLM sparse KV in a batch-centric framing; this RFC extends the general draft-target DSD lookup surface with an explicit ctx-length axis. See the [MagicDec anchor comment below](#issuecomment-5091701498) for the framing detail. **TurboSpec / SmartSpec lineage:** The dynamic-schedule surface this RFC extends was formalized as SmartSpec (Liu et al., [arXiv:2406.14066](https://arxiv.org/abs/2406.14066), later retitled TurboSpec) — the batch-adaptive predecessor of the `num_speculative_tokens_per_batch_size` API landed via #5886 / #32374. TurboSpec's evaluation focuses on the ≤1k-context regime where the crossover we observe at ctx≈2k sits to their right; their reported gains and the 1.29-1.36× we measure at ctx 900/1900/4000 are adjacent regions of the same K-vs-(B, ctx) surface rather than competing claims. The ctx-length axis this RFC adds makes both regions addressable from one declarative schedule. Known limitations of our data: one model/drafter pair (Gemma-4-31B FP8 + QAT-matched MTP head — drafter lineage alone moved acceptance 51.6% → 67.7% in our A/B, so table *values* are deployment-specific and need calibration; the *shape* of the surface is what this RFC relies on); the ctx sweep is from a prefix-cache-hit regime (miss-heavy traffic unmeasured); single GPU, TP=1. ## Implementation We are happy to contribute the patch: the change is contained (schema validation + dense-builder in `dynamic/utils.py`, one lookup site in `scheduler.py`, docs), backward compatible, and covered by the validation rules above. A second-engine replication of the mechanism (SGLang `_route` extension) is in progress on our side and can inform the design review. ## Feedback Period 2 weeks, or as maintainers prefer. ## CC @ekagra-ranjan @benchislett @luccafong @MatthewBonanni @JumpingRain", "url": "https://github.com/vllm-project/vllm/issues/48627", "createdAt": "2026-07-14T15:28:22Z", "updatedAt": "2026-08-13T07:54:53Z", "timestamp": "2026-08-13T07:54:53Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:49012", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: nvfp4 reshape_and_cache_flash assumes NHD layout — silently mis-swizzles HND caches when num_kv_heads % 4 == 0", "text": "### Environment ``` GPU: RTX 5090 (SM120, 32 GB) | driver 595.71.05, CUDA 13.2 vLLM 0.25.1 | torch 2.11.0 ``` ### Bug `reshape_and_cache_nvfp4_dispatch` (`csrc/.../nvfp4_kv_cache_kernels.cu:210`) reads `block_size` from tensor dim-1, which is only correct for NHD caches. With an HND cache it picks up `num_kv_heads` instead: - if `num_kv_heads % 4 != 0` you get a loud error (\"block_size must be divisible by 4 for NVFP4 KV cache swizzle\") - confusing but at least visible - if `num_kv_heads % 4 == 0` (e.g. 8 KV heads) it swizzles across the wrong dimension and produces silently corrupt cache contents Repro is a ~30-line script: write random K/V through the op with an HND-shaped cache, dequantize, compare against the input. We hit this via `VLLM_KV_CACHE_LAYOUT=HND` on SM120 but the kernel behavior is layout-dependent regardless of arch. ### Good In NHD the kernel is excellent - we byte-compared its output (data + both scale-factor tensors) against flashinfer's `nvfp4_quantize_paged_kv_cache` on identical input and got 100% identical results, including with non-unit global scales (vLLM's `k_scale` convention lines up with `1/global_sf` on the flashinfer side). So this is purely a layout-indexing gap: either add HND indexing or reject non-NHD caches explicitly. Found while prototyping nvfp4-KV on SM120 (#49011).", "url": "https://github.com/vllm-project/vllm/issues/49012", "createdAt": "2026-07-18T04:24:00Z", "updatedAt": "2026-08-12T18:58:02Z", "timestamp": "2026-08-12T18:58:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "0xdespot", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:49210", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Engine core livelock (100% CPU, no crash) with MTP speculative decoding + xgrammar structured outputs - regression from v0.24.0", "text": "## Your current environment ``` vLLM version: 0.25.1 (pip install, venv) xgrammar version: 0.2.1 PyTorch: 2.11.0+cu130 CUDA: 13.0 GPU: 1x NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) Driver: 580.159.03 OS: Ubuntu 25.10, kernel 6.17.0-40-generic Model: nvidia/Qwen3.6-27B-NVFP4 (compressed-tensors, NVFP4 weights, FP8 KV cache) ``` Full serve command: ``` vllm serve /path/to/nvidia--Qwen3.6-27B-NVFP4 \\ --gpu-memory-utilization 0.92 \\ --served-model-name local_qwen \\ --tensor-parallel-size 1 \\ --max-model-len 131072 \\ --max-num-seqs 256 \\ --max-num-batched-tokens 32768 \\ --long-prefill-token-threshold 4096 \\ --enable-prefix-caching \\ --renderer-num-workers 4 \\ --mm-processor-cache-gb 0 \\ --trust-remote-code \\ --quantization compressed-tensors \\ --dtype bfloat16 \\ --kv-cache-dtype fp8 \\ --calculate-kv-scales \\ --limit-mm-per-prompt '{\"image\":20}' \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_xml \\ --default-chat-template-kwargs '{\"enable_thinking\": false}' \\ --scheduling-policy priority \\ --speculative-config '{\"method\":\"qwen3_next_mtp\",\"num_speculative_tokens\":3}' \\ --structured-outputs-config '{\"backend\": \"xgrammar\", \"disable_any_whitespace\": true}' ``` ## 🐛 Describe the bug Under sustained concurrent load with structured outputs (JSON schema / tool calling), the engine core enters a **silent livelock**: the `VLLM::EngineCore` main thread spins at 100% CPU indefinitely, the scheduler never advances, and the API server keeps accepting requests that never return. There is **no crash, no `EngineDeadError`, no watchdog trigger** - the service stays `active (running)` while being completely unresponsive. In our case it stayed in this state for 8.5 hours until we noticed and restarted manually. This does **not** reproduce on v0.24.0 with a byte-identical configuration (same model, same flags, same xgrammar 0.2.1, same load) — details below. ### Timeline (from journald) Engine start 07:48. For ~25 minutes everything works, but xgrammar logs a steady stream of rollback warnings - **652 occurrences, every single one for token id 198** (the `\\n` token in the Qwen tokenizer): ``` [08:11:36] /project/cpp/grammar_matcher.cc:612: Warning: The matcher has terminated after accepting the stop token, but is trying to accept new token with id 198. ``` At 08:12:25, ten seconds before the hang, the spec decoding metrics report a **draft acceptance rate above 100%**, i.e. more accepted than drafted tokens — which should be impossible and suggests accounting is already corrupted at this point: ``` 08:12:25 SpecDecoding metrics: Mean acceptance length: 1.98, Accepted throughput: 5.00 tokens/s, Drafted throughput: 4.80 tokens/s, Accepted: 50 tokens, Drafted: 48 tokens, Per-position acceptance rate: 0.353, 0.333, 0.294, Avg Draft acceptance rate: 104.2% ``` One more normal stats interval, then the last line the engine ever logs: ``` 08:12:35 Engine 000: Avg prompt throughput: 1545.3 tokens/s, Avg generation throughput: 440.0 tokens/s, Running: 3 reqs, Waiting: 1 reqs, Deferred: 1 reqs, GPU KV cache usage: 4.3% 08:12:45 Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 3 reqs, Waiting: 1 reqs, Deferred: 1 reqs, GPU KV cache usage: 4.3% ``` After that: zero log output of any kind (checked 8.5 hours later - nothing, not even the 10s stats logger). ### Process state during the hang ``` $ ps -p -o pid,stat,etime,pcpu,comm PID STAT ELAPSED %CPU COMMAND 3213241 Rl 08:51:56 101 VLLM::EngineCore ``` Main thread in state `R` at ~100% CPU the whole time (8h51m accumulated CPU), all other threads sleeping. `nvidia-smi` reports 100% GPU utilization and full memory allocation held by the process, consistent with a busy-wait / spin rather than real work. The API server process stays healthy: it accepts connections and requests, which then never complete (a trivial 5-token completion runs into any client timeout). ### Control experiment: identical config on v0.24.0 is stable We run a second machine with the **byte-identical** serve command (same model snapshot, same flags including the same `--speculative-config` and `--structured-outputs-config`, two instances on two GPUs) on **vLLM 0.24.0 with the same xgrammar 0.2.1**, serving the same workload: - zero `grammar_matcher.cc:612` warnings in 7 days of journald logs, - no hangs, uptime spanning multiple days of sustained traffic. So with the variables isolated: same model, same config, same xgrammar version, same workload - v0.24.0 never triggers the rollback warning; v0.25.1 floods it and livelocks within ~25 minutes. This looks like a regression introduced with the v0.25 execution-path changes (Model Runner V2 default / reworked spec decoding), not an xgrammar version issue. ### Root cause hypothesis The symptoms match the FSM state corruption described in #27210: xgrammar's `GrammarMatcher.rollback()` does not reset the internal `terminated_` flag when rolling back past a terminating token, so the matcher believes the FSM is terminated while vLLM's wrapper believes it is active. Every subsequent `accept_token` for the draft-proposed `\\n` then fails with the `grammar_matcher.cc:612` warning. #27210 (on the pre-0.25 path) ended in a hard `AssertionError` crash and was closed via #28298, which clears `spec_token_ids` after scheduling instead of asserting. My suspicion is that on the v0.25 MRv2 path the same underlying rollback/terminated inconsistency no longer crashes but instead leaves the scheduler in a state it can never advance from - turning the former crash into a silent livelock (arguably worse operationally, since nothing restarts the engine). The impossible >100% acceptance-rate metric right before the hang supports the theory that draft-token accounting and grammar state disagree at that point. Possibly related: #34650 (MTP breaks stop/think-token detection in structured outputs), #40926 / #41530 (MTP worker hangs under sustained traffic, though those die with `EngineDeadError` via RPC timeout, while this one spins forever with TP=1 and no RPC in the path). ### Reproduction conditions - MTP speculative decoding (`qwen3_next_mtp`, 3 speculative tokens) + xgrammar structured outputs with `disable_any_whitespace: true` - Sustained concurrent traffic (~10–35 concurrent requests), high share of JSON-schema / tool-call requests - Hang occurred ~25 min after engine start under this load; the `token id 198` warnings start almost immediately after startup and flood continuously until the hang, so warning onset is quick even if the full livelock takes a while Happy to provide full logs, run a patched build, or test with `py-spy` attached if that helps narrow it down.", "url": "https://github.com/vllm-project/vllm/issues/49210", "createdAt": "2026-07-20T15:00:02Z", "updatedAt": "2026-08-13T06:58:47Z", "timestamp": "2026-08-13T06:58:47Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [], "author": "KotlinFactory", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:49413", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: KV offload event path refactor — provenance-carrying events and key-only removals", "text": "### Motivation. > Code references are pinned to `main` at > `3d204dfdaaf09d67d49c7855630ef949754e0f8f`; implementation starts from > current `main`. [#48679](https://github.com/vllm-project/vllm/pull/48679) added self-describing KV-offload events as an initial implementation. This RFC finishes the event-path refactor in two PRs. Today, `OffloadKey` contains only `(block_hash, group_idx)` ([base.py:23-41](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L23-L41)), and a raw `OffloadingEvent` has no request context ([base.py:153-159](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L153-L159)). The tracker therefore builds the full event payload at scheduler lookup/store time and keeps it until eviction ([events.py:78-114](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/distributed/kv_transfer/kv_connector/v1/offloading/events.py#L78-L114)). ```text lookup / prepare_store | snapshot full payload in a global map | async store / promotion completion | BlockStored | keep part of the snapshot until removal ``` This has two problems: - Async promotion and secondary-tier completions can lose the request provenance needed to build a full `BlockStored`. Promotions already retain `ReqContext` through `PendingPromotion` and `TransferJob` ([manager.py:62-68](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/tiering/manager.py#L62-L68), [manager.py:469-490](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/tiering/manager.py#L469-L490)), but raw events do not carry it. - Full removal metadata stays in vLLM from store until eviction. That lifetime can be much longer than the request, especially for persistent tiers. ### Proposed Change. #### Scope map | PR | Scope | | --- | --- | | **PR 1 — stored-event provenance** | Carry request context through existing async store, cascade, and promotion paths. Build `BlockStored` only when the raw stored event is translated. | | **PR 2 — key-only removal** | Emit only the chunk key in `BlockRemoved`, remove vLLM's long-lived removal table, and move chunk expansion/owner cleanup to the router. | This is the scope agreed in [#49506](https://github.com/vllm-project/vllm/pull/49506#issuecomment-5263904604). `OffloadKey`, single-key admission, and worker transfer reshaping are not part of this RFC. #### PR 1: stored-event provenance ```text Request + KV-group geometry | ReqContext event state | existing async job paths | CPU / FS / OBJ completion | OffloadingEvent(key, ReqContext) | full BlockStored ``` The implementation will: - Keep `OffloadKey = (block_hash, group_idx)`. - Attach request-scoped event state to `ReqContext`, which already supports typed per-request state ([base.py:90-104](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L90-L104)). - Lazily resolve an event key to its full-chunk or partial-tail position when the stored event is translated. - Carry the exact `ReqContext` through CPU, FS, OBJ, cascade, and promotion completion. - Generate per-block `extra_keys`, matching GPU events ([kv_events.py:50-76](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/distributed/kv_events.py#L50-L76)). - Keep placeholder events for external work, unsupported cache shapes, and token-mutating request paths whose hashes are not safe yet. - Keep only a small detached CPU removal record for PR 2 compatibility. It contains chunk hashes and group index, never `Request` or `ReqContext`. The normal lifetime is bounded by existing owners: ```text RequestOffloadState -> GPU store completion TransferJob / PendingPromotion -> secondary or promotion completion raw OffloadingEvent -> event translation ``` Ownership moves with the work: request state -> pending job -> raw stored event -> event translation. Failure and shutdown release their owned context; reset drains in-flight tier work and preserves any still-valid queued secondary stored event until translation. A synchronous failed tier submission unwinds the job and its primary-tier pin. Each job keeps the exact context object, so request-ID reuse cannot bind old work to a new request. PR 1 does not add a new refcount or retain request state until storage eviction. [#51614](https://github.com/vllm-project/vllm/pull/51614) is now merged and provides the KV-group block-granularity base used here. #### PR 2: key-only removal For one offload chunk: ```text chunk members = [H0, H1] chunk key = H1 vLLM: BlockRemoved([H1]) router: find H1's chunk -> remove that owner from H0 and H1 ``` PR 2 will: - Send only the removed chunk key and group. - Delete vLLM's detached removal-expansion table. - Let the router own `chunk key -> members` and member ownership. - Define CPU reset/restart cleanup for the producer owner scope. - Land with the matching router contract and implementation in [Dynamo #11930](https://github.com/ai-dynamo/dynamo/pull/11930). vLLM and router will move together; no compatibility window is planned. FS/OBJ deletion, TTL cleanup, and shared-storage global invalidation remain out of scope. ### Observable behavior. - PR 1: locally initiated promotions and secondary stores can emit full `BlockStored` payloads instead of placeholders. - PR 1: stored events include per-block `extra_keys`; plain-text events may carry explicit `None` entries. - PR 2: `BlockRemoved` changes from expanded block hashes to one chunk key. PR 1 does not change `OffloadKey`, manager admission, batching, worker transfers, or the public KV-event schema. ### Out of scope. - Single-key `prepare_store`, `OffloadReject`, and k-way admission ordering. - Worker transfer reshaping; [#44865](https://github.com/vllm-project/vllm/pull/44865) owns that work. - Adding `chunk_idx` to `OffloadKey`. - Request-side medium/locality filtering; [#48123](https://github.com/vllm-project/vllm/pull/48123) owns that API. - Fixing token-mutating request paths. [#49377](https://github.com/vllm-project/vllm/issues/49377) tracks stale `Request.block_hashes`; PR 1 keeps their event payloads as placeholders until their hash and offload state are safe. - FS/OBJ deletion events and shared-storage global invalidation. ### Implementation order. 1. PR 1: stored-event provenance, based on current `main`. 2. PR 2: key-only removal plus the matching Dynamo owner-index change. ### Feedback Period. The design discussion is complete. Implementation review should focus on event parity, promotion coverage, context cleanup on every terminal path, and the vLLM/router lockstep contract for PR 2. ### CC List. @orozery @ronensc @jthomson04 ### Any Other Things. | Related work | Relationship | | --- | --- | | [#43468](https://github.com/vllm-project/vllm/pull/43468) | Introduced self-describing offload events and expanded removals. | | [#51614](https://github.com/vllm-project/vllm/pull/51614) | Merged KV-group block-granularity fix used by PR 1. | | [#49377](https://github.com/vllm-project/vllm/issues/49377) | Tracks stale request hashes after token truncation. | | [Dynamo #11930](https://github.com/ai-dynamo/dynamo/pull/11930) | Router-side chunk ownership and key-only removal consumer. | AI assistance was used to trace the event lifecycle, prepare this RFC, and implement tests. All code links were verified against `main` at `3d204dfdaaf09d67d49c7855630ef949754e0f8f`. ### Before submitting a new issue... - [x] I searched for relevant issues and asked the documentation chatbot.", "url": "https://github.com/vllm-project/vllm/issues/49413", "createdAt": "2026-07-22T06:14:12Z", "updatedAt": "2026-08-13T05:43:52Z", "timestamp": "2026-08-13T05:43:52Z", "metrics": { "reactions": 1, "comments": 10 }, "labels": [], "author": "Change72", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:49548", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Dynamic speculative decoding (num_speculative_tokens_per_batch_size) causes catastrophic aggregate-throughput collapse under concurrency at the batch-size threshold (MTP, V1/PIECEWISE)", "text": "### Report of performance regression Enabling **dynamic speculative decoding** via `num_speculative_tokens_per_batch_size` produces two effects on a Qwen3.5-122B MTP (k=2) setup: 1. **Expected:** the documented `FULL_AND_PIECEWISE → PIECEWISE` cudagraph downgrade, which costs ~14% single-stream (59.6 → 50.0 tok/s). This is fine and matches the warning. 2. **Unexpected:** a **catastrophic aggregate-throughput collapse under concurrency** that looks like a pathological stall, *not* just \"no speculation at high batch.\" With the schedule `[[1,4,2],[5,512,0]]` (MTP k=2 for batch ≤ 4, spec **disabled** for batch ≥ 5), 8 concurrent 180-token requests dropped from **~232 tok/s aggregate** (static `num_speculative_tokens: 2`) to **24–157 tok/s**, with **wall-clock 40–60 s** for a workload that completes in **~6.3 s** under the static config — same model, same prompts, same token counts. Single-stream under the *same* dynamic config was a coherent 50 tok/s (the expected PIECEWISE level), so the collapse is specific to concurrency crossing the schedule's batch-size threshold. The naive expectation is that at batch ≥ 5 (K=0) throughput should fall back to roughly the **non-spec** decode rate. Instead it is ~1.5–10× *worse* than a plain non-spec run would be. That gap is what this issue is about. #### Environment - vLLM **0.23.1rc1.dev1302+ge765bbc97** (eugr/spark-vllm nightly) - **NVIDIA GB10 / DGX Spark**, sm_121a, unified memory (single device) - Model: **Qwen3.5-122B-A10B INT4 (AutoRound)**, native **MTP** speculator, `num_speculative_tokens: 2` - Attention backend: **FLASH_ATTN** - `enforce_eager=false`, cudagraph_mode requested `FULL_AND_PIECEWISE` - `VLLM_USE_V2_MODEL_RUNNER` **not** set (V1 runner) #### Speculative config Static (baseline, ~232 tok/s agg8, ~6.3 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2} ``` Dynamic (collapses to 24–157 tok/s, 40–60 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2, \"num_speculative_tokens_per_batch_size\": [[1, 4, 2], [5, 512, 0]]} ``` #### Numbers | Metric | static `k=2` | dynamic `[[1,4,2],[5,512,0]]` | |---|---:|---:| | Single-stream tok/s | 59.6 | 50.0 (expected PIECEWISE downgrade) | | Aggregate tok/s, 8 concurrent × 180 tok | ~232 | **24–157** | | Wall-clock, 8 concurrent | ~6.3 s | **40–60 s** | | Output token counts | identical | identical | #### Startup warning (as documented / expected) ``` Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` We have **not** yet tested `VLLM_USE_V2_MODEL_RUNNER=1` with the dynamic schedule (there is a separate open report, #48494, that V2 + `num_speculative_tokens_per_batch_size` + MTP crashes at speculator decode capture, so we held off). ### Misc discussion on performance — code-reading hypotheses We can't attach a profiler run yet (the box is a shared production node), so we read `main` (`f83de6d`) to make this actionable. Candidate mechanisms, strongest first: **H1 — At K=0 the MTP drafter still runs a full draft-model forward every step.** `SpecDecodeBaseProposer.propose` short-circuits on K=0 only **after** the first-pass draft forward has already executed: - `vllm/v1/spec_decode/llm_base_proposer.py:580-590` runs `self.model(**model_kwargs)` (the MTP-layer forward), - then `vllm/v1/spec_decode/llm_base_proposer.py:606-618` returns an empty tensor with the explicit comment *\"The prefill forward pass above already ran to keep the drafter KV cache in sync.\"* The runner also has **no K==0 guard** — the drafter is invoked whenever `spec_config is not None` and `input_fits_in_drafter` (`vllm/v1/worker/gpu_model_runner.py:4570-4601` and the MTP branch at `:5096-5212`, which calls `drafter.propose(num_speculative_tokens=0, ...)`). So \"spec disabled at batch ≥ 5\" is **not** equivalent to a non-spec run: every decode step at high batch still pays a full MTP forward plus all input prep (`set_inputs_first_pass`, attention-metadata build) for zero drafts. Combined with the per-step spec-decode overhead already noted in #47277, this alone can erase — and invert — the expected fallback throughput. **H2 — Dynamic SD disables the uniform-spec padding that preserves full cudagraphs, forcing PIECEWISE + variable per-step shapes.** The decode-padding path that pads to uniform spec size is explicitly gated off whenever a dynamic schedule is present: `vllm/v1/core/sched/scheduler.py:822-838` (`(self.num_spec_tokens > 0 and self.dynamic_sd_lookup is None)`). This is the documented `FULL_AND_PIECEWISE → PIECEWISE` downgrade (`vllm/config/vllm.py:819-836`). Under concurrency, per-step query lengths vary as K flips at the batch-4↔5 boundary and as requests enter/leave, so the target forward (`gpu_model_runner._determine_batch_execution_and_padding`, `vllm/v1/worker/gpu_model_runner.py:3876`) and the drafter forward (`vllm/v1/spec_decode/llm_base_proposer.py:1780`) see a churn of shapes, each dispatched/padded separately with attention metadata rebuilt every step. The dispatcher pads up rather than recapturing, so this isn't naive recapture — but PIECEWISE replay + eager glue over a constantly-shifting shape set, at batch 8, on a 273 GB/s unified-memory device, compounds. **H3 — K is keyed off the raw scheduled-request count, so the threshold thrashes.** `vllm/v1/core/sched/scheduler.py:1116-1120` selects `num_spec_tokens_to_schedule = self.dynamic_sd_lookup[len(num_scheduled_tokens)]`. `len(num_scheduled_tokens)` counts **all** scheduled requests including prefills/chunks, so a single prefill joining 4 decodes bumps the count to 5 and silently disables spec for the whole step. As the running set oscillates around the threshold, K flips 2↔0 step-to-step, changing the scheduled verify-token count and therefore both forward shapes, defeating cudagraph reuse and forcing repeated pad/dispatch/metadata rebuilds. This interacts badly with H2. **H4 — KV lookahead is over-reserved at K=0.** `num_lookahead_tokens` is fixed to the static `num_spec_tokens` at init (`vllm/v1/core/sched/scheduler.py:234, 246-258`) and passed to `allocate_slots` as `effective_lookahead_tokens` (`:886-889, 912-925`) — it is **not** lowered to the dynamic K. So at batch 8 with K=0 the scheduler still reserves `num_spec_tokens` (=2) lookahead slots per request every step. Reserved lookahead slots are normally transient, but they still count against the free-block admission check, so under KV pressure this could trigger earlier can't-schedule/preemption events than a true non-spec run — which would present exactly as a throughput stall. (Lowest-confidence; flagging for completeness.) H1 and H2 are the most defensible and, together, would explain why K=0-at-high-batch is materially worse than plain non-spec rather than equal to it. ### Offer We have a **GB10 / DGX Spark (sm_121a)** available and are happy to run follow-up diagnostics if a maintainer wants them — e.g. a torch profiler / nsys trace of the 8-concurrent dynamic run, `vllm:spec_decode_*` counters at batch ≥ 5, per-step cudagraph dispatch mode logging, or an A/B with the H1 wasted-forward path short-circuited. Just say which knobs/logging you'd like enabled. ### Your current environment ```text vLLM 0.23.1rc1.dev1302+ge765bbc97 (eugr/spark-vllm nightly) NVIDIA GB10 / DGX Spark, sm_121a, 128 GB unified memory (~273 GB/s) Model: Qwen3.5-122B-A10B INT4 AutoRound, native MTP, num_speculative_tokens=2 Attention backend: FLASH_ATTN (collect_env.py output can be attached on request from the box) ``` ### Before submitting a new issue... - [x] Searched existing/past issues. Closest are #47277 (native MTP per-step overhead, static K, single-stream), #42505 (DFlash slower at concurrency > 8, static K), #49369 (DSpark aggregate halved, static K), and #48494 (`num_speculative_tokens_per_batch_size` + MTP crashes full-cudagraph capture under V2). None cover this dynamic-schedule **concurrency-stall** signature (single-stream fine, aggregate collapses at the batch threshold under V1/PIECEWISE).", "url": "https://github.com/vllm-project/vllm/issues/49548", "createdAt": "2026-07-23T07:25:02Z", "updatedAt": "2026-08-13T17:35:41Z", "timestamp": "2026-08-13T17:35:41Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [], "author": "tobby168", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:49986", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Perf] DSD arms pay a large baseline tax vs no-spec under production defaults; PIECEWISE override identified as one factor", "text": "## Motivation While benchmarking `num_speculative_tokens_per_batch_size` (Dynamic SD) schedules on H100 NVL 94GB (`prithivMLmods/gemma-4-31B-it-qat-FP8` target + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` draft, vLLM `c5d967c23`) I found that **every arm using a `speculative_config` pays a substantial throughput tax vs no-spec under production defaults**, and the gap is large enough at short ctx that it dominates the aggregate spec-bench signal for schedules that lower K on the short-ctx majority. Position-balanced 2-trial (T1 order: C→N→S→A, T2 reverse: A→S→N→C), cache-wipe + cold-start-burn methodology, `prefix_repetition` c=256, 3 warmup + 3 measure per cell. | ctx | no-spec (K=0) | static K=3 | 3-item batch schedule (A′) | 2D 6-cell schedule (C′) | |---:|---:|---:|---:|---:| | 400 | **2711.7** | 2139.8 (−21%) | 1875.6 (**−31%**) | 1890.7 (−30%) | | 900 | 1987.1 | 1838.9 (−7%) | 1453.5 (−27%) | 1874.6 (−6%) | | 1900 | 1815.2 | 1822.4 (~0%) | 1416.6 (−22%) | 1848.2 (+2%) | | 4000 | 1535.9 | 1692.8 (+10%) | 1232.8 (−20%) | 1680.4 (+9%) | Two observations: 1. **The gap is largest at short ctx**, where a K=0 tier should be nearly free (drafter is loaded but the verifier is not asked to check any drafted tokens). At ctx=400, the DSD-mode arms are 20-31% slower than no-spec. 2. **The gap narrows as ctx grows**, and the 2D schedule overtakes no-spec at ctx ≥ 1900. The mechanism is memory-bound decode amortization — but the crossover would happen much earlier if the baseline tax at short ctx were smaller. ## Identified factor: PIECEWISE override Server logs confirm the DSD arms are downgraded from `FULL_AND_PIECEWISE` to `PIECEWISE`: ``` WARNING vllm.py:829 Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` The trigger is in `vllm/config/vllm.py::_maybe_override_dynamic_sd_cudagraph_mode` (lines 824-841): ```python def _maybe_override_dynamic_sd_cudagraph_mode(self) -> None: if ( speculative_config is None or not speculative_config.uses_dynamic_speculative_decoding() or not self.compilation_config.cudagraph_mode.has_full_cudagraphs() or self.use_v2_model_runner ): return self.compilation_config.cudagraph_mode = CUDAGraphMode.PIECEWISE ``` So all DSD schedules (including trivial `[[1, 512, 3]]` static schedules) lose full CUDA graphs unless `VLLM_USE_V2_MODEL_RUNNER=1` is set. The no-spec baseline retains full graphs, which likely explains part of the −31% gap at short ctx. **But cudagraph is not the only factor.** In an eager-mode control (`enforce_eager=True`, both arms lose graphs equally) at ctx=1900, static K=3 was still 6% slower than no-spec (0.94×), not faster as the mechanism would predict. So there is a residual DSD-mode overhead beyond the cudagraph-mode difference — probably drafter forward on K=0 steps, spec bookkeeping, admission cost — that needs to be decomposed. ## Proposed decomposition + fix path 1. **Decompose the tax** into: (a) PIECEWISE downgrade, (b) drafter forward on K=0 steps, (c) spec scheduling bookkeeping, (d) tier-boundary ramp effects (a small fraction of steps at c≈192-256 fire lower tiers than the aggregate suggests, per the ramp-leak observation in RFC #48627 §Motivation). 2. **True K=0 fast path**: when a DSD schedule selects K=0 for a given (bs, ctx), route through the no-spec path entirely for that step (skip drafter forward, skip spec accounting). This should recover most of the short-ctx gap without touching cudagraph capture. 3. **K-keyed FULL cudagraph capture**: since a declarative schedule enumerates the K set at startup (unlike learned per-step K predictors), the (batch bucket × K)-shape graphs are statically enumerable. Capturing them removes the need for the PIECEWISE downgrade in the DSD path — the \"resource-neutral\" property claimed for declarative schedules becomes literal once graphs are keyed on K. ## Repro Config: `prithivMLmods/gemma-4-31B-it-qat-FP8` + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`, H100 NVL 94GB, driver 580, CUDA 13.0, vLLM `c5d967c23` (branch `feat/dsd-2d-ctx-schedule` of #48944, but the tax is present with any DSD schedule including the merged `num_speculative_tokens_per_batch_size` from #32374). ```bash # static K=3 arm (reproduces the DSD-mode override): vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 \\ --speculative-config '{\"model\":\"\",\"num_speculative_tokens\":4,\"num_speculative_tokens_per_batch_size\":[[1,512,3]]}' # no-spec baseline: vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 # Bench (either arm): vllm bench serve --model --port 8000 \\ --dataset-name prefix_repetition --prefix-repetition-prefix-len 400 \\ --prefix-repetition-suffix-len 96 --prefix-repetition-num-prefixes 1 \\ --prefix-repetition-output-len 100 \\ --num-prompts 256 --max-concurrency 256 --ignore-eos \\ --percentile-metrics ttft,tpot,itl ``` Full raw data (server logs, per-cell measurements, snapshot deltas) available on request; happy to share the tarball or push to a public repo if that's easier for review. ## Context This came out of the measurement decomposition on #48944 — full 6-arm, 4-ctx table and methodology are in that PR's comment thread. The 2D ctx-axis schedule in #48944 is one specific instance of a DSD schedule; the baseline tax observed here applies to all DSD-mode arms (including the pre-existing `num_speculative_tokens_per_batch_size` API from #32374), so this issue is orthogonal to that PR and worth tracking separately. Happy to prototype the K=0 fast path as a follow-up PR if this direction sounds right — would appreciate a sanity check from @ekagra-ranjan (as the DSD API author) and @benchislett before I start.", "url": "https://github.com/vllm-project/vllm/issues/49986", "createdAt": "2026-07-27T13:04:39Z", "updatedAt": "2026-08-13T17:35:41Z", "timestamp": "2026-08-13T17:35:41Z", "metrics": { "reactions": 0, "comments": 10 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50282", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Expose startup status and health endpoint before model engine is ready", "text": "### 🚀 The feature, motivation and pitch vLLM currently exposes its HTTP API only after the model engine has completed initialization. For models with expensive startup phases, this can leave the service unreachable for a long time even though the process is alive and making progress. For example, startup may include: Loading model checkpoints Compiling execution paths Allocating KV cache FlashInfer or backend autotuning CUDA graph capture Model-specific kernel warmup With large or hybrid models, these phases can take many minutes. During that period, an external orchestrator or monitoring system cannot distinguish between: Normal initialization A stalled startup An out-of-memory condition A crashed worker A restart loop A process that is alive but not yet ready to serve inference I would like vLLM to expose a lightweight management or health endpoint before the model engine is ready. A possible design would separate process liveness, startup state, and inference readiness: ``` GET /livez GET /readyz GET /startupz ``` Example startup response: ``` { \"state\": \"starting\", \"phase\": \"model_warmup\", \"detail\": \"mamba2_ssd_kernel_warmup\", \"elapsed_seconds\": 842 } ``` A minimal implementation does not need an estimated completion time or exact percentage. Even a stable state and phase enum would be useful: ``` PROCESS_STARTED LOADING_MODEL COMPILING ALLOCATING_KV_CACHE AUTOTUNING_KERNELS CAPTURING_GRAPHS WARMING_UP READY FAILED ``` The management endpoint should be served by a lightweight component that starts before engine initialization and remains available if model initialization fails. This would make vLLM much easier to operate behind Kubernetes, systemd, appliance supervisors, load balancers, and enterprise monitoring systems. The goal is not necessarily to reduce startup time. The primary goal is to make long startup sequences observable and diagnosable. **Do not optimize startup first. Make startup observable first.** ### Alternatives The current alternatives are all external workarounds and provide incomplete information. Monitoring the process An orchestrator can verify that the vLLM process or container is running, but process liveness does not indicate whether initialization is progressing or stalled. `Process alive != Model ready` **Monitoring the inference port** The OpenAI-compatible endpoint can be polled until it becomes reachable. However, this only exposes two states: ``` Unavailable Available ``` It does not expose the difference between normal startup, initialization failure, deadlock, OOM, or repeated restarts. Parsing logs A supervisor can parse vLLM logs and infer startup progress from messages such as checkpoint loading, compilation, cache allocation, or warmup. This is fragile because: - Log messages are not a stable API - Formats may change between versions - Different model architectures use different startup stages - Logs may come from multiple worker processes - Structured failure state is difficult to derive reliably **Adding an external wrapper or sidecar** A wrapper can expose its own health endpoint and inspect the child process, logs, or inference port. However, it still cannot reliably access internal engine initialization state without parsing implementation-specific output. An external supervisor is useful for lifecycle management, but vLLM itself is the component that knows the current startup phase and failure reason. **Increasing startup probe timeouts** Kubernetes `startupProbe` or equivalent timeouts can be increased to accommodate slow model initialization. This prevents premature restarts but still provides no visibility into whether startup is progressing. ### Additional context This becomes increasingly important for large models and non-trivial architectures where startup is not dominated only by checkpoint I/O. A real startup sequence may look like: ``` Process started ↓ Checkpoint loading ↓ torch.compile ↓ KV cache allocation ↓ Backend kernel autotuning ↓ Model-specific kernel warmup ↓ API server ready ``` On large models, the interval between process creation and API readiness may be tens of minutes. Long initialization itself is acceptable in many production environments when it is expected and observable. The operational problem is that, during this interval, monitoring systems currently cannot tell whether vLLM is: ``` 🟡 Starting normally 🔴 Failed 🔴 Stalled 🔴 Out of memory 🔴 Restarting ``` This is also relevant for multi-model systems. A management plane may need to report states such as: ``` Model A: READY Model B: WARMING_UP Model C: FAILED ``` without treating the entire appliance or node as unavailable. It may also be useful to expose the same information as Prometheus metrics, for example: ``` vllm_engine_startup_state{state=\"warming_up\"} 1 vllm_engine_startup_phase{phase=\"kernel_warmup\"} 1 vllm_engine_startup_elapsed_seconds 842 ``` The exact endpoint names and phase granularity are open to discussion. The key requirements are: The status interface becomes reachable before model engine initialization completes. It distinguishes liveness from inference readiness. It exposes a stable, machine-readable startup state. It reports initialization failure when possible. It does not depend on parsing human-readable logs. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/50282", "createdAt": "2026-07-29T14:42:40Z", "updatedAt": "2026-08-13T03:47:02Z", "timestamp": "2026-08-13T03:47:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "xexxo2002-arch", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50587", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Kimi K3 Performance Optimization", "text": "### 🚀 The feature, motivation and pitch Tasks - [x] https://github.com/vllm-project/vllm/pull/50383 @jeejeelee - [x] https://github.com/vllm-project/vllm/pull/51146 @jeejeelee - [ ] https://github.com/vllm-project/vllm/pull/50478 @robertgshaw2-redhat - [x] https://github.com/vllm-project/vllm/pull/50484 @GirasoleY - [x] https://github.com/vllm-project/vllm/pull/50585 @yewentao256 - [ ] https://github.com/vllm-project/vllm/pull/50634 @JohnQinAMD - [x] https://github.com/vllm-project/vllm/pull/50656 @tlrmchlsmth - [x] https://github.com/vllm-project/vllm/pull/50912 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51070 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51311 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51725 + https://github.com/vllm-project/vllm/pull/51726 @yewentao256 ### Alternatives _No response_ ### Additional context I don't have enough GPU resources to run an end-to-end benchmark currently. If you have available GPU capacity and are willing to share access, I would greatly appreciate it! ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/50587", "createdAt": "2026-07-31T15:49:29Z", "updatedAt": "2026-08-12T19:06:52Z", "timestamp": "2026-08-12T19:06:52Z", "metrics": { "reactions": 4, "comments": 0 }, "labels": [ "feature request", "kimi", "k3" ], "author": "yewentao256", "state": "open", "assignees": [ "yewentao256" ], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:50682", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[ROCm][AMD] Kimi-K3 Gap and Roadmap Tracking", "text": "Kimi-K3 is a very successful launch. This issue is to track vLLM upstream work for feature enablement and performance optimization on ROCm.
Day 0 features/baselines - AITER fused-moe: a16w4 (GENERAL) and a8w4 (INTERLEAVE ) were integrated for performance. Flydsl/opus a8w4 is perf is 3.5% higher than flydsl a16w4. - AITER MLA attention: non-causal mla-gluon kernel was used for 12 head. - AITER triton conv2d - AMD vendor specific KDA - DSpark speculative decoding: Target model used AITER MLA (gluon) and draft model used TRITON_MLA. Perf improvement on lower concurrency (c1-c16). - Day 0 docker: vllm/vllm-openai-rocm:kimi-k3 - Day 0 recipes: https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x&features=reasoning - vLLM upstream PRs for day 0 K3 code: #50000 [#50089](https://github.com/vllm-project/vllm/pull/50089) [#50090](https://github.com/vllm-project/vllm/pull/50090)
**Feature Gaps**: - [ ] fp8 kv-cache-dtype support: Needed to avoid AgentX performance degradation. - asm-mla path: - https://github.com/vllm-project/vllm/pull/51040 - https://github.com/vllm-project/vllm/pull/51011 - gluon-mla path: https://github.com/vllm-project/vllm/pull/50619 - - [ ] gfx950/MI350 distributed inference: Required for multi-node K3 enablement and scaling. - https://github.com/vllm-project/vllm/pull/51052 - [x] DSpark: prefix-caching needs work - https://github.com/ROCm/aiter/pull/4474 - [ ] offloading performance/compatibility: Needed for offloading support and agentic workload. - cpu offloading + rocm 7.14/7.15 - [x] Dependency: 1. [ ] ROCm Base Docker: v0.1.19+ AITER upgrade - https://github.com/vllm-project/vllm/pull/49361 2. [ ] Triton 3.6 to 3.7+ upgrade - https://github.com/vllm-project/vllm/pull/50607 - Resolving issue of triton 3.7.x with gluon mla kernel on DistributedLinearLayout -- in progress * triton: https://github.com/ROCm/triton/pull/960 * vLLM to pick the new commit https://github.com/vllm-project/vllm/pull/51464 3. [ ] Triton-kernel package/naming management. - https://github.com/vllm-project/vllm/pull/50328 - [x] model runner V2 was disabled @vllmellm (Ye Hur, Pin Siang) - https://github.com/vllm-project/vllm/pull/51653 - [ ] gfx942 support and distributed inference - https://github.com/vllm-project/vllm/pull/50817 - **Performance Optimization**: - DSpark Speculative decoding: . * Goal: Agentic workflow: spec perf should be better than non-spec - fp8 kv-cache-dtype support: Dspark+fp8 kv cache+asm-mla path - minimal fix: https://github.com/vllm-project/vllm/pull/51860 - refactored: https://github.com/vllm-project/vllm/pull/51606 - qlen (K, num_speculative_tokens) tuning and support - https://github.com/vllm-project/vllm/pull/51590 - https://github.com/vllm-project/vllm/pull/51171 - - Update AITER in vLLM base docker: - https://github.com/ROCm/aiter/pull/4521 (merged) - https://github.com/ROCm/aiter/pull/4452 (merged) - https://github.com/ROCm/aiter/pull/4555 (merged) - https://github.com/ROCm/aiter/pull/4474 (merged) - https://github.com/ROCm/aiter/pull/4494 (merged) - https://github.com/ROCm/aiter/pull/4646 - https://github.com/ROCm/aiter/pull/4647 - https://github.com/ROCm/aiter/pull/4509 - - vLLM CPU offloading: Improve CPU-offloading performance (ROCm 7.14+). * https://github.com/vllm-project/vllm/pull/43018 * upgrade base docker to rocm 7.14 or above * - Agentic workload/benchmarking optimization. * profiling * max-num-seqs tuning - Mixed Attention optimization (Linear KDA 3 + MLA 1) - KDA : - Fusion: - #50654 - Prefill: - https://github.com/vllm-project/vllm/pull/51862 - https://github.com/vllm-project/vllm/pull/51540 - - MLA: - AITER Gluon MLA optimization (need new aiter release): * https://github.com/ROCm/aiter/pull/4509 (?) * https://github.com/ROCm/aiter/pull/4555 - Dispatch small head to AITER asm MLA (Done) * #50371 and #50578 and #51088 overlapped. Need consolidate to #50578 and add an opt-in env - - MoE: Flydsl a8w4 optimizations (multi-node, multi-stream) - [[FlyDSL] [MoE]: reuse stage-1(gate up) scratch buffer across layers and graph captures by xiaohuguo…](https://github.com/ROCm/aiter/pull/4647) - - Attention Res (triton and gluon), - Fusion: https://github.com/vllm-project/vllm/pull/50593 - Agentic workload (LMCache/Mooncake) - #51004 - gemm tuning - MORI-io + K3: (1) K3 KDA statement management. (2) UMBP (Unified Memory & Bandwidth Pool) to reduce cost without sacrificing performance. **Misc: Bug fixes, cleanup/refactor, documentation, recipe, packaging** - https://github.com/vllm-project/vllm/pull/50582 - #50649 - https://github.com/vllm-project/vllm/pull/51682 - - [ ] - LMCache packaging to vLLM docker https://github.com/vllm-project/vllm/pull/51208 - [ ] - Mooncake packaging to vLLM docker **PR List** https://github.com/vllm-project/vllm/pulls?q=is%3Aopen+is%3Apr+label%3Arocm+label%3Ak3 Note that: (1) There are some overlap and duplicated efforts, for example, #50371 and #50578. (2) PRs that do not meet vLLM Policy will not be reviewed, and may be closed by maintainers. ** Live Issues** https://github.com/vllm-project/vllm/issues?q=is%3Aissue%20state%3Aopen%20label%3Arocm%20label%3Ak3 --- Things are moving fast. Please comment if you have a PR or PRs for review. For each PR, please ensure you run the full gsm8k eval if for text-only. For multi-modal fixes, please run OCRBench/MMMU Pro. For performance optimization PR, in addition to ensure accuracy (eval test result is mandatory), please show end-to-end `vllm serve` command and environment variables you used, and the before/after comparison.", "url": "https://github.com/vllm-project/vllm/issues/50682", "createdAt": "2026-08-01T12:39:29Z", "updatedAt": "2026-08-13T15:25:30Z", "timestamp": "2026-08-13T15:25:30Z", "metrics": { "reactions": 0, "comments": 16 }, "labels": [ "rocm", "kimi", "k3" ], "author": "hongxiayang", "state": "open", "assignees": [ "hongxiayang" ] }, { "id": "github:vllm-project/vllm:issue:50690", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: gpt-oss chat completions return 500 \"Unexpected token 200002 while expecting start token 200006\" when ignore_eos=true", "text": "### Your current environment - vLLM **0.26.0** and **0.18.0** (both reproduce), official `vllm/vllm-openai` images - `openai-harmony` 0.0.8 - Model: `openai/gpt-oss-120b`, 1x H200, TP=1 - `vllm serve openai/gpt-oss-120b --max-model-len 32768 --tensor-parallel-size 1` ### 🐛 Describe the bug A Chat Completions request with `ignore_eos: true` returns **HTTP 500** whenever the model finishes its Harmony turn before `max_completion_tokens` is reached: ```json {\"error\":{\"message\":\"Unexpected token 200002 while expecting start token 200006\",\"type\":\"InternalServerError\",\"param\":null,\"code\":500}} ``` (200002 = `<|return|>`, 200006 = `<|start|>`) **Minimal reproducer** — one request, no streaming: ```bash curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\": \"openai/gpt-oss-120b\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hi.\"}], \"max_completion_tokens\": 512, \"ignore_eos\": true }' ``` The prompt matters: the answer must be **shorter** than `max_completion_tokens`. A prompt whose answer fills the budget never reaches the terminator and succeeds, which makes this look intermittent. | arm | 0.18.0 | 0.26.0 | |---|---|---| | short answer + `ignore_eos` | 500 | 500 | | answer longer than `max_completion_tokens` + `ignore_eos` | OK | OK | | short answer, no `ignore_eos` | OK | OK | ### Why `<|return|>` / `<|call|>` are the Harmony assistant stop tokens. `ignore_eos=true` suppresses stopping on them (`SamplingParams.update_from_generation_config`, the `if not self.ignore_eos:` guards), so generation continues past the end of the turn. Those trailing tokens are then fed to the `StreamableParser`, which is a state machine: after `<|end|>` it expects `<|start|>`, gets `<|return|>`, and raises. The exception is not caught, so the whole request fails. The parser half reproduces **without a server or GPU**: ```python from openai_harmony import HarmonyEncodingName, Role, StreamableParser, load_harmony_encoding enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS) e = lambda s: enc.encode(s, allowed_special=\"all\") stream = e(\"<|channel|>\") + e(\"final\") + e(\"<|message|>\") + e(\"answer\") + e(\"<|end|>\") + e(\"<|return|>\") p = StreamableParser(enc, role=Role.ASSISTANT) for t in stream: p.process(t) # HarmonyError: Unexpected token 200002 while expecting start token 200006 ``` vLLM is on both sides of this: it permits `ignore_eos` to override its own Harmony stop tokens, then treats the resulting token stream as a parse error. ### Impact `ignore_eos` is how benchmark tools pin output length so ISL/OSL combinations are controlled — both `vllm bench serve` (`--ignore-eos`) and GuideLLM use it. It is therefore hard to benchmark gpt-oss over Chat Completions at all. Measured with GuideLLM against 0.18.0 at ISL/OSL 1024/1024: **65% of requests failed** (971 errors, all this message). At OSL 128 the error rate was 0%, because the model rarely finishes a turn that early — so the failure rate scales with output length and can easily be mistaken for server saturation. `/v1/completions` is unaffected (no Harmony parsing on that path). ### Related - #41731 \"[Bugfix] Respect ignore_eos for GPT-OSS Harmony stop tokens\" — describes this exact cause, closed without merging (went stale on merge conflicts) - #41950 \"[Bugfix] Stop Harmony stream parsing after parser errors\" — the parser-side guard, closed without merging - #44009 \"[Frontend] Clean up stop_token_ids override for Harmony\" — merged, removed the `default_sampling_params` stop-token injection, but **does not fix this**: 0.26.0 still returns 500 (table above) - #22519 — same parser error via the tool-call path Either half would fix it: honor the Harmony stop tokens even under `ignore_eos` (#41731), or stop feeding a failed parser and fall back to raw tokens (#41950). ### Before submitting a new issue... - [x] Made sure I searched the documentation and existing issues.", "url": "https://github.com/vllm-project/vllm/issues/50690", "createdAt": "2026-08-01T15:05:45Z", "updatedAt": "2026-08-13T13:15:58Z", "timestamp": "2026-08-13T13:15:58Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "rebel-joonhyunglee", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50699", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DiffusionGemma: runtime CUDA OOM under concurrent decode (fp32 canvas×vocab temporaries)", "text": "### Summary Block-diffusion decode produces `canvas_length` (default 256) logit rows **per request per step**, and `_compiled_sample_step` in `vllm/model_executor/models/diffusion_gemma.py` materializes ~6 fp32 temporaries of shape `[num_decode, canvas_length, vocab_size=262144]`. At `--max-num-seqs 8` each of these tensors is 2.1 GiB (>12 GiB total transient). Startup/warmup passes (KV pool still empty), but at runtime the KV pool grows to fill the memory budget, leaving <1 GiB physical headroom → the engine **dies with CUDA OOM on the first full batch**. This is diffusion-specific: autoregressive decode has 1 logit row per request, so this transient never scales with concurrency there. ### Environment - vLLM `0.22.1rc1.dev357+g74b5964f0` (docker `vllm/vllm-openai:gemma`); the same pattern is present in current `main` - GPU: RTX 5090 32GB, driver 610.43.02, torch 2.11.0+cu130, CUDA 13.3 - Model: `RedHatAI/diffusiongemma-26B-A4B-it-NVFP4` (~17 GiB weights) ### Reproduction ```bash vllm serve RedHatAI/diffusiongemma-26B-A4B-it-NVFP4 \\ --max-num-seqs 8 --max-model-len 131072 \\ --gpu-memory-utilization 0.75 --kv-cache-dtype fp8 \\ --generation-config vllm \\ --hf-overrides '{\"diffusion_sampler\":\"entropy_bound\",\"diffusion_entropy_bound\":0.1}' \\ --diffusion-config '{\"canvas_length\":256}' ``` Then send 8 concurrent chat requests (`max_tokens=512`). ### Observed ``` torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 31.40 GiB of which 1.01 GiB is free. File \".../vllm/model_executor/models/diffusion_gemma.py\", line 525, in _compiled_sample_step ``` Memory math at `num_decode=8, CL=256, vocab=262144`: one fp32 tensor = `8×256×262144×4 B` = 2.1 GiB; the compiled step keeps several alive simultaneously (`logits_3d.float()`, `scaled`, gumbel noise, `log_probs`, `probs`, entropy product, plus the fp32 `scaled` it returns). ### Local workaround (validated) Chunking decode requests through `_compiled_sample_step` (e.g. 2 requests/call) fixes it: all state tensors are slot-indexed, so per-chunk in-place updates are equivalent to one call; only the zeroing of `sampled`/`num_sampled` has to move out of the per-chunk call. Combined with row-chunked `compute_logits` (preallocated output buffer), `--max-num-seqs 16` survives repeated 16-way concurrent load tests (0 OOM), with unchanged outputs. Happy to clean this up into a PR if the approach sounds reasonable. --- *Disclosure: the investigation, patch and this write-up were done with AI assistance (a local coding agent), verified with real load tests on the hardware above.*", "url": "https://github.com/vllm-project/vllm/issues/50699", "createdAt": "2026-08-01T20:27:40Z", "updatedAt": "2026-08-13T15:19:02Z", "timestamp": "2026-08-13T15:19:02Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "pdos95", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50722", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: With qwen3.5-35b-a3b, the performance is relatively poor both when using dflash and when not using it, but the accepted length of dflash is around 5–6.", "text": "### Your current environment
```text vllm 0.26.0 +12.9 torch 2.11+12.6 { \"architectures\": [ \"DFlashDraftModel\" ], \"auto_map\": { \"\": \"config.DFlashSpeculatorConfig\" }, \"aux_hidden_state_layer_ids\": [ 1, 6, 11, 16, 22, 27, 32, 37 ], \"block_size\": 8, \"draft_vocab_size\": 24576, \"dtype\": \"bfloat16\", \"mask_token_id\": 248077, \"sample_from_anchor\": false, \"sliding_window_non_causal\": false, \"speculators_config\": { \"algorithm\": \"dflash\", \"default_proposal_method\": \"greedy\", \"proposal_methods\": [ { \"accept_tolerance\": 0.0, \"proposal_type\": \"greedy\", \"speculative_tokens\": 7, \"verifier_accept_k\": 1 } ], \"verifier\": { \"architectures\": [ \"Qwen3_5MoeForConditionalGeneration\" ], \"name_or_path\": \"\" } }, \"speculators_model_type\": \"dflash\", \"speculators_version\": \"0.7.0.dev149\", \"target_hidden_size\": null, \"tie_word_embeddings\": false, \"transformer_layer_config\": { \"attention_bias\": false, \"attention_dropout\": 0.0, \"bos_token_id\": 1, \"eos_token_id\": 2, \"head_dim\": 256, \"hidden_act\": \"silu\", \"hidden_size\": 2048, \"initializer_range\": 0.02, \"intermediate_size\": 6144, \"layer_types\": [ \"sliding_attention\", \"sliding_attention\", \"sliding_attention\", \"sliding_attention\", \"sliding_attention\" ], \"max_position_embeddings\": 262144, \"mlp_bias\": false, \"model_type\": \"llama\", \"num_attention_heads\": 32, \"num_hidden_layers\": 5, \"num_key_value_heads\": 2, \"pad_token_id\": null, \"pretraining_tp\": 1, \"rms_norm_eps\": 1e-06, \"rope_parameters\": { \"rope_theta\": 1000000, \"rope_type\": \"default\" }, \"sliding_window\": 2048, \"tie_word_embeddings\": false, \"use_cache\": true, \"vocab_size\": 248320 }, \"transformers_version\": \"5.10.4\" } ```
### 🐛 Describe the bug With qwen3.5-35b-a3b, the performance is relatively poor both when using dflash and when not using it, but the accepted length of dflash is around 5–6. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/50722", "timestamp": "2026-08-12T17:16:19Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "bug" ], "author": "Jim2016713", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:50765", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][P/D]: NIXL handshake failure with asymmetric TP in PD disaggregation (pTP2 + dTP4)", "text": "### Your current environment - vLLM version: `0.1.dev1+g51f799c1a` - NVIDIA GPU: H200 (140GB) - CUDA: 13 - NIXL transport: UCX/RDMA (RoCE) ### Model google/gemma-4-26B-A4B (MoE, 26B total, ~4B active) ### Describe the bug NIXL KV transfer handshake fails with `NIXL_ERR_BACKEND` when using asymmetric tensor parallelism in PD disaggregation (prefill TP=2, decode TP=4). The failure occurs during `loadRemoteMD` — the RDMA metadata exchange — before any actual KV transfer happens. The prefill pods (2×TP2) start and serve correctly. The decode pod (1×TP4) loads the model and begins serving, but crashes when the first request triggers a NIXL handshake with a prefill pod. The block count mismatch between prefill (`num_remote_blocks: 113`) and decode (`num_local_blocks: 211`) causes the handshake to fail. After the decode pod crashes and restarts, subsequent handshakes continue to fail (possibly related to #38840). Symmetric TP configurations (e.g., pTP4 + dTP4) work correctly. ### Configuration ``` Prefill: 2 pods × TP2 Decode: 1 pod × TP4 Total GPUs: 8 (single node) enable_bidirectional_kv: true ``` ### Error logs (decode pod) ``` (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] NIXL transfer failure: handshake_failed | Context: {'failure_type': 'handshake_failed', 'request_id': 'cmpl-39ab6c4d-b1df-473c-a398-b6c02944f8c8-0-ada18e77', 'engine_id': '48174aad-7e0e-48bf-9494-00877081f134', 'remote_engine_id': '7cd7cde3-e7cc-4347-a286-92f4f2581f70', 'remote_host': '10.131.1.160', 'remote_port': 5600, 'num_local_blocks': 211, 'num_remote_blocks': 113, 'local_block_ids_sample': [2474, 2475, 2476]} (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] Traceback (most recent call last): (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 764, in request_ready (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] f.result() (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 733, in done_callback (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] self._remote_agents[eid] = f.result() (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 581, in _nixl_handshake (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] remote_agent_name = self.add_remote_agent( (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 1309, in add_remote_agent (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] remote_agent_name = self.nixl_wrapper.add_remote_agent( (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm/lib64/python3.12/site-packages/nixl_cu13/_api.py\", line 832, in add_remote_agent (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] agent_name = self.agent.loadRemoteMD(metadata) (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND ``` ### Related issues - #21576 — NIXLConnector does not support P TP > D TP (closed as stale, not fixed) - #22663 — Fix heterogeneous TP (merged, fixed KV cache layout to HND, but doesn't fix `loadRemoteMD` handshake failure) - #38840 — Handshake race when same-node workers re-register with new engine IDs", "url": "https://github.com/vllm-project/vllm/issues/50765", "createdAt": "2026-08-02T19:58:11Z", "updatedAt": "2026-08-13T02:12:38Z", "timestamp": "2026-08-13T02:12:38Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "bbenshab", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50851", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] DSpark speculative decoding broken on nightly", "text": "## Summary DSpark speculative decoding (`method: \"dspark\"`) is non-functional on current main. Multiple code paths assume only `\"dflash\"` when DSpark reuses the same proposer/speculator infrastructure (`DSparkSpeculator` extends `DFlashSpeculator`, both use `DFlashProposer`). The result is that DSpark either crashes at startup or produces wrong results at inference time. Tested on GLM-5.2-FP8 (753B MoE) with `RedHatAI/GLM-5.2-speculator.dspark` on multi-node Wide-EP H200 clusters. ## Bugs ### 1. `use_dflash()` excludes DSpark — model runner never creates `DFlashProposer` **File:** `vllm/config/speculative.py` (line ~1395) ```python def use_dflash(self) -> bool: return self.method == \"dflash\" # BUG: excludes \"dspark\" ``` `use_dspark()` exists (line ~1398) but is never checked in the model runner's drafter initialization elif chain (`gpu_model_runner.py`). DSpark falls through without getting `DFlashProposer` or `use_aux_hidden_state_outputs = True`, so the target model doesn't extract multi-layer aux hidden states. **Fix:** `return self.method in (\"dflash\", \"dspark\")` ### 2. `DFlashProposer.__init__` asserts `method == \"dflash\"` **File:** `vllm/v1/spec_decode/dflash.py` ```python assert vllm_config.speculative_config.method == \"dflash\" ``` Even after fixing #1, this assert rejects DSpark at proposer creation time. **Fix:** `assert method in (\"dflash\", \"dspark\")` ### 3. `llm_base_proposer.py` missing \"dspark\" in method lists **File:** `vllm/v1/spec_decode/llm_base_proposer.py` Three locations where DSpark needs to be included alongside DFlash: - **`model_returns_tuple()`** (~line 1015): exclusion list `(\"mtp\", \"draft_model\", \"dflash\")` missing `\"dspark\"` → `ValueError: too many values to unpack (expected 2)` at line ~591 - **`combine_hidden_states` gate** (~line 526): inclusion list `(\"eagle3\", \"dflash\")` missing `\"dspark\"` → aux hidden states not combined for the drafter - **`parallel_drafting_token_id` ValueError** — raises for unknown method instead of setting a default for DSpark ### 4. FA3 AOT scheduler head count mismatch (separate PR) Already addressed in #50694 — the FA3 AOT scheduler reads head counts from the target model config, but the draft model may have a different GQA ratio. ## Error Traces ### Without any hotfixes — `model_returns_tuple` crash: ``` ValueError: too many values to unpack (expected 2) File \"vllm/v1/spec_decode/llm_base_proposer.py\", line 591 ``` ### After adding \"dspark\" to `model_returns_tuple` but not fixing routing — `combine_hidden_states` dimension mismatch: ``` ValueError: DFlash drafter expects 30720 concatenated aux hidden features but received 6144 File \"vllm/model_executor/models/qwen3_dflash.py\", line 761 File \"vllm/v1/spec_decode/llm_base_proposer.py\", line 536 ``` (30720 = 5 layers × 6144 hidden_size — target model only outputs last hidden state because `use_aux_hidden_state_outputs` was never set) ## Environment - vLLM nightly (`vllm/vllm-openai:nightly`, v0.26.1rc1.dev255) - Model: `zai-org/GLM-5.2-FP8` + `RedHatAI/GLM-5.2-speculator.dspark` - Config: `--speculative-config {\"model\":\"...\",\"num_speculative_tokens\":5,\"method\":\"dspark\"}` - Multi-node Wide-EP with H200 GPUs, DP=16, EP enabled", "url": "https://github.com/vllm-project/vllm/issues/50851", "createdAt": "2026-08-03T10:28:17Z", "updatedAt": "2026-08-13T06:44:40Z", "timestamp": "2026-08-13T06:44:40Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [], "author": "elvircrn", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:50895", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Populate completion_tokens_details in streaming and non-streaming usage responses", "text": "### 🚀 The feature, motivation and pitch The OpenAI chat/completions spec defines `completion_tokens_details` as part of `CompletionUsage`, which is shared between streaming and non-streaming responses. vLLM does not populate this field in either mode. vLLM 0.23+ fixed this for `prompt_tokens_details` (#44961): both streaming and non-streaming responses now include `cached_tokens` when `--enable-prefix-caching` and `--enable-prompt-tokens-details` are set. `completion_tokens_details` has no equivalent support. The concrete gap: ``` Non-streaming response (vLLM 0.25.x): { \"usage\": { \"prompt_tokens\": 72, \"completion_tokens\": 30, \"total_tokens\": 102, \"prompt_tokens_details\": {\"cached_tokens\": 64}, // works (0.23+) \"completion_tokens_details\": null // missing } } Streaming final chunk (stream_options.include_usage: true): { \"usage\": { \"prompt_tokens\": 77, \"completion_tokens\": 30, \"total_tokens\": 107 // completion_tokens_details: absent } } OpenAI spec expects: { \"usage\": { \"prompt_tokens\": 72, \"completion_tokens\": 30, \"total_tokens\": 102, \"prompt_tokens_details\": {\"cached_tokens\": 64}, \"completion_tokens_details\": { \"reasoning_tokens\": 0, \"accepted_prediction_tokens\": 0, \"rejected_prediction_tokens\": 0 } } } ``` **Why this matters for downstream consumers:** Metering and billing systems that sit behind an OpenAI-compatible gateway parse the `usage` object from the response to produce per-request records. Without `completion_tokens_details`, these systems cannot distinguish reasoning tokens from regular completion tokens. The workaround is to treat all completion tokens as one price tier, which prevents operators from offering pricing that reflects the different cost structure of reasoning-capable models. The Red Hat AI / OpenShift AI ecosystem uses vLLM as the model server behind Kuadrant/Connectivity Link gateways, and the billing layer needs these fields to price accurately. ### Alternatives Without this feature, downstream consumers must either: 1. Treat all completion tokens uniformly (loses pricing granularity for reasoning-capable models). 2. Attempt to recount reasoning tokens by re-parsing the response body outside vLLM (duplicates work the model server already did, fragile, and not possible for all model architectures). ### Additional context There are four open PRs implementing `reasoning_tokens` support, all currently in merge-conflict state: #35037, #44398, #45471, and #45802. Of these, #45802 (@chaunceyjiang) has the deepest integration through the streaming parser engine, and #45471 (@nv-nedelman-1) has the most thorough test coverage. Community contributor @dsgrafiniert has also implemented `count_reasoning_tokens()` for the 7 reasoning parsers that were missing it (posted on #35037). The foundation work is already merged: reasoning parser hooks (#33513) and the streaming parser engine (#45701). This issue tracks the broader ask: the full `completion_tokens_details` object, with streaming parity included from the start, following the same pattern as the `prompt_tokens_details` fix in #44961. We are happy to contribute a PR if that would help move this forward. Fields by relevance: | Field | Source | Status | |-------|--------|--------| | `reasoning_tokens` | Reasoning parsers | Four open PRs (#35037, #44398, #45471, #45802) + parser fork | | `accepted_prediction_tokens` | Speculative decoding | Not yet tracked | | `rejected_prediction_tokens` | Speculative decoding | Not yet tracked | `reasoning_tokens` is the highest priority. The speculative-decoding fields can follow separately. cc @cemigo114", "url": "https://github.com/vllm-project/vllm/issues/50895", "createdAt": "2026-08-03T16:41:50Z", "updatedAt": "2026-08-13T06:31:43Z", "timestamp": "2026-08-13T06:31:43Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [], "author": "rdwj", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51181", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeekV4 DSpark CUDA graph capture failure on H100", "text": "### Your current environment
The output of python collect_env.py ```textCollecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.14.0-284.118.1.el9_2.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H100 80GB HBM3 GPU 1: NVIDIA H100 80GB HBM3 GPU 2: NVIDIA H100 80GB HBM3 GPU 3: NVIDIA H100 80GB HBM3 GPU 4: NVIDIA H100 80GB HBM3 GPU 5: NVIDIA H100 80GB HBM3 GPU 6: NVIDIA H100 80GB HBM3 GPU 7: NVIDIA H100 80GB HBM3 Nvidia driver version : 580.173.02 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6430 CPU family: 6 Model: 143 Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 76% CPU max MHz: 3400.0000 CPU min MHz: 800.0000 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities L1d cache: 3 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 128 MiB (64 instances) L3 cache: 120 MiB (2 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 NUMA node2 CPU(s): 16-23 NUMA node3 CPU(s): 24-31 NUMA node4 CPU(s): 32-39 NUMA node5 CPU(s): 40-47 NUMA node6 CPU(s): 48-55 NUMA node7 CPU(s): 56-63 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.15.post1 [pip3] helion==1.1.0 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.26.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.14.1 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.1rc1.dev385+g7fd4af585 (git sha: 7fd4af585) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX SYS SYS SYS SYS 0-7 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PXB SYS SYS SYS SYS 0-7 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS PXB NODE SYS SYS 16-23 2 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS PIX NODE SYS SYS 16-23 2 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS PXB SYS 32-39 4 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS PIX SYS 32-39 4 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS PXB 48-55 6 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS PIX 48-55 6 N/A NIC0 PIX PXB SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS NIC1 SYS SYS PXB PIX SYS SYS SYS SYS SYS X NODE SYS SYS NIC2 SYS SYS NODE NODE SYS SYS SYS SYS SYS NODE X SYS SYS NIC3 SYS SYS SYS SYS PXB PIX SYS SYS SYS SYS SYS X SYS NIC4 SYS SYS SYS SYS SYS SYS PXB PIX SYS SYS SYS SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=/var/run/nvidia-container-devices CUDA_COREDUMP_SHOW_PROGRESS=1 CUDA_COREDUMP_GENERATION_FLAG=skip_nonrelocated_elf_images,skip_global_memory,skip_shared_memory,skip_local_memory,skip_constbank_memory VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=1024 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_NO_USAGE_STATS=1 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=1024 VLLM_ENFORCE_STRICT_TOOL_CALLING=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root VLLM_WORKER_MULTIPROC_METHOD=spawn ```
### 🐛 Describe the bug While launching DeepSeek V4 0731 with DSpark speculative decoding on, the server crashes on the CUDA graph capture stage. Exact reproduction requires single H100 * 8 machine. # Launch script ```shell vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --kv-cache-dtype fp8 \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --all2all-backend deepep_v2 --max-num-seqs 32 \\ --tokenizer-mode deepseek_v4 \\ --speculative-config.method dspark \\ --speculative-config.num_speculative_tokens 5 \\ --reasoning-parser deepseek_v4 \\ --enable-auto-tool-choice \\ --tool-call-parser deepseek_v4 ``` # Error traceback ```log ... Capturing CUDA graphs (PIECEWISE): 65%|███████ | 28/43 [01:16<00:20, 1.38s/it] [19:25:48.403492] coredump: Starting GPU coredump generation ... ... ... [19:25:48.404432] coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS (5) [19:25:48.404435] coredump: - Device: 4 [19:25:48.404437] coredump: - SM: 124 [19:25:48.404439] coredump: - Warp: 1 [19:25:48.404440] coredump: - PC 0x7ef928bc46b0 [19:25:48.404688] coredump: Stack trace (lane masks: active 0xFFFFFFFF, valid 0xFFFFFFFF): [19:25:48.404692] coredump: # 0 0x7ef928bc46b0 _ZN9deep_gemm5sched30sm90_paged_mqa_logits_metadataILj96ELj256ELj132ELb0EEEvjjbPKjS3_Pj ... ``` # Root Cause Analysis As printed in coredump, the DeepGEMM version built in latest main has an OOB bug. A fix is submitted on deepseek-ai/DeepGEMM#399 , but this bug is really easy to hit in practice - the dummy inputs used for CUDA graph capture is pretty small so that the OOB edge condition (`q_idx == batch_size`) is easily met. I'd rather suggest pulling the PR into vllm built-in DeepGEMM instead of waiting for the review upstream. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51181", "createdAt": "2026-08-05T16:33:30Z", "updatedAt": "2026-08-13T13:12:44Z", "timestamp": "2026-08-13T13:12:44Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug" ], "author": "cjackal", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51275", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Race-free port management: pick ports at bind time, publish over an existing channel", "text": "## Motivation `get_open_port()` probes by binding a port, closes that socket immediately, and returns only the number. A later process or library performs the real bind. Another process can claim the port in the interval, producing ZMQ `Address already in use`, PyTorch `DistNetworkError` / `EADDRINUSE`, or Rust `AddrInUse` during concurrent cold starts. The governing rule is: **the process that binds chooses the port at bind time, and publishes the actual endpoint over an existing control path.** A port number returned from a probe is already unreserved. ## Failure classes | Category | Failure | Correct direction | |---|---|---| | I. Probe-then-bind | The probe releases a port before its eventual consumer binds it. | Hold/pass the socket, or let the final library bind `:0` and report the resulting endpoint. | | II. Scan collision | Multiple deployments scan from the same `VLLM_PORT` base. | Avoid shared scans; maintain only explicitly justified reserved ranges. | | III. Wrong-host probe | Driver chooses a port locally, but a remote worker binds it. | Have the binding worker choose and publish its endpoint. | | IV. Static formula | Independently launched deployments compute the same `base + offset`. | Deployment/service-discovery configuration; vLLM has no shared allocator. | Category IV is out of scope: two independent deployments cannot safely “skip” to different values without a shared discovery mechanism. ## Current inventory and coverage | Scenario | Site | Category | Status / fix | PR | |---|---|---:|---|---| | Single-node uniproc / mp | torch rendezvous | I | Replace TCP rendezvous with `file://`. | #50999 | | Multi-node mp | `shm_broadcast` remote ZMQ subscription | I | ZMQ binds `:0` itself; report the actual endpoint. | #50960 / #44495 | | DP world groups | port selected before model load | I | Rank 0 holds its socket while publishing through the coordination store. | #51018 | | `--enable-fault-tolerance` recovery | recreated Engine/worker DP groups | I | Binding rank holds the socket while publishing through the existing DP store. | #51404 | | Rust/Ray API frontend | Driver pre-allocated ZMQ ports before the final frontend binder | I | Final frontend binds `:0` and publishes its actual endpoints. | #51033 | | EngineCore startup handshake | Configured ZMQ RPC endpoint | N/A | Port `0` cannot be discovered by remote nodes; require a fixed port in `1..65535` and use it directly. | #51469 | | `VLLM_PORT` | shared scanning and DP reservation overlap | II | Do not scan into the DP reserved range. | #50965 | | Ray executors v1/v2 | Driver chose TCPStore endpoint although rank 0 binds it in a Ray actor | III; residual I | Rank-0 actor now chooses and advertises its own reachable endpoint. A held-socket TCPStore handoff remains separate work. | #50969 | | Rust managed-engine | `allocate_handshake_port()` drops a `TcpListener`, then passes its number to a spawned Python EngineCore | I | Uncovered; child must bind/publish its endpoint or receive a held reservation. | to be opened | | ROCm MoRIIO connector | Python probes `local_kv_port`, then external `mori.io.IOEngine` performs the real bind | I | Uncovered; needs an upstream IOEngine API that either binds `:0` and reports its endpoint or accepts a supplied listener. | to be opened | | External launcher | launcher must choose a port before spawning all nodes | I | Uncovered; pass a held listener fd to the binding child (`LISTEN_FDS`-style). | to be opened | ### Evidence recorded on 2026-08-07 - #51469 audit correction (2026-08-08): the EngineCore handshake has no channel that can publish a dynamically selected port to headless secondary nodes. The prior squatter experiment proved only a local bind race after forcing the unsupported value `0`; it did not prove end-to-end reachability. The rewritten PR rejects `0` and removes the `get_open_port()` fallback. - #51404: deterministic FT Engine DP recreation failed with `EADDRINUSE`; the PR branch passed with a held TCPStore listener. - #51018: deterministic DP world-group TCPStore bind failed with `EADDRINUSE`; the PR branch initialized through the held listener. - #50969: on a real two-node Ray cluster (h200-0 driver `172.27.48.114`, h200-1 actor `172.27.55.233`), old v1 formed `tcp://172.27.48.114:47021` while the h200-1 actor owned the bind and a squatter caused errno 98. The updated branch selected `tcp://172.27.55.233:55147` from rank 0 itself. This validates Category III only; a same-host probe-to-bind interval remains. - Rust managed-engine: calling the real `allocate_handshake_port()` returned a port after dropping its `TcpListener`; a squatter then claimed it and the simulated downstream bind failed with errno 98 on both H200 hosts. ## Proposed guardrail 1. Provide and standardize `bind_ephemeral()` for APIs that can accept a socket. 2. Require new `get_open_port()` / `get_open_ports_list()` call sites to be allowlisted with a binder/consumer justification. The inventory above is the migration checklist. 3. In code review, identify the final binder before accepting any port-selection code. If the final library only accepts an address string, it must own `:0` binding and expose the endpoint; holding an unrelated reservation socket is not a safe substitute. The desired end state is an empty allowlist for probe-then-bind call sites. The three external integrations above need an API/design change rather than a mechanical replacement. --- AI assistance (Claude Code and OpenAI Codex) was used for investigation, experiments, implementation, and drafting. A human submitter must review each change and test result before requesting maintainer review. - #51018 follow-up: the original PR incorrectly limited coordination-store creation to MoE models. A non-MoE DP=2 `launch_core_engines()` reproducer failed on both H200 hosts before commit `2827357fb7`; it now passes and is covered by a focused regression test. - MoRIIO evidence: with the production `MoRIIOConfig.from_vllm_config()` on both H200 nightly containers, a deterministic squatter claims the released `local_kv_port` and an ABI-equivalent external binder fails `98: Address already in use`. `mori.io` and `/dev/kfd` are absent there, so this is a vLLM handoff proof, not ROCm/RDMA hardware validation. - Rust evidence rechecked: a Rust 1.95 container compiled and called the repository's real `allocate_handshake_port()` on both H200 hosts. A squatter then claimed the returned port and the downstream bind failed with OS error 98. - Audit closure (2026-08-07): searched Python, Rust, C/CUDA sources for `get_open_port*`, `TcpListener::bind`, `TCPStore`, and direct dynamic `bind`. No additional dynamic port-selection path exists outside the inventory above: Rust engine-core-client/coordinator and Mooncake bind `:0` in the final owning socket; C/CUDA has no network-listener implementation. Static configured endpoints (KV events, LMCache, etc.) are Category IV deployment configuration, not probe-then-bind races. - Fresh verification (2026-08-07): #51404 head `320bbbb6a9` was re-run in h200-1 nightly against base `47228db84c` on h200-0; base produced TCPStore `code: -98, name: EADDRINUSE`, head completed FT DP reinit with the held listener. - Re-enumeration: restricting the source search to first-party `vllm/` and `rust/` confirms the remaining selector sites are exactly uniproc/mp, Ray v1/v2, FT, engine API, DP reservation, shm broadcast, MoRIIO, and Rust managed-engine. The EngineCore handshake uses a fixed configured port and is not a dynamic selector site. Elastic EP creates its coordination store with final `TCPStore(..., port=0)` and its subsequent stateless groups use the held-socket path; it is not an extra probe-to-bind site. ## First-party example entrypoints (late audit expansion, 2026-08-07) The prior source re-enumeration intentionally scoped to `vllm/` and `rust/`; it did **not** establish whole-repository coverage. Searching first-party `examples/` found additional real probe-to-bind consumers, so the earlier “no additional path” wording is superseded. No open PR from the mandatory duplicate queries (`51275`, `rlhf get_open_port`, `NCCL rendezvous port`) covers them. | Entry points | Binder / handoff | Category | Status | |---|---|---:|---| | `examples/rl/rlhf_nccl.py`, `rlhf_async_new_apis.py`, `rlhf_sparse_nccl.py`, `rlhf_http_nccl.py` | trainer probes, then `NCCLWeightTransferEngine.trainer_init()` creates the rank-0 TCPStore | I | Uncovered; the trainer must hold a socket until `StatelessProcessGroup.create()` takes ownership, then publish its actual port before inference workers join. | | `examples/rl/rlhf_nccl_fsdp_ep.py` | driver probes FSDP `MASTER_PORT` for Ray workers; rank-0 FSDP worker binds later; its separate NCCL transfer endpoint is another delayed trainer bind | I + III; I | Uncovered; requires rank-0 actor-owned bootstrap and a held-socket handoff. | | `examples/rl/rlhf_ipc_fsdp_ep.py` | driver probes both FSDP and vLLM DP ports before spawning Ray actors/engine | I + III | Uncovered; same actor-owned/bootstrap design required. | | `examples/features/data_parallel/data_parallel_offline.py` | parent probes a single-node DP port before spawning the rank-0 child | I | Uncovered; reuse the single-node file rendezvous approach or pass an owned listener to the final binder. | These are example/application flows, not currently reached by the standard serving paths covered by #50999/#51018/#51404/#51033/#50969. They are nevertheless first-party runnable code and remain in scope for the desired empty probe-to-bind allowlist. - Example runtime evidence (2026-08-07): on both H200 hosts in `vllm/vllm-openai:nightly`, production `NCCLWeightTransferEngine.trainer_init()` was given a squatted value returned by `get_open_port()`. It reached `StatelessProcessGroup.create()` and failed at the real `listen_socket.bind()` with `[Errno 98] Address already in use`. This is the exact final-binder path used by the four simple RL NCCL examples and the transfer half of `rlhf_nccl_fsdp_ep.py`. - Child-binder evidence (2026-08-07): on both H200 nightly containers, a parent called production `get_open_port()`, a squatter took the released number, then a spawned child executed real `torch.distributed.init_process_group(\"gloo\", rank=0, world_size=1)` with that `MASTER_PORT`. Both failed with TCPStore `code: -98, name: EADDRINUSE`. This directly validates the parent→FSDP/offline child handoff class; the Ray-host variation is separately demonstrated by #50969. ### Test-only inventory A whole-tree search also finds `get_open_port()` in distributed/kernel/model tests and in test server fixtures. These are not user-facing serving paths, but most use the same probe→spawn/bind pattern and can be flaky under a competing local process (notably `tests/distributed/{test_dcp_a2a,test_mnnvl_alltoall,test_same_node,test_node_count,test_weight_transfer}.py`, `tests/models/test_vision.py`, kernel spawn helpers, and `tests/utils.py` server helpers). The tests are intentionally kept separate from the runtime migration table; they should be converted to a test-owned listener or file rendezvous in a dedicated test-infrastructure change, rather than claimed as covered by the production fixes. - Cross-host example evidence (2026-08-07): h200-0 used production `get_open_port()` to return `53567`; h200-1 then deliberately claimed `172.27.55.233:53567` and spawned a real Gloo rank-0 TCPStore child with that address. The child failed `code: -98, name: EADDRINUSE`. This is the exact driver→remote-FSDP-actor shape in the two FSDP/Ray examples, independently of the already-recorded Ray executor v1/v2 evidence. ## Example remediation design The example inventory should not be “fixed” by retrying `get_open_port()`: retries retain the same reservation gap. The feasible split is: 1. **NCCL weight-transfer examples**: add a small trainer-side reservation API to `nccl_common`/`NCCLWeightTransferEngine`. `prepare_trainer_rendezvous(host)` performs `bind_ephemeral(host)` and returns the actual `(host, port)` while retaining the socket in the trainer process/actor. Extend `trainer_init(..., listen_socket=...)` to pass that socket to `StatelessProcessGroup.create(..., listen_socket=...)`, which already hands its fd to `TCPStore`. The trainer publishes only the actual port to inference workers after binding. This covers the four simple NCCL examples and the transfer phase in `rlhf_nccl_fsdp_ep.py`. 2. **Single-node FSDP and offline examples**: replace their parent-selected TCP rendezvous with a unique `file://` store path, as in #50999, when the documented topology is explicitly one node. This needs no port reservation and covers the FSDP startup phase of `rlhf_nccl_fsdp_ep.py`, `rlhf_ipc_fsdp_ep.py`, and `data_parallel_offline.py`. 3. **Multi-node/external FSDP**: ordinary `torch.distributed.init_process_group(env://)` cannot take a pre-bound listener fd. The rank-0 actor must therefore own a bootstrap store/socket and publish the resulting endpoint through Ray/existing coordination before other ranks initialize, or the launcher must provide an explicit FD-passing protocol. Do not paper over this with driver-side scanning. This naturally belongs in a follow-up **weight-transfer/example rendezvous** PR, not #50969 (which fixes executor-owned Ray TCPStore host selection) and not #51404 (which fixes FT-owned groups). It may reuse `bind_ephemeral()` once #51404 is available, but should remain independently reviewable. - Full first-party tree closure (2026-08-07): searched `csrc/`, `vllm/`, `rust/`, `examples/`, `benchmarks/`, `tools/`, `docs/`, and `tests/` for port selectors, TCPStore, Rust listeners, and direct Internet sockets. `csrc/`, benchmarks, tools, and docs add no listener. The runtime results are exactly the core/Rust/external rows plus the new example rows; test-only uses are separately enumerated. `stateless_coordinator` is safe because rank 0 retains its sockets until TCPStore ownership, and API-server server-socket construction is a final bind rather than a probe.", "url": "https://github.com/vllm-project/vllm/issues/51275", "timestamp": "2026-08-12T16:23:27Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "RFC" ], "author": "aoshen02", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:51303", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Tracking][Spec Decode] Adaptive DSpark Bring-Up Tracker", "text": "Tracking issue to organize the adaptive verification support for DSpark speculative decoding ## Backend Support Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, the following backends are known to support variable lengths: - FLASH_ATTN - DSV4 Attention **TODO**: - Hybrid / Linear attention (requires significant effort) - - Mamba (for NVIDIA Nemotron models) - - GDN (for Qwen3.x models) - - Kimi KDA (for Kimi K3) - DSV3-style MLA - FlashInfer GQA: - - TRTLLM kernel support exists, just needs to be plugged into vLLM. Minor change. - - CUTLASS backend of FlashInfer may need additional support to avoid the cpu query lengths ## Online Profiling The initial version does profiling at startup to estimate the verification step time by replaying the recorded CUDA graphs. This is known to have some drift: \"Image\" We want to extend this to fully online profiling in the future. However, there are some challenges that make this tricky: - Online profiling may lead to \"dead spots\": one-time latency spikes make a certain batch size appear unattractive, leading to the adaptive verifier to never select that batch size, which leads to negative feedback and stale data limiting our range of profiled values. Generally there is an \"exploration vs exploitation\" problem here, as well as a requirement for some way to detect and eliminate outliers (such as those caused by JIT compilation) - Variable sequence lengths make it hard to measure a single reliable estimate of execution time for a given number of tokens. In practice we will want some way to account for this: either tracking measured points in 2D (num_tokens, max_seq_len or mean_seq_len), or estimating the overhead added per-token of KV reads ## Tuning Confidence Scores The confidence scores are not always perfectly calibrated. In particular, it seems like the deeper positions have higher variance and some bias in their expectation. The DSpark paper suggests [Sequential Temperature Scaling (STS)](https://arxiv.org/html/2607.05147v1) as an online solution to calibrating these estimates. I did not see much benefit when adding this feature, but we should experiment more thoroughly and assess the potential benefit of improved confidence estimates. Here are the results from one run with DSV4-Flash-0731: \"Image\" ## First-Class Observability ### Exposing Timings and Adaptive-SD Metrics Now that we are profiling different phases of the worker execution (model runner, proposer), and tracking the step time by batch size, we will want to expose these to the user for observability. This will help with performance tuning and development, as well as for monitoring of production deployments at scale. We will want to propagate the metrics to both the prometheus / frontend, as well as to the iteration-level logging feature. We will also want some way to report our other metrics for adaptive verification: confidence success rate (mismatch between \"expected\" AL and \"achieved\" AL), performance model reliability (difference between \"expected\" step time and \"actual\" step time), etc. ### Benchmarking Compatibility Adaptive speculative decoding will change what it means for speculative decoding to have high acceptance rate. AL is no longer a \"golden\" metric for success. As a start, this PR enables detailed, per-request and per-step speculative decoding metrics to be exposed in the OpenAI response. This can be used to report exact speculative decoding statistics and aggregated on the client to fit the use-case. https://github.com/vllm-project/vllm/pull/48915 ## Feature Compatibility - [ ] Logprob extraction support ([see here](https://github.com/vllm-project/vllm/pull/47808#discussion_r3723768178))", "url": "https://github.com/vllm-project/vllm/issues/51303", "createdAt": "2026-08-06T19:27:44Z", "updatedAt": "2026-08-13T13:07:09Z", "timestamp": "2026-08-13T13:07:09Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "speculative-decoding", "DSv4", "dflash", "kimi", "k3" ], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51313", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Kimi-K3 with --kv-cache-dtype fp8 is unusable on H200/Hopper — assertion demands use_prefill_query_quantization, but that flag is silently ignored on non-Blackwell devices", "text": "## Summary Serving **Kimi-K3** with `--kv-cache-dtype fp8` on **H200 (Hopper, SM 9.0)** fails at runtime with an assertion telling the user to enable `--attention-config '{\"use_prefill_query_quantization\": true}'`. **Setting that flag does not resolve the error** — the same assertion fires again. On any non-Blackwell (non-capability-100) device the flag is silently ignored, so the error is circular and unsatisfiable on Hopper. ## Environment - **vLLM:** nightly (`vllm/vllm-openai:nightly`); code line references below are from `main` @ `bc4e425883` (2026-08-06). - **Model:** `moonshotai/Kimi-K3` (nvidia MLA path, `vllm/models/kimi_k3/nvidia/mla.py`). - **Hardware:** 4 nodes × 8× **H200** (Hopper, compute capability **9.0**). - **Serve flags (relevant):** `--kv-cache-dtype fp8`, `--attention-backend FLASHMLA`, `--moe-backend humming`, `--all2all-backend deepep_v2`, `--tensor-parallel-size 8`, `--enable-expert-parallel`. ## Error ``` ERROR [multiproc_executor.py:1029] AssertionError: Kimi-K3 fp8 KV cache requires an fp8 prefill query; enable --attention-config '{\"use_prefill_query_quantization\": true}'. ``` Adding `--attention-config '{\"use_prefill_query_quantization\": true}'` to the serve command does **not** fix it; the identical assertion still fires. ## Root cause The assertion at `vllm/models/kimi_k3/nvidia/mla.py:700-704` requires `fp8_prefill` (prefill query dtype == fp8) for a plain `fp8` (non-`fp8_ds_mla`) KV cache: ```python elif is_quantized_kv_cache(self.kv_cache_dtype): assert fp8_prefill, ( \"Kimi-K3 fp8 KV cache requires an fp8 prefill query; enable \" \"--attention-config '{\\\"use_prefill_query_quantization\\\": true}'.\" ) ``` `fp8_prefill` derives from `q_data_type`, decided in `determine_prefill_query_data_type` (`vllm/model_executor/layers/attention/mla_attention.py:1807-1848`): ```python use_fp8 = ( is_quantized_kv_cache(vllm_config.cache_config.cache_dtype) # True for \"fp8\" and vllm_config.attention_config.use_prefill_query_quantization # True (flag set) and backend_supports_prefill_query_quantization() # <-- False on H200 ) if use_fp8: return fp8_dtype elif vllm_config.attention_config.use_prefill_query_quantization: logger.info_once( \"Unable to perform FP8 prefill attention when\" \" use_prefill_query_quantization is enabled. ...\" ) return model_dtype # <-- returns bf16 => fp8_prefill = False => assertion fires ``` `backend_supports_prefill_query_quantization()` (`mla_attention.py:1541-1566`) returns `False` on Hopper for **two** independent reasons: ```python if not current_platform.is_device_capability_family(100): # H200 is SM 9.0 -> False, returns early return False ... return backend_cls.get_name() in (\"FLASHINFER\", \"TRTLLM_RAGGED\", \"TOKENSPEED_MLA\") # FLASHMLA not in list ``` `is_device_capability_family(100)` is `(current_capability // 10) == (100 // 10)`, i.e. **Blackwell (SM 10.x) only**. H200 (SM 9.0 → `90 // 10 = 9 != 10`) fails immediately, before the backend check. **Net effect:** on H200 (and any non-Blackwell device / unsupported prefill backend), `use_prefill_query_quantization=true` can *never* make `q_data_type` fp8, so the Kimi-K3 plain-`fp8` KV-cache assertion is unsatisfiable. The user is instructed to enable a flag that has no effect on their hardware — a circular error with no escape path from the message itself. ## Expected behavior Any one (ideally the first) of: 1. **Fail fast with an accurate message at config-validation time**: reject `--kv-cache-dtype fp8` for Kimi-K3 when the device is non-Blackwell or the prefill backend isn't in the supported set, and point users at `fp8_ds_mla` (bf16 prefill query) or a supported backend/device — instead of a misleading runtime assertion that recommends an ineffective flag. 2. The Kimi-K3 assertion message (`mla.py:701-704`) should reflect the actual device/backend gate (Blackwell + FlashInfer / TRT-LLM Ragged / TokenSpeed), not recommend `use_prefill_query_quantization` unconditionally. 3. `determine_prefill_query_data_type` should **warn or raise** when `use_prefill_query_quantization=true` but the device/backend can't honor it, rather than silently downgrading to `model_dtype` and letting a downstream assertion fire. ## Workaround (for other users hitting this) Use `--kv-cache-dtype fp8_ds_mla` (DS-MLA fp8, 656B per-tile self-scaled) instead of `--kv-cache-dtype fp8`. That branch (`mla.py:681-699`) uses a **bf16 prefill query** (it asserts `not fp8_prefill`) and works on Hopper without any prefill query quantization. Drop `--attention-config '{\"use_prefill_query_quantization\": true}'` when switching.", "url": "https://github.com/vllm-project/vllm/issues/51313", "createdAt": "2026-08-06T21:38:39Z", "updatedAt": "2026-08-13T14:37:06Z", "timestamp": "2026-08-13T14:37:06Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "elvircrn", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51456", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: online FP8 (--quantization fp8) produces corrupted, non-EOS-terminating output on Qwen2.5-1.5B-Instruct", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect GPU models and configuration : GPU 0: NVIDIA H200 Nvidia driver version : 570.124.06 HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.6 [pip3] numpy==2.1.2 [pip3] torch==2.10.0 [pip3] transformers==5.14.1 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== vLLM Version : 0.19.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled ``` (Full output, including the complete `pip freeze` and NIC topology dump, omitted here for length -- available on request. GPU/CUDA/vLLM-relevant fields above are the complete, unedited values from the script.)
### 🐛 Describe the bug `vllm serve --quantization fp8` (online/on-the-fly W8A8, checkpoint not pre-quantized) produces incoherent, mixed-script output from the *first* generated token and essentially never reaches EOS -- not degraded-but-readable text, and not the kind of subtle numerical drift normal quantization noise causes. **Repro:** ```bash vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8 ``` ```bash curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\": \"Qwen/Qwen2.5-1.5B-Instruct\", \"messages\": [ {\"role\": \"system\", \"content\": \"You are a warm, emotionally present conversational companion in a live voice call. Reply in one or two short spoken sentences. Never use lists or markdown.\"}, {\"role\": \"user\", \"content\": \"I have been thinking about something all week and I cannot shake it.\"} ], \"max_tokens\": 80, \"temperature\": 0.7, \"stream\": false, \"seed\": 0 }' ``` **Expected** (and what the identical request against the same checkpoint with no `--quantization` flag returns): ``` \"content\": \"It's okay to spend time on thoughts that matter to you. What is the thought or idea you've been pondering? Sometimes talking through what's on your mind can help clear things up.\" \"finish_reason\": \"stop\" ``` **Actual**, `--quantization fp8`, same prompt/seed, re-confirmed live immediately before filing this issue: ``` \"content\": \"袅 解.Resolve\\\";} yabǃ qualidade感じるolving (...ñas andaLab yab standardized zweolving � lastnameolics\\tresolvefeatures ectñas learlab尼亚resolve yab.resolveresolve clock benchmarks anda ǃ yab！ijase第八ourcem原文地址olv巬 elimination eliminatedany세resolution qi!\\n\\n\\n /> `standard, synthetic, block` (default `standard`) - `synthetic_acceptance_rate`: renamed to `synthetic_acceptance_rates` (list), with `synthetic_acceptance_length` noted as mutually exclusive ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions. Update: fix PR submitted - https://github.com/vllm-project/vllm/pull/51611", "url": "https://github.com/vllm-project/vllm/issues/51609", "createdAt": "2026-08-10T03:45:16Z", "updatedAt": "2026-08-13T10:56:29Z", "timestamp": "2026-08-13T10:56:29Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "documentation" ], "author": "qwerqwerqwe8688-jpg", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51737", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Fix]: MTP speculative decoding crashes with RuntimeError on heterogeneous per-layer head_dim, fixed below and created a PR", "text": "### Bug Description Serving Gemma 4 31B with its native assistant model (`google/gemma-4-31B-it-assistant`) on `vllm/vllm-openai:latest` crashes during parameter loading with `RuntimeError: start (0) + length (4096) exceeds dimension size (2048)`. ### Root Cause Gemma 4 uses heterogeneous attention layers (45 sliding layers with `head_dim=256`, 15 full layers with `global_head_dim=512`). During MTP draft layer parameter loading, `_ColumnvLLMParameter.load_qkv_weight` uses global `global_head_dim=512` for all layers, slicing past the 2048-dim bounds of sliding layers. ### Reproduction ```bash docker run --gpus all --ipc host vllm/vllm-openai:latest \\ --model RedHatAI/gemma-4-31B-it-FP8-dynamic \\ --speculative-config '{\"method\": \"draft_model\", \"model\": \"google/gemma-4-31B-it-assistant\", \"num_speculative_tokens\": 8}' ``` ### Traceback ```text File \"vllm/model_executor/parameter.py\", line 195, in load_qkv_weight param_data = param_data.narrow(self.output_dim, shard_offset, shard_size) RuntimeError: start (0) + length (4096) exceeds dimension size (2048) ``` ### Proposed Fix **`vllm/model_executor/parameter.py`**: ```python max_param = param_data.shape[self.output_dim] max_loaded = loaded_weight.shape[self.output_dim] shard_offset = min(shard_offset, max_param) shard_size = min(shard_size, max_param - shard_offset) loaded_start = min(shard_id_int * shard_size, max_loaded) shard_size = min(shard_size, max_loaded - loaded_start) if shard_size <= 0: return ``` **`vllm/model_executor/model_loader/weight_utils.py`**: ```python if param.size() != loaded_weight.size() and param.dim() == 1 and loaded_weight.dim() == 1: min_len = min(param.size(0), loaded_weight.size(0)) param.data[:min_len].copy_(loaded_weight[:min_len]) ```", "url": "https://github.com/vllm-project/vllm/issues/51737", "createdAt": "2026-08-10T22:21:56Z", "updatedAt": "2026-08-13T08:19:51Z", "timestamp": "2026-08-13T08:19:51Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "quivent", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51744", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm/vllm-openai:latest fails to start Gemma4 with Transformers 5.15.0", "text": "### Your current environment ```text Docker image: vllm/vllm-openai:latest vLLM: 0.27.0 Transformers: 5.15.0 Model: yasu-oh/gemma-4-31B-it-qat-NVFP4 Tensor parallel size: 2 KV cache dtype: fp8 ``` The Transformers version in the image can be confirmed with: ```bash docker run --rm \\ --entrypoint python3 \\ vllm/vllm-openai:latest \\ -c 'import vllm, transformers; print(\"vLLM:\", vllm.__version__); print(\"Transformers:\", transformers.__version__)' ``` Output: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` ### 🐛 Describe the bug The current official release image `vllm/vllm-openai:latest` contains: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` When serving: ```text yasu-oh/gemma-4-31B-it-qat-NVFP4 ``` vLLM fails during startup with: ```text transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via config.per_layer_config[i].head_dim. ``` The exception occurs in: ```text vllm/transformers_utils/model_arch_config_convertor.py ``` while accessing the Gemma4 `head_dim`. ### Workaround Downgrading only Transformers to 5.14.1 while keeping the same `vllm/vllm-openai:latest` base image and serving configuration resolves the issue. ```dockerfile FROM vllm/vllm-openai:latest RUN /usr/bin/python3 -m pip install --no-cache-dir 'transformers==5.14.1' ``` With Transformers 5.14.1: * the same model loads successfully, * the vLLM engine initializes successfully, * the API server starts successfully, * `/v1/chat/completions` returns HTTP 200, * 32 concurrent requests run successfully. ## Related PRs * #49797 ? Fix Gemma 4 for upcoming Transformers version * #49959 ? Handle Transformers 5.15 Gemma4 schema ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51744", "createdAt": "2026-08-11T00:42:58Z", "updatedAt": "2026-08-13T14:26:47Z", "timestamp": "2026-08-13T14:26:47Z", "metrics": { "reactions": 5, "comments": 14 }, "labels": [ "bug" ], "author": "yasu-oh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51758", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: upgrade vllm from 0.26.0 to 0.27.0 run deepseek v4 flash error", "text": "### Your current environment
The output of python collect_env.py ```text python collect_env.py Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : 18.1.3 (1ubuntu1) CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by conda-forge | (main, Mar 5 2026, 16:50:00) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-6.8.0-137-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.3.73 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 1: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 2: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 3: NVIDIA RTX PRO 6000 Blackwell Server Edition Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8488C CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 21% CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 192 MiB (96 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchcodec==0.15.0 [pip3] torchvision==0.28.0 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] flashinfer-python 0.6.16.post3 pypi_0 pypi [conda] nccl4py 0.3.1 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.1.3 pypi_0 pypi [conda] nvidia-cuda-cccl 13.3.3.4.1 pypi_0 pypi [conda] nvidia-cuda-crt 13.3.73 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvcc 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvdisasm 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.20.0.48 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.27.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-core 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu12 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu13 4.6.0 pypi_0 pypi [conda] nvidia-ml-py 13.610.43 pypi_0 pypi [conda] nvidia-nccl-cu13 2.29.7 pypi_0 pypi [conda] nvidia-nvjitlink 13.3.33 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] nvidia-nvvm 13.3.73 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] tokenspeed-triton 3.8.10.post20260721 pypi_0 pypi [conda] torch 2.13.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchcodec 0.15.0 pypi_0 pypi [conda] torchvision 0.28.0 pypi_0 pypi [conda] transformers 5.15.0 pypi_0 pypi [conda] triton 3.7.1 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.0 vLLM Build Flags: CUDA Archs: Blackwell; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE NODE 48-95,144-191 1 N/A GPU1 NODE X NODE NODE 48-95,144-191 1 N/A GPU2 NODE NODE X NODE 48-95,144-191 1 N/A GPU3 NODE NODE NODE X 48-95,144-191 1 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== TORCH_CUDA_ARCH_LIST=Blackwell LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda-12.8/lib64 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_yafeng_wang ```
### 🐛 Describe the bug [bugreport.txt](https://github.com/user-attachments/files/30921135/bugreport.txt) ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51758", "createdAt": "2026-08-11T03:42:26Z", "updatedAt": "2026-08-13T04:16:50Z", "timestamp": "2026-08-13T04:16:50Z", "metrics": { "reactions": 0, "comments": 17 }, "labels": [ "bug" ], "author": "WangHHY19931001", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51771", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "EAGLE/MTP block drop + prefix caching is untested for hybrid models with ≥3 attention groups (DeepSeek-V4-Flash + DSpark lands there)", "text": "## Summary #33524 fixed the \"0% prefix cache hits with hybrid attention + EAGLE\" bug for the 2-attention-group (GPT-OSS) shape, and its author noted in the PR body that more complex models were not addressed: > However, it is worth noting that for more complicated models with multiple attention groups, this PR does not fully address the EAGLE spiral block drop issue either. A general fix to this issue cannot directly cache the hit_blocks list returned by each attention type, because SWA attn and Mamba-style attn do not follow the downward-closed property (cache hit at token j does not indicate cache hit at i where i < j). So we need some more fundamental changes there. > > **Fortunately, we don't have such complex models yet, so this is not a huge issue for now.** There is such a model now, and it is one vLLM ships. **DeepSeek-V4-Flash with DSpark speculative decoding builds 4 attention groups** — 1 full MLA (`block_size=256`) plus 3 sliding-window-family groups (`block_size` 64 / 4 / 8) — and `SpeculativeConfig.use_eagle()` returns `True` for `method=\"dspark\"`, so the EAGLE block-drop path is active. `is_simple_hybrid` is `False`, so it takes the general convergence loop rather than #33524's early-break. This is a supported configuration on main: `_annotate_eagle_groups_deepseek_v4` lives in `vllm/v1/core/kv_cache_utils.py`. **Nothing in CI exercises that path.** Every EAGLE-enabled prefix-caching test in `tests/v1/core/test_prefix_caching.py` resolves to at most 2 attention groups, and the file says so itself: ```python # Test cases with eagle enabled: Only test a single simple case for now. # - 2 groups: 1 full + 1 other _EAGLE_HYBRID_MODEL_TEST_CASES = [ # 2 groups: 1 full + 1 other pytest.param([\"full\", \"sliding_window\"], 3, id=\"2g-full+sw\"), ] ``` ```python def test_prefill_hybrid_model_combinations_eagle(...): \"\"\" Test prefix caching with hybrid models (1 full attn + 1 other) with EAGLE. More complex hybrid models with EAGLE are not yet supported (see issue #32802). \"\"\" ``` (The tests that look like they have more groups do not: `make_kv_cache_config_hybrid_model` emits 3 KV groups but two of them share one spec, and `test_eagle_grouped_swa_siblings_use_same_cache_mask` likewise builds `swa_main` and `swa_mtp` from the same `swa_spec` object. `verify_and_split_kv_cache_groups` dedupes by spec, so both land at 2 attention groups and hit the `is_simple_hybrid` early break.) We hit a **total (0.0%) prefix-cache loss** on this configuration, on a fork whose base sits between #42258 and #44082. We think **main is fine for this model shape** — we worked the arithmetic and #44082's mask does admit the blocks the read gate needs, for all three of our sliding-window groups (table in §4). The reason for filing anyway: our base carries #42258's admission mask without #44082's fix, and on that combination this model class loses prefix caching entirely. Main carried the same combination between 2026-05-15 and 2026-06-02; we have not run main from that window, so we can only say this is what we would expect it to have done there. Either way nothing would have reported it — #44082 shipped without adding a test (its only change under `tests/` is the `use_eagle` → `drop_eagle_block` rename) — and the hole stays open for the next change to the SWA admission mask or to the convergence loop. So this is a test-coverage issue with a worked instance attached, not a bug report against main. We are happy to send the regression test as a PR — see §8. --- ## 1. Configuration Runtime dump from `HybridKVCacheCoordinator` construction (instrumented print, real engine): ``` num_attention_groups=4 lcm_block_size=256 grp0: spec=MLAAttentionSpec block_size=256 isinstance_FullAttentionSpec=True kv_group_ids=[0] manager=FullAttentionManager grp1: spec=SlidingWindowMLASpec block_size=64 isinstance_FullAttentionSpec=False kv_group_ids=[1, 2, 3] manager=SlidingWindowManager grp2: spec=SlidingWindowMLASpec block_size=4 isinstance_FullAttentionSpec=False kv_group_ids=[4] manager=SlidingWindowManager grp3: spec=SlidingWindowMLASpec block_size=8 isinstance_FullAttentionSpec=False kv_group_ids=[5] manager=SlidingWindowManager is_simple_hybrid=False ``` 6 KV cache groups collapse to 4 attention groups. Sliding windows: grp1 `sliding_window=128` (from `hf_config.sliding_window`); grp2/grp3 come from the compressor, where `coff = 1 + (compress_ratio == 4)` and `sliding_window = coff * compress_ratio` gives **w=8 for C4** and **w=128 for C128**. Model config: `sliding_window=128`, `num_hidden_layers=43`, `compress_ratios=[0,0,4,128,4,128,...]`, `dspark_target_layer_ids=[40,41,42]`, `num_speculative_tokens=7`. Two properties are what put this outside the tested envelope: 1. **`len(attention_groups) == 4`** — `is_simple_hybrid` requires exactly 2 with `attention_groups[0].spec` a `FullAttentionSpec`, so the convergence loop runs to fixpoint. 2. **Three *distinct* sliding-window specs**, with `block_size` 64/4/8 against `alignment_tokens = lcm = 256`. All three satisfy `block_size != alignment_tokens` and `tail < per_segment`, so both the SWA admission mask and the alignment gate in `find_longest_cache_hit` are live for each of them, at three different moduli. --- ## 2. Measured impact Two independent harnesses, each a controlled A/B on our base with only the EAGLE block drop toggled (env switch that empties `eagle_group_ids`; nothing else changes). Same server build, same `ctx=262144`, same `max-num-batched-tokens=2048`. **Harness A — 5-turn agentic coding session, identical prompts, sequential:** | | drop enabled (stock) | drop disabled | |---|---:|---:| | Prefix cache hit rate | **0.0%** | **72.7%** | | Wall clock, 5 turns | 64.27 s | 30.61 s | | Mean acceptance length | 4.80 | 4.76 | **Harness B — benchmark script:** | | drop enabled (stock) | drop disabled | |---|---:|---:| | Prefix cache hit rate | **0.0%** | **48.6%** | | TTFT, repeated shared prefix | 12355.6 ms | 995.1 ms | | TTFT, short request w/o shared prefix (control) | 3499.6 ms | 3509.5 ms | | Mean acceptance length | 2.38 | 2.26 | The no-shared-prefix control moving 0.3% confirms the effect is confined to the prefix-cache path. Hit rates are load-dependent — 72.7% and 48.6% come from different workloads and should not be compared to each other, only within a column pair. Acceptance-length cost of disabling the drop spans **−0.8% (A) to −5.0% (B)**, n=1 per arm on different workloads. We would not describe that as lossless; see §7. --- ## 3. Mechanism on our base (why it is 0% and not ~3%) Worth writing down because the failure mode differs from what #32802's title suggests. #32802 describes a spiral of progressive block drops. Here it is a **single-step collapse**: one group returns zero blocks on its very first query, `curr_hit_length` goes to 0, and the loop converges at 0 on the next iteration. The `eagle_verified` machinery from #32802 is present on our base and never gets a chance to engage, because the failure needs only one drop. The read side (identical on main modulo the `drop_eagle_block` rename): ```python # SlidingWindowManager.find_longest_cache_hit if num_contiguous_blocks == 0 and block_size != alignment_tokens: post_pop_blocks = i if use_eagle else i + 1 if (post_pop_blocks * block_size) % alignment_tokens != 0: continue ``` Under EAGLE the first accepted block of a run must sit at `i % per_segment == 0`. The write side on our base (`_cache_block_mask`, as introduced by #42258) admits only `i % per_segment >= per_segment - tail`: ```python per_segment = alignment_tokens // self.block_size tail = cdiv(self.sliding_window - 1, self.block_size) if tail >= per_segment: return None skip = per_segment - tail return [i % per_segment >= skip for i in range(num_cached_blocks, num_full_blocks)] ``` For every sliding-window group in this model the two sets are **disjoint**, so the right-to-left scan `continue`s past every cached block, `num_contiguous_blocks` never leaves 0, `match_found` stays `False`, and the group returns 0 blocks: | group | `block_size` | window | `per_segment` | `tail` | `skip` | mask admits `i % per_segment ∈` | gate needs run start at | |---|---:|---:|---:|---:|---:|---|---| | grp1 SWA MLA | 64 | 128 | 4 | 2 | 2 | {2, 3} | `i % 4 == 0` | | grp2 C4 | 4 | 8 | 64 | 2 | 62 | {62, 63} | `i % 64 == 0` | | grp3 C128 | 8 | 128 | 32 | 16 | 16 | {16, …, 31} | `i % 32 == 0` | This also explains why the GPT-OSS shape never showed this symptom even before #44082: the gate is guarded by `block_size != alignment_tokens`, so when all groups share a page size the whole branch is skipped. The two bugs are genuinely different, which is why carrying #32802's fix did not help us. --- ## 4. Is main affected? Our reading: no, for this shape #44082 changed the write side so that the admitted set contains what the gate asks for — `need = tail + 1` under EAGLE and the retained run shifted one block right: ```python need = cls._contiguous_blocks_for_hit(window_size=..., block_size=block_size, use_eagle=use_eagle) shift = 1 if use_eagle else 0 ... if i >= shift and (i - shift) % per_segment >= per_segment - need: mask[i - start_block] = True ``` plus `HybridKVCacheCoordinator.cache_blocks` making the lookahead block eligible: ```python if manager.use_eagle and aligned_num_computed_tokens > 0: num_tokens_to_cache = min(num_computed_tokens, aligned_num_computed_tokens + manager.block_size) ``` Worked out for our three groups (a hit needs blocks `i, i-1, …, i-(need-1)` with `i % per_segment == 0`): | group | `per_segment` | `need` | main admits `i % per_segment ∈` | required run | satisfied on main? | |---|---:|---:|---|---|---| | grp1 SWA MLA | 4 | 3 | {0, 2, 3} | 0, 3, 2 | **yes** | | grp2 C4 | 64 | 3 | {0, 62, 63} | 0, 63, 62 | **yes** | | grp3 C128 | 32 | 17 | {0, 16, …, 31} | 0, 31, …, 16 | **yes** | Versus our base, where the same three rows are all \"no\". So we are not claiming a live defect on main — we are claiming this arithmetic is currently load-bearing for a shipped model and is asserted by no test. Note also that this is only the SWA branch of `reachable_block_mask`; the `retention_interval` / `reachable_boundaries` branches added later interact with the same invariant and are likewise unexercised in combination with 4 groups. --- ## 5. Reproduction This reproduces on CPU with no model and no GPU, exercising only `vllm/v1/core/kv_cache_coordinator.py` and `single_type_kv_cache_manager.py`. We drove the real `HybridKVCacheCoordinator.find_longest_cache_hit`, the real `BlockPool`, the real `SlidingWindowManager.find_longest_cache_hit` and the real `cache_blocks`, hand-building four group specs to match the runtime dump in §1, caching an 8192-token prefix, then querying the same prefix. Which group carries the EAGLE flag was enumerated A/B: | EAGLE-flagged group | hit tokens / 8192 | hit rate | |---|---:|---:| | grp0 full MLA (bs=256) | 7936 | 96.9% | | grp1 SWA MLA (bs=64, w=128) | 0 | **0.0%** | | grp2 C4 (bs=4, w=8) | 0 | **0.0%** | | grp3 C128 (bs=8, w=128) | 0 | **0.0%** | | all four (coordinator fallback) | 0 | **0.0%** | | none (drop disabled) | 8192 | 100.0% | Prefix-length sweep, to show this is structural rather than a corner case: | prefix tokens | all flagged | grp0 only | grp1 only | no EAGLE | |---:|---:|---:|---:|---:| | 512 | 0 | 256 | 0 | 512 | | 1,024 | 0 | 768 | 0 | 1,024 | | 2,048 | 0 | 1,792 | 0 | 2,048 | | 8,192 | 0 | 7,936 | 0 | 8,192 | | 32,768 | 0 | 32,512 | 0 | 32,768 | | 131,072 | 0 | 130,816 | 0 | 131,072 | The `grp0 only` column is the ~3% single-block cost one would expect from an EAGLE drop on a full-attention group (`prefix − 256`). The 0% columns are the disjointness above. That contrast is the part worth turning into a test: **a correct EAGLE drop costs one block; a broken block-admission mask costs everything, and the two are not distinguishable from the aggregate hit-rate metric.** We never captured `eagle_group_ids` from a live engine in the failing arm, so which group actually carries the flag in production is inferred rather than observed: the measured 0.0% rules out \"grp0 only\" (that arm gives 96.9%), leaving \"at least one sliding-window group is flagged\" — by `_annotate_eagle_groups_deepseek_v4` or by the coordinator fallback in §6. Nothing above depends on resolving it, since every sliding-window placement and the fallback all give 0%. --- ## 6. Two secondary observations on main Both are pre-existing and neither is dangerous today given §4; flagging them because they are what turns \"annotation missed\" into \"prefix caching silently off\". **a) The all-groups fallback.** `kv_cache_coordinator.py`: ```python if use_eagle and not self.eagle_group_ids: self.eagle_group_ids = set(range(len(kv_cache_config.kv_cache_groups))) ``` For a model where no group gets annotated, this flags every group as EAGLE. On main that costs about one block per group; on a base without #44082 it is indistinguishable from the worst case (row 5 of the §5 table) — i.e. on a sliding-window model it is the most aggressive choice available, not the most conservative one. If a missing annotation is a bug, failing loudly seems better than silently enabling the drop everywhere. **b) The annotation is still keyed to one model string.** `_annotate_eagle_groups_deepseek_v4` returns early unless some spec has `model_version == \"deepseek_v4\"`, and picks its group via `next(reversed(kv_cache_spec))` with an in-source `FIXME(yifan): avoid/generalize this hacky check.` Every other multi-group architecture therefore reaches (a). #41640 proposes moving `is_eagle` onto `AttentionSpec`; it has been open since 2026-05-04. --- ## 7. Workaround we are running, and what it costs Disclosed for completeness, **not** recommended as a fix: we added an env switch that empties `eagle_group_ids`, i.e. disables the block drop entirely. That is the \"drop disabled\" column in §2 — it restores the hit rate because reader and writer agree again, at the cost of the drop's actual purpose. We cannot certify it as output-preserving. We ran a token-level equivalence gate and **it produced no signal**: at `temperature=0` our non-speculative reference arm failed to reproduce *itself* across runs on 3 of 4 prompts, so the harness is non-deterministic (batch-composition dependent numerics on our side — we are not making any claim about vLLM here) and the comparison against the patched arm is meaningless. We are reporting this so the acceptance-length deltas in §2 are not read as an equivalence argument. They are not one. If you are on a base between #42258 and #44082 with a >2-group hybrid model, the right move is to take #44082, not this switch. --- ## 8. What we would suggest The minimal ask is regression coverage for the general convergence path under EAGLE — i.e. **≥3 distinct attention-group specs so `is_simple_hybrid` is `False`** — asserting a nonzero hit rate. Two pieces, because they cover different things: **a) The convergence loop with more than one non-full group.** `_make_hybrid_kv_cache_config` already supports `\"sliding_window\"` (w=2·bs) and `\"sliding_window_large\"` (w=4·bs) as distinct specs, so this is a one-line addition to the table that currently reads \"Only test a single simple case for now\": ```python _EAGLE_HYBRID_MODEL_TEST_CASES = [ pytest.param([\"full\", \"sliding_window\"], 3, id=\"2g-full+sw\"), pytest.param([\"full\", \"sliding_window\", \"sliding_window_large\"], ..., id=\"3g-full+sw+sw_large\"), ] ``` **b) The block-admission mask at more than one modulus.** (a) alone does not reach the failure in §3. `make_kv_cache_manager` derives `scheduler_block_size` as the lcm of the group block sizes, and `_cache_hit_alignment_tokens` returns exactly that whenever `enable_partial_hash_hits` is `False` (no Mamba group). Since every group `_make_hybrid_kv_cache_config` builds shares one `block_size`, that lcm *is* the block size, so `alignment_tokens == block_size` and the gate's `block_size != alignment_tokens` guard skips the branch entirely. The mask/gate agreement is only exercised when per-group block sizes differ. `test_eagle_swa_alignment_caches_extra_block` is the right template — full at `4*block_size` against SWA at `block_size` gives alignment `4*block_size` — and what is missing is a **third** sliding-window spec at yet another block size, so that `is_simple_hybrid` is `False` *and* two distinct `per_segment` values are live at once. The DeepSeek-V4-Flash moduli are a good stress case for (b): alignment 256 against block sizes 64/4/8, giving `per_segment` 4/64/32 and `tail` 2/2/16 — including one group where `tail` is large relative to `per_segment` and one where it is tiny. (b) would have caught the mask/gate disagreement in §3 on its first CI run. **We are happy to open that PR if maintainers want it** — we have the harness and the shape already, and we can write it against main's API rather than our base. Beyond the test, we do not have a general-fix proposal we would defend. We explored moving the drop out of per-group lookup entirely (reduce with the drop disabled, then step back one `alignment_tokens` and re-query, which needs no downward-closed property from any manager), but it degenerates for prefixes shorter than `alignment_tokens` and we have not checked it against Mamba-style managers, so we mention it only as a direction rather than a recommendation. --- ## Environment - 8× V100-SXM2-32GB (SM70), TP8, DGX-1 - DeepSeek-V4-Flash, `fp8_ds_mla`, DSpark speculative decoding, `num_speculative_tokens=7`, `ctx=262144`, `max-num-batched-tokens=2048` - vLLM `1.2.3.dev3+g6f946b603a`, from the 1cat SM70 fork - Base predates #44082: `_cache_block_mask` present (#42258), while `reachable_block_mask`, `_contiguous_blocks_for_hit`, `drop_eagle_block`, `eagle_margin` and `enable_partial_hash_hits` are all absent from `single_type_kv_cache_manager.py` and `kv_cache_coordinator.py` The fork is our environment, not the subject. V100/SM70 specifics play no part in any of the above — the mechanism, the arithmetic and the coverage gap are all in architecture-independent KV-cache code, and the §4/§5 analysis is against main's sources. ## Limitations - Single machine, single checkpoint, single fork. No upstream-main run of DeepSeek-V4-Flash + DSpark + prefix caching; §4 is arithmetic against main's sources, not an execution of main. - §5 uses real vLLM classes but hand-built group specs matched to the §1 runtime dump; it is not a full engine run. - Which group carries the EAGLE flag in production is inferred by elimination, not observed (§5). - Output equivalence under the workaround is **not** established (§7). - Each A/B arm is n=1; acceptance-length deltas span −0.8% to −5.0% across the two workloads. --- cc @ivanium (authored #33524 / #42258 / #44082 and the caveat this answers), @heheda12345 (KV cache manager), @benchislett (spec decode)", "url": "https://github.com/vllm-project/vllm/issues/51771", "createdAt": "2026-08-11T06:22:08Z", "updatedAt": "2026-08-13T10:37:28Z", "timestamp": "2026-08-13T10:37:28Z", "metrics": { "reactions": 0, "comments": 5 }, "labels": [], "author": "kezboard233", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51776", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: CuTe FA4 forward (flash_fwd.py SM80/SM120) fails on first call: NameError 'mDynamicCausal' and missing self.is_split_kv", "text": "### Summary Two independent Python-level bugs make `vllm/vllm_flash_attn/cute/flash_fwd.py` (`FlashAttentionForwardSm80`, and by inheritance `FlashAttentionForwardSm120`) fail on the very first call, before any kernel runs. They look like the code path has never been executed: SM80-class GPUs use the C++ FA2/FA3 kernels, so this CuTe DSL path is effectively dead there — but on **sm_120** it is selected, and anything routing through `vllm.vllm_flash_attn.cute.flash_attn_varlen_func` hits it. Both reproduce on unmodified `vllm/vllm-openai:v0.26.0` with a ~25-line script: single GPU, no model, no distributed init, batch size 1. One-line fixes for both are suggested below. Found while investigating a workaround for #51405 (paged KV not implemented for SM 12.0 in this same path). These two bugs are independent of that and of any particular model. ### Environment - 2× NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), **sm_120**, PCIe-only - Driver 595.84, CUDA 13.2, Ubuntu 25.10 - vLLM `0.26.0` (`vllm/vllm-openai:v0.26.0`), also reproduced on `0.26.1.dev1` built from source - `nvidia_cutlass_dsl` as shipped in that image ### Reproducer ```python import torch from vllm.vllm_flash_attn.cute import flash_attn_varlen_func SEQLEN_K, BATCH = 1024, 1 NUM_HEADS, NUM_KV_HEADS, HEAD_DIM = 16, 4, 128 dev, dtype = torch.device(\"cuda:0\"), torch.bfloat16 total_q = SEQLEN_K * BATCH q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device=dev, dtype=dtype) k = torch.randn(BATCH, SEQLEN_K, NUM_KV_HEADS, HEAD_DIM, device=dev, dtype=dtype) v = torch.randn_like(k) out = flash_attn_varlen_func( q=q, k=k, v=v, cu_seqlens_q=torch.arange(0, total_q + 1, SEQLEN_K, device=dev, dtype=torch.int32), seqused_k=torch.full((BATCH,), SEQLEN_K, device=dev, dtype=torch.int32), max_seqlen_q=SEQLEN_K, page_table=None, softmax_scale=HEAD_DIM ** -0.5, causal=True, num_splits=1, return_lse=False, ) torch.cuda.synchronize() print(\"OK\", (out[0] if isinstance(out, tuple) else out).shape) ``` ```bash docker run --rm --gpus all -e CUDA_VISIBLE_DEVICES=0 -v \"$PWD:/work\" \\ --entrypoint python3 vllm/vllm-openai:v0.26.0 /work/repro.py ``` Today this prints the Bug 1 error. With the Bug 1 patch applied it prints the Bug 2 error. With both applied it prints `OK`. --- ## Bug 1 — `NameError: name 'mDynamicCausal' is not defined` ``` error: NameError in `__call__`: name 'mDynamicCausal' is not defined --> /usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/cute/flash_fwd.py:840 = note: Caused exception: name 'mDynamicCausal' is not defined ``` `mDynamicCausal` is a parameter of the `__call__` **method** (`flash_fwd.py:681`): ```python mDynamicCausal: Optional[cute.Tensor] = None, ``` but it is referenced from the body of `@cute.kernel def kernel(...)` at `flash_fwd.py:840`: ```python psc = mDynamicCausal[batch_size] if const_expr(mDynamicCausal is not None) else None ``` `kernel` is a separate method of the class, not a closure nested inside `__call__`, so the name is not in scope there. The launch site (`self.kernel(...)`, `flash_fwd.py:761`) passes its arguments positionally and ends at `output_scale`, so it does not supply it either. Suggested fix — append to both the kernel signature and the positional launch list, leaving the order of existing arguments untouched: ```diff aux_data, fastdiv_mods, output_scale, + mDynamicCausal, ).launch( ``` ```diff aux_data: AuxData = AuxData(), fastdiv_mods=None, output_scale: Optional[cute.Tensor] = None, + mDynamicCausal: Optional[cute.Tensor] = None, ): # Thread index, block index ``` ## Bug 2 — `AttributeError: 'FlashAttentionForwardSm120' object has no attribute 'is_split_kv'` With Bug 1 fixed, the same script fails with: ``` AttributeError: 'FlashAttentionForwardSm120' object has no attribute 'is_split_kv' ``` `flash_fwd.py` reads `self.is_split_kv` in four places (≈ lines 365, 381, 408, 413), but `FlashAttentionForwardSm80.__init__` (`flash_fwd.py:620`) never assigns it. Both sibling implementations do: - `flash_fwd_sm90.py:70` — `self.is_split_kv = is_split_kv` - `flash_fwd_sm100.py:188` — `self.is_split_kv = is_split_kv` (The `is_split_kv` at `flash_fwd.py:194` is a parameter of `_check_type`, not of `__init__`.) The intended value for this path looks unambiguous: the kernel launch hardcodes `False, # is_split_kv`, and `interface.py` already asserts `not is_split_kv` for SM 12.0. Suggested fix: ```diff def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) assert self.output_quant_key is None, ( f\"Fused quant output not implemented for {type(self).__name__}\" ) + self.is_split_kv = False ``` --- ### After both fixes The kernel compiles and runs, and returns finite, plausible output across the configurations I tried (batch 1–8, seqlen_k 128–8192, causal, sliding window, decode-shaped q, with and without `score_mod`/`aux_tensors`). One regime still faults with an out-of-bounds device read — `aux_tensors` combined with batch > 1 — which I have written up separately in a comment on #51405, since reaching it at all requires working around that issue's paged-KV gap. I have sm_120 hardware available and am happy to test patches.", "url": "https://github.com/vllm-project/vllm/issues/51776", "createdAt": "2026-08-11T07:15:36Z", "updatedAt": "2026-08-12T19:42:54Z", "timestamp": "2026-08-12T19:42:54Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "IgnacyPawlak", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:51788", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Suffix decoding on GPU (`suffix_gpu`) — GPU-resident suffix decoding compatible with async scheduling", "text": "## Summary Suffix decoding gets the best acceptance lengths among model-free drafters on repetitive and agentic workloads, but the in-tree implementation (`method=\"suffix\"`, #25784, from Arctic Inference) is CPU-side and rejected by the async-scheduling whitelist. This RFC proposes `suffix_gpu`: a GPU-resident suffix drafter that follows the same device-state drafter contract as `ngram_gpu` (#29184), so suffix decoding composes with async scheduling — no host sync on the draft path, drafts come from the previous step's device sampled ids. The in-tree diff is small (config + a proposer wrapper + runner wiring + unit tests, ~500 lines). The drafter core (suffix automaton over device tensors, cross-request global index with background rebuilds, fused Triton kernels, CUDA-graph capture) lives in a standalone package ([SuffixGPU](https://github.com/zip95297/SuffixGPU)), imported lazily — vLLM works without it installed, same pattern as the CPU suffix method's `arctic-inference` dependency. ## Motivation - #38069 asked for n-gram/suffix speculative decoding in the new runner under the condition that it stays compatible with async scheduling; it went stale, but the gap is still there. - Async scheduling is on the way to being the default. `method=\"suffix\"` forces `--no-async-scheduling` today (`config/vllm.py` whitelist: \"async scheduling is only supported with EAGLE/MTP/Draft Model/NGram GPU/DSpark\"), giving up CPU/GPU overlap exactly where spec decode helps most: high-concurrency TPOT. - The reason is structural, not incidental: the CPU suffix tree needs the committed host token ids each step, and under async scheduling the scheduler runs one step ahead of the worker, so those ids do not exist on the host at schedule time. A GPU-state drafter sidesteps this by drafting from the previous step's sampled ids that are already on device — the mechanism #29184 built for `ngram_gpu`. So: keep suffix decoding's draft quality, get async scheduling's overlap, by moving the drafter's state and compute onto the GPU. ## Design `suffix_gpu` plugs into the existing `ngram_gpu` machinery; no scheduler changes. - **Config** (`config/speculative.py`): `method=\"suffix_gpu\"`, reusing the `suffix_decoding_*` knobs (max_tree_depth, max_spec_factor, min_token_prob, max_cached_requests) plus GPU-specific ones (`suffix_gpu_global_capacity`, `suffix_gpu_delta_capacity`, `suffix_gpu_max_occurrences`, `suffix_gpu_use_cuda_graph`, `suffix_gpu_ingest_chunk`). A `use_gpu_state_drafter()` helper groups `ngram_gpu` + `suffix_gpu` for the shared device-state paths; the async whitelist admits `suffix_gpu`. - **Proposer** (`v1/spec_decode/suffix_proposer_gpu.py`): wraps the SuffixGPU drafter behind the `ngram_gpu` contract — same `update_token_ids_ngram` device bookkeeping (borrowed verbatim), same `propose(k, num_tokens_no_spec, token_ids, sampled, counts) -> (drafts [B,k], valid counts [B])`, same async D2H of per-request valid draft counts feeding the existing worker-side trim (`update_scheduler_for_invalid_drafts`). Two suffix-specific additions: - a per-request local matcher plus a cross-request global suffix index (the analogue of the CPU method's global tree / `max_cached_requests`), fed off the critical path: in-flight responses are ingested on a side stream, finished requests are final-flushed before their persistent batch rows are reused; - the whole draft chain is captured into CUDA graphs, bucketized by batch size, pre-captured at engine warmup so the first serving step pays no Triton JIT / capture latency (eager Triton fallback if capture fails or `suffix_gpu_use_cuda_graph=0`). - **Runner** (`v1/worker/gpu_model_runner.py`): the `ngram_gpu`-only gates (resident token buffers, optimistic-accept bookkeeping + rejection correction, valid-count D2H, scheduler_output shallow copy) become `use_gpu_state_drafter()` gates; suffix adds the two ingest hooks and graph pre-capture. No new scheduler interplay. Packaging follows the `suffix` / `arctic-inference` precedent: external optional dependency, lazy import, config validation fails with a clear install hint. (Open question below.) ## Correctness All numbers below: vLLM branch commit `f646af65c`, 1x NVIDIA L20 48GB, torch 2.13.0+cu130, drafter package suffix-gpu 0.1.1 (PyPI, tag `v0.1.1`; runs used the equivalent package code at repo commit `1ac9238`, installed editable — https://github.com/zip95297/SuffixGPU). - **Unit tests** (`tests/v1/spec_decode/test_suffix_gpu.py`): draft correctness on repetitive history, CUDA-graph vs eager draft agreement, graph pre-capture at warmup, JIT warmup with graphs disabled, global index cross-request drafting. Skips cleanly without CUDA or the package. Result: 6/6 passed (L20). - **E2E GSM8K** (in-tree pattern, same as ngram_gpu's async test): `test_suffix_gpu_with_async_scheduling` — Llama-3.1-8B-Instruct, suffix_gpu k=16 under async scheduling, 1319 questions 5-shot: accuracy 0.762 (threshold 0.70), passed. - **Acceptance e2e** (`tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance`): `test_suffix_gpu_acceptance` mirrors the CPU suffix acceptance test — 10 warm-up rounds over the same 100 prompts, suffix_gpu k=16 + async, pinned `suffix_gpu_ingest_chunk=1`, `num_backoff=8`, `max_occurrences=128`, prefix caching off, V1 runner. Warm-up series: round-0 rate 0.428 / AL 2.03 -> round-9 rate 0.857 / AL 5.70; the end rate clears the CPU test's 0.80 floor (0.79-0.86 across reruns — the final-round delta is ~1.4k drafted tokens, greedy near-ties move it a few points). Both tests pass together (`2 passed`); both — including the pre-existing CPU one — pin `enable_prefix_caching=False`, since warm-round prefix-cache hits change decode batch composition and depress the measured rate below the floor for the CPU method too. - **Greedy consistency**: spec on/off must produce token-identical greedy outputs. Run under `VLLM_BATCH_INVARIANT=1` + enforce_eager (a bitwise gate is meaningless otherwise: spec decode changes batch composition, and batch-dependent FP reduction order flips near-tie argmax even between two no-spec runs — measured on this box before enabling the invariant mode). Legs: async no-spec vs async suffix_gpu, plus suffix_gpu-eager / ngram_gpu / sync suffix CPU discriminators, 16 prompts x 256 tokens. Result: all legs token-identical (0/16 mismatches each; graph replay output == eager output). - **Drafter-level equivalence and latency vs the CPU suffix tree** (no engine, deterministic Spec-Bench replay): 221 tests pass including fuzz-equivalence against arctic on unambiguous corpora; replay tokens/step within -7%..0% of the CPU tree cold and parity to +3% warm; <= 532 MB reserved VRAM in the largest tested configuration. On drafting cost, the CPU tree wins small batches but its sequential per-request walk grows linearly with batch size, while graph-mode drafting is one flat batched launch: crossover at B64-128, and beyond it CUDA-graph mode is 2.1-2.7x faster per step even in the CPU's best case (B=256: 1.85 vs 4.98 ms; B=512: 5.15 vs 10.68 ms); under replay conditions that include per-step tree updates the GPU drafter is already faster from B32 (CPU 1.0-3.4 ms at B=58-80 vs a flat ~0.5 ms). This is what makes the drafter viable exactly in the high-concurrency regime async scheduling targets. Full tables and repro commands: https://github.com/zip95297/SuffixGPU/blob/main/RESULTS.md ## Performance Setup mirrors #25784 (Spec-Bench, k x concurrency grid, TPOT + drafted / accepted tokens) and #29184 (async-vs-sync same-method comparison): meta-llama/Llama-3.1-8B-Instruct, bf16, TP=1, 1x NVIDIA L20 48GB, `vllm bench serve --dataset-name spec_bench --spec-bench-output-len 256 --no-oversample`, concurrency 1/4/16/64/128/192/256, k in {5,16}, prefix caching off, all variants on model runner v1. Every variant gets `--max-num-batched-tokens 8192 --max-num-seqs 320` (spec decode schedules (k+1) padded token slots per running request before invalid ones are trimmed, so the serve-context defaults silently cap effective concurrency at the top grid points) and an identical 64-prompt warmup pass before measurement (cold CUDA graphs / Triton JIT / suffix index otherwise land in the first grid point; drafted/accepted numbers are the per-run deltas `vllm bench serve` records in each result json, so warmup traffic is excluded by construction). Reproduction script: `benchmarks/bench_pr_matrix.sh` (attached to the PR). Variants: async no-spec baseline; `suffix` + sync scheduling (what you must run today); `ngram_gpu` + async (in-tree GPU drafter baseline); `suffix_gpu` + async (this RFC). **k=5 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 58 (17.2) | 245 (15.4) | 754 (19.8) | 1297 (45.5) | 1343 (86.0) | 1375 (127.2) | 1373 (169.5) | | ngram_gpu_async | 46 (21.6) | 168 (23.3) | 515 (30.3) | 1126 (52.7) | 1341 (89.2) | 1389 (128.5) | 1396 (172.3) | | suffix_gpu_async | 57 (17.6) | 239 (15.6) | 744 (19.3) | 1316 (43.2) | 1410 (80.5) | 1461 (117.6) | 1467 (154.5) | **k=16 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 55 (18.2) | 239 (15.9) | 755 (19.5) | 1273 (46.3) | 1322 (86.0) | 1357 (125.8) | 1363 (167.2) | | ngram_gpu_async | 43 (23.0) | 161 (24.3) | 519 (29.9) | 1156 (52.2) | 1339 (89.6) | 1387 (131.0) | 1405 (170.6) | | suffix_gpu_async* | 52 (19.2) | 230 (16.2) | 736 (19.5) | 1276 (44.8) | 1349 (84.3) | 1402 (121.5) | 1417 (154.6) | \\* suffix_gpu k=16 ran at `gpu_memory_utilization=0.85` (other rows 0.9): at 0.9 the device-resident drafter state plus the (k+1)-slot verification batch OOMed the EngineCore at c256 on the 46GB L20 (see Known limits). **Drafted / accepted tokens per point (AL, acceptance rate)** | variant | k | c1 | c16 | c64 | c256 | |---|---|---|---|---|---| | suffix_sync | 5 | 79832/22918 (1.54, 28.7%) | 86405/33544 (1.94, 38.8%) | 86516/32697 (1.89, 37.8%) | 86240/31708 (1.85, 36.8%) | | suffix_gpu_async | 5 | 79742/22332 (1.52, 28.0%) | 87458/33453 (1.93, 38.3%) | 88350/35551 (2.04, 40.2%) | 88444/38485 (2.23, 43.5%) | | ngram_gpu_async | 5 | 33913/9467 (2.40, 27.9%) | 34863/9738 (2.40, 27.9%) | 34527/9332 (2.35, 27.0%) | 34358/9667 (2.41, 28.1%) | | suffix_sync | 16 | 83232/22867 (1.54, 27.5%) | 95070/34453 (1.98, 36.2%) | 93340/33505 (1.94, 35.9%) | 92211/32536 (1.89, 35.3%) | | suffix_gpu_async | 16 | 82552/22582 (1.53, 27.4%) | 95486/34166 (2.00, 35.8%) | 98323/36571 (2.13, 37.2%) | 105420/40513 (2.33, 38.4%) | | ngram_gpu_async | 16 | 35468/9978 (2.41, 28.1%) | 34844/9586 (2.38, 27.5%) | 33685/9643 (2.43, 28.6%) | 34316/9778 (2.42, 28.5%) | **Per-position acceptance rate — suffix CPU vs suffix_gpu, c=1 only** Measured at concurrency 1 (the c1 grid point, after the identical warmup pass), where scheduling noise is minimal, so this is the cleanest drafter-quality comparison. Each row also lists that run's drafts / draft tokens / accepted tokens / AL / overall acceptance rate. k=5, c=1: | variant | drafts | draft toks | accepted toks | AL | rate | p0 | p1 | p2 | p3 | p4 | |---|---|---|---|---|---|---|---|---|---|---| | suffix_sync | 42793 | 79832 | 22918 | 1.54 | 28.7% | 0.336 | 0.120 | 0.049 | 0.019 | 0.012 | | suffix_gpu_async | 42945 | 79742 | 22332 | 1.52 | 28.0% | 0.330 | 0.114 | 0.046 | 0.019 | 0.012 | k=16, c=1: | variant | drafts | draft toks | accepted toks | AL | rate | p0 | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | p12 | p13 | p14 | p15 | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | suffix_sync | 42677 | 83232 | 22867 | 1.54 | 27.5% | 0.329 | 0.115 | 0.046 | 0.018 | 0.010 | 0.005 | 0.004 | 0.002 | 0.002 | 0.002 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | | suffix_gpu_async | 42293 | 82552 | 22582 | 1.53 | 27.4% | 0.328 | 0.113 | 0.045 | 0.018 | 0.011 | 0.005 | 0.004 | 0.002 | 0.002 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.000 | The two per-position curves are essentially identical at both k values: the GPU drafter reproduces the CPU suffix tree's draft quality request-locally. Its aggregate advantage appears only as concurrency grows (AL 1.53 -> 2.33 from c1 to c256 at k=16, vs 1.98 -> 1.89 for the CPU tree), i.e. it comes from the cross-request global index compounding under concurrent traffic, not from a different per-draft profile. What the numbers show: 1. `suffix_gpu` + async beats `suffix` + sync from c64 up (+7% at k=5 saturation, 1467 vs 1373 tok/s; +4% at k=16, 1417 vs 1363; lower TPOT), and matches it at c1-c16 — resolving the suffix x async conflict is a real win once batches grow, exactly where async overlap matters. 2. `suffix_gpu` + async beats `ngram_gpu` + async at every grid point (up to +44%, k=5 c16: 744 vs 515 tok/s) — it is the stronger GPU drafter on this workload. 3. Against async no-spec, spec decoding wins through c64 (TPOT -31% at c4, -27% at c16, -8% at c64) but loses beyond c128 at saturation on Spec-Bench's mixed categories (1467 vs 1654 tok/s at k=5 c256) — true for all three drafters here (suffix_sync 1373, ngram_gpu 1396), see Known limits. ## Known limits - Saturation: beyond ~c128 on Spec-Bench's mixed categories, async no-spec out-throughputs every drafter here (k=5 c256: no-spec 1654 vs suffix_gpu 1467, suffix_sync 1373, ngram_gpu 1396 tok/s) — once the GPU is compute-bound, verification FLOPs for rejected drafts cost more than acceptance saves. On repetition-heavy traffic the crossover moves right; measured, not hidden. (Contrary to the usual GPU-drafter expectation, c=1 does not regress on this workload: 57-58 tok/s vs 47 no-spec, the warm suffix index pays for the drafter cost.) - Memory headroom at k=16: the device-resident drafter state plus the (k+1)-slot verification batch OOMed the EngineCore at c256 with the default `gpu_memory_utilization=0.9` on a 46GB L20; the k=16 row was measured at 0.85. Large-k + high-concurrency deployments need to budget for the drafter's device state. - Spec-decode acceptance metrics for GPU-state drafters counted scheduler-padded slots in the denominator (affects `ngram_gpu` equally). This PR includes the worker-side fix — per-request trim counts ride on `ModelRunnerOutput` and are subtracted in `make_spec_decoding_stats` — with method-neutral naming; mechanism-wise it matches the stale open ngram-only PR #44056 (happy to rebase or split if maintainers prefer; #50518 covers the scheduler-side view for uniform batches). ## Proposed change Land `suffix_gpu` as a speculative decoding method: config surface, `SuffixProposerGPU`, runner wiring behind `use_gpu_state_drafter()`, unit tests. Branch: `zip95297/suffix_gpu` on https://github.com/zip95297/vllm-dev. Docs plan (same shape as #25784's docs addition): a `suffix_gpu` subsection in `docs/features/spec_decode.md` — when to pick it over `suffix`/`ngram_gpu`, the `suffix_gpu_*` knobs and their defaults, the async-scheduling compatibility note, the memory-headroom guidance for large k, and the package install hint. Lands with this PR or as an immediate follow-up once the packaging question below is settled (in-tree vs external changes the install section). ## Open question Keep the drafter core as an external optional dependency (arctic-inference precedent; small in-tree diff, kernels iterate on their own cadence) or port it in-tree under `v1/spec_decode/` (~1.5 kLOC pure Python + Triton, torch-only deps; one source of truth, CI coverage)? I lean in-tree for a decode-hot-path feature but either works; the wiring in this PR is the same either way. ## Future work Variable draft-length scheduling for GPU-state drafters (Dynamic SD #32374 integration and per-request draft lengths) is deliberately out of scope; it applies to `ngram_gpu` and `suffix_gpu` equally and deserves its own design discussion.", "url": "https://github.com/vllm-project/vllm/issues/51788", "createdAt": "2026-08-11T09:12:06Z", "updatedAt": "2026-08-13T04:53:39Z", "timestamp": "2026-08-13T04:53:39Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "RFC" ], "author": "zip95297", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51798", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Kimi-K3-NVFP4 on 8xB300 produces degenerate, incoherent output in the reasoning channel on v0.27.0", "text": "### Your current environment We are not able to run `collect_env.py` against the affected build. It was a production deployment that has since been rolled back, so that container no longer exists. The information below is therefore split into the node-level facts (identical across both builds, since it is the same physical machine) and the identity of the affected build. **Node — unchanged between the working and the failing build** | | | |---|---| | GPUs | 8x NVIDIA B300 SXM6 AC (288 GB each), NV18 full mesh | | NVIDIA driver | 595.71.05 | | CUDA runtime | 13.0.88 | | Host OS | Talos Linux (immutable, no host userland), kernel `6.18.34-talos` | | Container userland | Ubuntu 24.04.3 LTS (x86_64), glibc 2.39, Python 3.12.3 — from the vLLM image | | CPU | 224 vCPU, 2 NUMA nodes, KVM guest | **Affected build:** the published `v0.27.0` release image (`vllm/vllm-openai:v0.27.0`) — vLLM 0.27.0, torch 2.13.0+cu130, FlashInfer 0.6.16.post3, Triton 3.7.1, transformers 5.15.0, as pinned for that tag. ### 🐛 Describe the bug ## What we run `RedHatAI/Kimi-K3-NVFP4` served on a single node of 8x B300, tensor parallel 8, weights loaded. Serving flags: ``` --tensor-parallel-size=8 --gpu-memory-utilization=0.95 --max-model-len=262144 --max-num-batched-tokens=8192 --max-num-seqs=64 --kv-cache-dtype=fp8 --attention-config={\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"} --enable-prefix-caching --load-format=fastsafetensors --no-enable-flashinfer-autotune --trust-remote-code --enable-auto-tool-choice --tool-call-parser=kimi_k3 --reasoning-parser=kimi_k3 --enable-prompt-tokens-details ``` ## What we observed On `v0.27.0` the model produces degenerate, incoherent text in the reasoning channel. The prompt was as simple as `hello how are you`. The reasoning content came back as: ``` G now now now now now now now now now now now now now now now now now now now now now now { how are now now now now now now ) [ how are now } [ are now now now now now now now now are now [ is now now is now [ G is now now is now now ) G is now [ now now is now now now now now now now now now now now = is now now = now now is now - is now -G is now - now now - G now -G is this - this - this ``` - **It appeared immediately, on a trivially short prompt.** This did not develop after hours of serving, under load, or after a long-context request. It was present essentially from the first request after the version change, with a prompt of five words. - **Nothing crashed or reported an error.** No CUDA errors, no illegal memory access, no assertion failures, no NaN-related aborts, no engine restarts. The pod stayed `Running` with 0 restarts and kept answering requests at normal latency. The failure was completely silent from an operational standpoint: health checks and metrics looked entirely normal while the output was garbage. That is the part we found most concerning — nothing in our monitoring would have caught this. ## What changed to trigger it We had been serving this model on a **nightly build from 2026-08-09** without this symptom. We then moved the deployment to the **`v0.27.0` release tag** and changed nothing else — same physical node, same GPUs, same driver, same weights, same flags. The degenerate output appeared with that change. As mitigation we rolled the deployment back to that same nightly. We want to be precise about the limits of this evidence: we changed builds, we did not bisect individual commits, so we cannot point at a specific change as the cause. What we can say is that the `v0.27.0` release tag shows it on this configuration and a nightly from a few days before the release did not. Kimi-K3 support is very new and appears to be changing quickly, so we wondered whether the release branch is simply missing model fixes that landed on `main` shortly before the tag was cut — but that is speculation on our part, and someone with a view of what went into the branch is far better placed to judge it than we are. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51798", "createdAt": "2026-08-11T10:01:53Z", "updatedAt": "2026-08-13T08:59:21Z", "timestamp": "2026-08-13T08:59:21Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "kimi", "k3" ], "author": "pyhp2017", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51842", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails (because vLLM expects k_proj in a fused-qkv ViT attention layer)", "text": "### Your current environment vLLM: 0.27.0 Python: 3.10.19 PyTorch: 2.13.0+cu130 GPU: NVIDIA GeForce RTX 4070 Ti SUPER (16 GB) NVIDIA driver: 610.62 OS: Ubuntu 22.04.5 LTS under WSL2 Model: MiniCPM-V-4.6 Model path: /mnt/e/BaiduNetdiskDownload/MiniCPM-V-4.6 --trust-remote-code enabled ### 🐛 Describe the bug ### Description Starting `OpenBMB/MiniCPM-V-4.6` with vLLM `0.27.0` fails while loading model weights. The failure occurs in vLLM's built-in `minicpmv4_6.py` implementation, before the server starts. The model's vision attention layer exposes fused `qkv_proj` parameters, but the vLLM weight-loading path attempts to load a non-existent `k_proj` parameter. ### Reproduction ```bash VLLM_WSL2_ENABLE_PIN_MEMORY=1 vllm serve \\ --model /mnt/e/BaiduNetdiskDownload/MiniCPM-V-4.6 \\ --trust-remote-code \\ --gpu-memory-utilization 0.9 \\ --port 8000 \\ --served-model-name OpenBMB/MiniCPM-V-4.6 \\ --max-model-len 10240 \\ --max-num-seqs 4 ``` Startup fails while loading weights: ```bash ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters belonging to (MiniCPMV4_6ViTWindowAttentionSelfAttn) are: {'qkv_proj.bias', 'qkv_proj.weight', 'out_proj.bias', 'out_proj.weight'} Relevant traceback: File \".../vllm/model_executor/models/minicpmv4_6.py\", line 1283, in load_weights return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) File \".../vllm/model_executor/models/minicpmv4_6.py\", line 709, in load_weights return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) File \".../vllm/model_executor/models/utils.py\", line 395, in _load_module raise ValueError(msg) ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters belonging to (MiniCPMV4_6ViTWindowAttentionSelfAttn) are: {'qkv_proj.bias', 'qkv_proj.weight', 'out_proj.bias', 'out_proj.weight'} ``` ### Suspected cause MiniCPMV4_6ViTWindowAttentionSelfAttn uses fused qkv_proj.{weight,bias} parameters, while the current vLLM checkpoint mapping or loader appears to target a separate k_proj parameter. Could the MiniCPM-V-4.6 vision-attention weight mapping be updated to handle the fused QKV projection layout? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51842", "createdAt": "2026-08-11T16:08:19Z", "updatedAt": "2026-08-13T03:50:58Z", "timestamp": "2026-08-13T03:50:58Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "llery2021", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51868", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable varlen for Nemotron-H (Mamba)", "text": "### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, it is not clear if our various Mamba decode kernels are compatible with varlen execution. We should audit the flashinfer and/or triton mamba decode kernels and check if we can run decode with a ragged batch. If it's feasible, we should wire it up, validate correctness, evaluate performance, and merge support. DEPENDS ON https://github.com/vllm-project/vllm/pull/47808 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51868", "createdAt": "2026-08-11T20:06:53Z", "updatedAt": "2026-08-12T21:18:52Z", "timestamp": "2026-08-12T21:18:52Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request", "nvidia" ], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51869", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable varlen for Qwen 3.x (GDN)", "text": "### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, it is not clear if our various GDN decode kernels are compatible with varlen execution. We should audit the flashinfer and/or triton GDN decode kernels and check if we can run decode with a ragged batch. If it's feasible, we should wire it up, validate correctness, evaluate performance, and merge support. DEPENDS ON https://github.com/vllm-project/vllm/pull/47808 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51869", "createdAt": "2026-08-11T20:07:42Z", "updatedAt": "2026-08-12T21:18:44Z", "timestamp": "2026-08-12T21:18:44Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request", "qwen" ], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51873", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable logprobs with adaptive verification", "text": "### 🚀 The feature, motivation and pitch Follow-up leftover from #47808. See: https://github.com/vllm-project/vllm/pull/47808#discussion_r3723768178 ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51873", "createdAt": "2026-08-11T20:26:26Z", "updatedAt": "2026-08-13T12:19:43Z", "timestamp": "2026-08-13T12:19:43Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51874", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Polish DSpark adaptive timings", "text": "### 🚀 The feature, motivation and pitch Following #47808, the engine profiles the forward pass and proposer timings for use in estimating the batch runtime. The measurement facility could use a refactor. It is valuable to expose these metrics to the frontend, via /metrics endpoint and also compatibility with iteration-level logging so we can easily observe the fine-grained engine GPU latency for different phases (drafting, verification, sampling, other). See https://github.com/vllm-project/vllm/pull/50919 for a starting point. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51874", "createdAt": "2026-08-11T20:39:19Z", "updatedAt": "2026-08-12T21:54:14Z", "timestamp": "2026-08-12T21:54:14Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request" ], "author": "benchislett", "state": "open", "assignees": [ "benchislett" ] }, { "id": "github:vllm-project/vllm:issue:51884", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: FP8 block-scaled weights fail on sm120 (RTX 5090) — DeepGEMM \"Unknown SF transformation\" during process_weights_after_loading", "text": "### Your current environment
Environment ``` vLLM: 0.26.1rc1.dev608+g99a10304d (docker.io/vllm/vllm-openai:muse-glimmer) GPU: 2x NVIDIA RTX 5090 (sm120 / device capability 12.0), PCIe Platform: Linux x86_64, CUDA 12.x Model: RedHatAI/Muse-Glimmer-30B-FP8-block (compressed-tensors, block-scaled FP8 weights) ```
### 🐛 Describe the bug FP8 block-scaled weights (`compressed-tensors` w8a8) fail to load on sm120 with a DeepGEMM assertion. It happens in `process_weights_after_loading` — during weight loading, before any inference — so the engine never starts: ``` RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation ``` Full call chain (both TP workers fail identically): ``` compressed_tensors_w8a8_fp8.py:169 process_weights_after_loading -> kernels/linear/scaled_mm/deep_gemm.py:96 process_weights_after_loading -> quantization/utils/fp8_utils.py:1138 deepgemm_post_process_fp8_weight_block -> quantization/utils/fp8_utils.py:1077 deepgemm_post_process_weight_scale_block -> utils/deep_gemm.py:494 transform_sf_into_required_layout -> _transform_sf_into_required_layout_impl -> deepgemm csrc/apis/layout.hpp:60 \"Unknown SF transformation\" ``` The reason this is reached rather than skipped is that vLLM treats sm120 as a supported DeepGEMM target. `vllm/utils/deep_gemm.py` gates on capability family 100 **or** 120 in two places (lines 42-43 and 79-80), and `is_deep_gemm_supported()` documents \"only Hopper and Blackwell GPUs are supported\" — so consumer Blackwell selects the DeepGEMM path, and then DeepGEMM's own layout transform rejects the scale-factor layout it is handed. Either the SF transform needs an sm120 case, or the capability-family-120 gate should exclude this path for block-scaled FP8 weights. ### Reproduction ```bash docker run --gpus all --ipc=host -e CUDA_VISIBLE_DEVICES=0,1 \\ -v ~/.cache/huggingface:/root/.cache/huggingface \\ vllm/vllm-openai:muse-glimmer \\ --model RedHatAI/Muse-Glimmer-30B-FP8-block \\ --tensor-parallel-size 2 --gpu-memory-utilization 0.88 --max-model-len 131072 ``` Fails ~75 s in, at weight loading, reproducibly. ### Workaround `VLLM_USE_DEEP_GEMM=0` (plus `VLLM_MOE_USE_DEEP_GEMM=0`) makes the same command work. With that set the model loads and serves normally on 2x RTX 5090 — 177.6 tok/s decode with DFlash speculative decoding, 663,458-token KV cache — so only the DeepGEMM SF-layout path appears to be affected, not FP8 on sm120 in general. I have not tested whether NVFP4 or non-block-scaled FP8 checkpoints hit the same transform on this hardware; the NVFP4 variant of the same model loads fine with DeepGEMM left at its default.", "url": "https://github.com/vllm-project/vllm/issues/51884", "createdAt": "2026-08-11T22:34:07Z", "updatedAt": "2026-08-13T06:35:17Z", "timestamp": "2026-08-13T06:35:17Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "quantization" ], "author": "abtraore", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51912", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Allow logging model output text without output token IDs", "text": "### 🚀 The feature, motivation and pitch In vLLM v0.27.1, `--enable-log-outputs` logs the generated text, `output_token_ids`, and finish reason together in the same INFO record: ```text Generated response : output: '...', output_token_ids: [...], finish_reason: stop ``` For production troubleshooting, I want to keep the human-readable generated output and finish reason while omitting output token IDs. Long token ID lists add substantial log volume, and token IDs are not anonymized data because they can be decoded back into text. I propose adding a BooleanOptionalAction-style frontend option: ```text --enable-log-output-token-ids --no-enable-log-output-token-ids ``` Suggested behavior: - Preserve current behavior by default (`enable_log_output_token_ids=True`). - The option is relevant only when `--enable-log-outputs` is enabled. - With `--no-enable-log-output-token-ids`, omit the entire `output_token_ids` field from output log records rather than logging `output_token_ids: None`. - Apply the behavior consistently to streaming and non-streaming output logging across supported serving endpoints. - When token ID logging remains enabled, continue applying `--max-log-len` as today. Prompt logging at INFO is intentionally outside this issue's scope because it is already covered by #42051 and #38583. The broader request/output logging use case is also discussed in #40155. ### Alternatives - Removing `--enable-log-outputs` also removes the generated text needed for troubleshooting. - `--max-log-len` truncates both the desired output text and token IDs, and it does not eliminate per-delta streaming log records. - A custom `--log-config-file` can change logger levels, but standard Python `dictConfig` cannot remove only `output_token_ids` because it is embedded in the same log message and argument tuple as the output and finish reason. Doing so requires a custom Python filter/formatter. - Filtering the field in an external logging pipeline is possible but brittle and makes the textual log format an implicit parsing contract. ### Additional context Relevant v0.27.1 implementation: - `RequestLogger.log_outputs`: https://github.com/vllm-project/vllm/blob/v0.27.1/vllm/entrypoints/serve/utils/request_logger.py#L70-L98 - Chat completion output logging: https://github.com/vllm-project/vllm/blob/v0.27.1/vllm/entrypoints/openai/chat_completion/serving.py#L1106-L1136 This request is based on an observed v0.27.1 deployment using `--enable-log-requests --enable-log-outputs`. ### Before submitting a new issue... - [x] I searched existing and past issues, open pull requests, and the official documentation for an existing output-token-ID logging control.", "url": "https://github.com/vllm-project/vllm/issues/51912", "createdAt": "2026-08-12T02:28:33Z", "updatedAt": "2026-08-13T13:11:53Z", "timestamp": "2026-08-13T13:11:53Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "ruanwenjun", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51914", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark", "text": "## Summary With **DeepSeek-V4-Flash-0731** on **vLLM v0.27.1** with **DSpark enabled**, we have intermittently observed malformed DSML tool-call output where the opening wrapper is corrupted from: ```text <｜DSML｜tool_calls> ``` to something like: ```text <｜DSML｜toolcalls> ``` while the rest of the DSML block remains structurally close to valid and the closing wrapper is still emitted as: ```text ``` This causes the tool-call parser to miss the call and surface the DSML as assistant content instead of producing `tool_calls`. This report intentionally uses a **synthetic example** and does not include the original production prompt, tool names, arguments, paths, or other private information. ## Environment - vLLM: **v0.27.1** - Model: **deepseek-ai/DeepSeek-V4-Flash-0731** - Speculative decoding: **DSpark enabled** - Tool-call parser: DeepSeek V4 / DSML path - Observed intermittently rather than deterministically ## Synthetic example Expected model output: ```text <｜DSML｜tool_calls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` Malformed output observed in the same class of failure: ```text <｜DSML｜toolcalls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` The difference is only the opening wrapper (`tool_calls` -> `toolcalls`), but that is enough for strict DSML parsing to fail. ## Expected behavior If the model emits an otherwise complete DSML invoke block for a tool that is actually declared in the request, vLLM should ideally either: 1. recover the tool call safely, or 2. at minimum detect the malformed DSML as a protocol error instead of silently returning the whole block as ordinary assistant content. Any recovery should remain conservative and validate the recovered tool name against the tools declared in the request. ## Relation to existing issues This appears related to #48931 and #49117, but it is a **different malformed-output variant**. - #48931 covers the case where the `<｜DSML｜tool_calls>` opening wrapper is **completely omitted**. - This case has an opening wrapper, but it is **corrupted/misspelled** (`tool_calls` -> `toolcalls`). - Therefore, recovery logic that only handles a fully missing wrapper may not cover this case as-is. It would be useful to add parser regression coverage for malformed opening wrappers in both streaming and non-streaming paths, especially with DeepSeek-V4-Flash-0731. ## DSpark note This was observed with DSpark enabled. I do **not** have enough evidence yet to claim that DSpark is the cause. A DSpark ON/OFF comparison would be useful because speculative decoding should not materially change protocol correctness if rejected draft tokens are handled correctly. The main issue here is that a small model-side DSML formatting error currently turns into a complete tool-call failure at the serving layer.", "url": "https://github.com/vllm-project/vllm/issues/51914", "createdAt": "2026-08-12T02:54:35Z", "updatedAt": "2026-08-13T15:03:07Z", "timestamp": "2026-08-13T15:03:07Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [], "author": "jinbagi", "state": "open", "assignees": [ "chaunceyjiang" ] }, { "id": "github:vllm-project/vllm:issue:51916", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][DSV4-Flash][DSpark] v0.27 weight loader regressed — KeyError routed_experts.w13_weight_scale (v0.26 gracefully skipped)", "text": "### Your current environment - vLLM: **0.27.1** (pip, `/data1/vllm027` venv) - Python 3.11 - torch 2.13.0+cu130, triton 3.7.1, flashinfer-python 0.6.16.post3, transformers 5.15.0, CUDA 13.0 - Hardware: 8× NVIDIA H20-3e (141 GB), TP=8 + expert parallel - (Full `collect_env.py` available on request.) ### How to reproduce Serve DeepSeek-V4-Flash-0731 with DSpark speculative decoding on v0.27.1 (the actual flags used): ``` vllm serve \\ --trust-remote-code \\ --tensor-parallel-size 8 --enable-expert-parallel \\ --gpu-memory-utilization 0.85 --max-model-len 524288 \\ --max-num-batched-tokens 16384 --max-num-seqs 64 \\ --kv-cache-dtype fp8 --block-size 256 --dtype auto \\ --enable-prefix-caching --enable-chunked-prefill \\ --kv-offloading-size 512 --kv-offloading-backend native \\ --speculative-config.method dspark \\ --speculative-config.model \\ --speculative-config.num_speculative_tokens 5 \\ --enable-auto-tool-choice --tool-call-parser deepseek_v4 \\ --reasoning-parser deepseek_v4 --disable-custom-all-reduce ``` **Pre-req to reach this code path on Python ≤ 3.11**: flashinfer 0.6.16's `flashinfer/comm/fd_exchange.py:55` has a return annotation `tuple[..., array.array[int]]` that raises `TypeError: type 'array.array' is not subscriptable` at import time (`array.array` gains `__class_getitem__` only in 3.12). That crash happens *earlier* and masks this bug. To reproduce **this** bug, either run on Python 3.12, or add `from __future__ import annotations` to flashinfer's `fd_exchange.py`. ### Error During draft-model weight load at startup (all TP/EP ranks fail identically): ``` File \".../vllm/v1/worker/gpu/spec_decode/speculator.py\", line 156, in load_model self.model = self.load_draft_model(target_model, target_attn_layer_names) File \".../vllm/v1/worker/gpu/spec_decode/dspark/speculator.py\", line 81, in load_draft_model model = load_dspark_model(target_model, self.vllm_config) File \".../vllm/v1/worker/gpu/spec_decode/dspark/utils.py\", line 45, in load_dspark_model draft_model = get_model(...) ... File \".../vllm/models/deepseek_v4/nvidia/dspark.py\", line 449, in load_weights param = params_dict[name_mapped] ~~~~~~~~~~~^^^^^^^^^^^^^ KeyError: 'model.layers.0.ffn.experts.routed_experts.w13_weight_scale' ``` → `WorkerProc failed to start` → engine init failed. ### Root cause v0.27's DSpark draft weight loader (`dspark.py:449` `load_weights`) looks up the mapped name with a direct subscript `params_dict[name_mapped]`. The DSpark draft module does not register `model.layers.0.ffn.experts.routed_experts.w13_weight_scale` (a target-model routed-experts weight the draft does not carry), so the lookup raises. **This is a regression from v0.26**: v0.26's loader skipped unknown draft weights with a `\"Skipping unknown DStorm weight ...\"` log line (graceful). v0.27 dropped that guard → hard `KeyError`. ### Suggested fix Restore the skip guard at `dspark.py:449`: ```python if name_mapped not in params_dict: continue param = params_dict[name_mapped] ``` Safe because spec-decoding draft weights are validated by the target model; an unmapped draft weight should be skipped, not fatal. Restores v0.26 behavior. ### Additional context Two further issues block v0.27.1 on this exact stack (DSV4-Flash + DSpark + Hopper + Py3.11), tracked separately: 1. **flashinfer 0.6.16 `fd_exchange.py` PEP-585 annotation crash on Py ≤ 3.11** (the pre-req above) — flashinfer side; one-line `from __future__ import annotations` fix. 2. After both fixes, the **V1 engine hangs in warmup** (`shm_broadcast: No available shared memory broadcast block found in 60s` repeating 7+ min, all 8 GPUs at 0% util). Root cause not yet isolated (may be a knock-on of the patched loader); not reporting yet pending a clean repro. Happy to provide the full raw traceback / `collect_env.py` / test a fix on this 8×H20-3e stack.", "url": "https://github.com/vllm-project/vllm/issues/51916", "createdAt": "2026-08-12T03:07:04Z", "updatedAt": "2026-08-13T05:07:44Z", "timestamp": "2026-08-13T05:07:44Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "xiongyidxx", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51921", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] v0.27.0 engine permanently stalls after ~1 min idle on 4-node TP=4 (GB10/sm_121, aarch64): shm_broadcast writer starves, requests never reach scheduler", "text": "## Bug On 4-node TP=4 (GB10/sm_121a, aarch64), a v0.27.0 engine serves its first requests normally, then **permanently stalls after roughly one minute of engine idleness**. Requests after the gap never enter the scheduler (`num_requests_running` stays 0), the API keeps answering `/v1/models` and `/tokenize`, and the head's EngineCore logs repeat: ``` [shm_broadcast.py:802] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work ``` every 60s indefinitely. All four rank containers stay alive; all TP workers' last log lines are healthy post-warmup (graph capture complete). Only a full fleet restart recovers. Reproduced 3/3 boots. Keeping the API warm (`/v1/models` polling every 45s) does NOT prevent it; the stall follows engine-level idleness, not connection idleness. Sequence, timestamped from one boot: warmup collectives complete across all ranks (so the broadcast queue works) → first 1-2 completions succeed (~1.3s each) → ~1-2 min idle → all subsequent requests hang forever. ## Environment Official `vllm/vllm-openai:v0.27.0-aarch64` + two patches required to get this far on sm_121: DeepGEMM repinned to `2fd67329` (#51758) and the `masked_mha_available` fix (#51920). GLM-5.2 (GlmMoeDsaForCausalLM), TP=4 via `--nnodes 4 --node-rank N`, mp executor, `FLASHINFER_MLA_SPARSE_SM120` + FLASH_ATTN MLA prefill, `fp8_ds_mla` KV, MTP k=2 spec decode, dual-rail RoCEv2, `VLLM_USE_BREAKABLE_CUDAGRAPH=0` set. torch 2.13.0+cu130 bundled NCCL. Related: #51758 validated v0.27.0 on 2x Spark (TP=2 and PP=2) with a 400-request soak and zero hangs — consistent with our result that the first requests DO work; the failure needs the multi-node idle gap, and possibly the 4-node width. This looks adjacent to the pre-0.27 TP-hang family that release fixed for 2-node. Full per-rank logs, exact launch argv, and timings available; the cluster reproduces this in ~20 minutes per attempt and I can run instrumented builds or targeted env matrices on request.", "url": "https://github.com/vllm-project/vllm/issues/51921", "createdAt": "2026-08-12T04:03:50Z", "updatedAt": "2026-08-13T16:51:51Z", "timestamp": "2026-08-13T16:51:51Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51964", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]:amd mi308x gpu, vllm 0.27.0~0.27.1, rocm 7.2.3, Kimi-K2.7-Coder start fails:AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942", "text": "### Your current environment
The output of python collect_env.py ```text (vllm) [fanlf@2025-h3c-83 tmp]$ python ./collect_env.py Collecting environment information... ============================== System Info ============================== OS : Rocky Linux 10.2 (Red Quartz) (x86_64) GCC version : (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4) Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+gitd0c8b1f Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Jul 10 2026, 00:00:00) [GCC 14.3.1 20251022 (Red Hat 14.3.1-4)] (64-bit runtime) Python platform : Linux-6.12.0-124.29.1.el10_1.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : AMD Instinct MI308X (gfx942:sramecc+:xnack-) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU(s) scaling MHz: 66% CPU max MHz: 3709.3569 CPU min MHz: 1500.0000 BogoMIPS: 4793.03 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap Virtualization: AMD-V L1d cache: 6 MiB (192 instances) L1i cache: 6 MiB (192 instances) L2 cache: 192 MiB (192 instances) L3 cache: 768 MiB (24 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-95,192-287 NUMA node1 CPU(s): 96-191,288-383 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.3.5 [pip3] onnx==1.22.0 [pip3] onnx-ir==0.2.1 [pip3] onnxscript==0.7.1 [pip3] onnxslim==0.1.95 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0+gitd0c8b1f [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.9.0+eaa9e4e [pip3] torchvision==0.24.1+d801a34 [pip3] transformers==5.15.0 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : 7.2.53211-c2d9476115 vLLM Version : 0.27.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 15 15 15 15 15 15 15 GPU1 15 0 15 15 15 15 15 15 GPU2 15 15 0 15 15 15 15 15 GPU3 15 15 15 0 15 15 15 15 GPU4 15 15 15 15 0 15 15 15 GPU5 15 15 15 15 15 0 15 15 GPU6 15 15 15 15 15 15 0 15 GPU7 15 15 15 15 15 15 15 0 ================================= Hops between two GPUs ================================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 1 1 1 1 1 1 1 GPU1 1 0 1 1 1 1 1 1 GPU2 1 1 0 1 1 1 1 1 GPU3 1 1 1 0 1 1 1 1 GPU4 1 1 1 1 0 1 1 1 GPU5 1 1 1 1 1 0 1 1 GPU6 1 1 1 1 1 1 0 1 GPU7 1 1 1 1 1 1 1 0 =============================== Link Type between two GPUs =============================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 XGMI XGMI XGMI XGMI XGMI XGMI XGMI GPU1 XGMI 0 XGMI XGMI XGMI XGMI XGMI XGMI GPU2 XGMI XGMI 0 XGMI XGMI XGMI XGMI XGMI GPU3 XGMI XGMI XGMI 0 XGMI XGMI XGMI XGMI GPU4 XGMI XGMI XGMI XGMI 0 XGMI XGMI XGMI GPU5 XGMI XGMI XGMI XGMI XGMI 0 XGMI XGMI GPU6 XGMI XGMI XGMI XGMI XGMI XGMI 0 XGMI GPU7 XGMI XGMI XGMI XGMI XGMI XGMI XGMI 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: 0 GPU[1] : (Topology) Numa Node: 0 GPU[1] : (Topology) Numa Affinity: 0 GPU[2] : (Topology) Numa Node: 0 GPU[2] : (Topology) Numa Affinity: 0 GPU[3] : (Topology) Numa Node: 0 GPU[3] : (Topology) Numa Affinity: 0 GPU[4] : (Topology) Numa Node: 1 GPU[4] : (Topology) Numa Affinity: 1 GPU[5] : (Topology) Numa Node: 1 GPU[5] : (Topology) Numa Affinity: 1 GPU[6] : (Topology) Numa Node: 1 GPU[6] : (Topology) Numa Affinity: 1 GPU[7] : (Topology) Numa Node: 1 GPU[7] : (Topology) Numa Affinity: 1 ================================== End of ROCm SMI Log =================================== ============================== Environment Variables ============================== VLLM_ROCM_USE_AITER=1 PYTORCH_ROCM_ARCH=gfx942 TORCH_BLAS_PREFER_HIPBLASLT=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_fanlf ```
### 🐛 Describe the bug 1. install： uv pip install vllm==\"0.27.0+rocm723\" --extra-index-url https://wheels.vllm.ai/rocm --no-cache 2. startup and error log： ``` log (vllm) [fanlf@2025-h3c-83 vllm]$ export AITER_ROCM_ARCH=\"gfx942\" export PYTORCH_ROCM_ARCH=\"gfx942\" export MORI_GPU_ARCHS=\"gfx942\" export VLLM_ROCM_USE_AITER=1 export TORCH_BLAS_PREFER_HIPBLASLT=1 export HIP_FORCE_DEV_KERNARG=1 (vllm) [fanlf@2025-h3c-83 vllm]$ vllm serve \\ /pool/models/moonshotai/Kimi-K2.7 \\ --served-model-name kimi \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --mm-encoder-tp-mode data \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.6 \\ --port 8003 \\ --host 0.0.0.0 \\ --tool-call-parser kimi_k2 \\ --enable-auto-tool-choice \\ --reasoning-parser kimi_k2 \\ --kv-cache-dtype fp8 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █ █ █▄ ▄█ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.0 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █▄█▀ █ █ █ █ model /pool/models/moonshotai/Kimi-K2.7 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:273] non-default args: {'model_tag': '/pool/models/moonshotai/Kimi-K2.7', 'enable_auto_tool_choice': True, 'tool_call_parser': 'kimi_k2', 'host': '0.0.0.0', 'port': 8003, 'model': '/pool/models/moonshotai/Kimi-K2.7', 'trust_remote_code': True, 'max_model_len': 262144, 'served_model_name': ['kimi'], 'reasoning_parser': 'kimi_k2', 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'mm_encoder_tp_mode': 'data'} ...... (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] WorkerProc hit an exception. (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] [vllm-rocm.md.txt](https://github.com/user-attachments/files/30973739/vllm-rocm.md.txt) ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51964", "createdAt": "2026-08-12T08:49:47Z", "updatedAt": "2026-08-13T13:57:20Z", "timestamp": "2026-08-13T13:57:20Z", "metrics": { "reactions": 0, "comments": 5 }, "labels": [ "bug", "rocm", "kimi" ], "author": "badrecover", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51971", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Qwen3 MoE GPTQ `qzeros` shape mismatch on ROCm gfx1201", "text": "### Your current environment
(Run from within remote container that I started via `docker --context` from host) ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.3 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) CMake version : version 3.31.10 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.12.0+git6bbd260 Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-7.0.0-28-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : (gfx1201) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: AuthenticAMD Model name: AMD Ryzen Threadripper PRO 9995WX 96-Cores CPU family: 26 Model: 8 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 1 Stepping: 1 Frequency boost: enabled CPU max MHz: 5460.5269 CPU min MHz: 1217.1060 BogoMIPS: 4992.72 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d debug_swap amd_lbr_pmc_freeze Virtualization: AMD-V L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 96 MiB (96 instances) L3 cache: 384 MiB (12 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-191 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Reduced Speculation Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.3.5 [pip3] onnx==1.22.0 [pip3] onnx-ir==0.2.1 [pip3] onnxscript==0.7.1 [pip3] onnxslim==0.1.95 [pip3] pyzmq==27.1.0 [pip3] torch==2.12.0+git6bbd260 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+34c52a6 [pip3] torchvision==0.27.1+df56172 [pip3] transformers==5.14.1 [pip3] triton==3.7.1+git0263a6a6 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : 7.2.53211-c2d9476115 vLLM Version : 0.26.1rc1.dev542+gb22afe45a (git sha: b22afe45a) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU0 0 ================================= Hops between two GPUs ================================== GPU0 GPU0 0 =============================== Link Type between two GPUs =============================== GPU0 GPU0 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: -1 ================================== End of ROCm SMI Log =================================== ============================== Environment Variables ============================== LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib: PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151 MAX_JOBS=16 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
### 🐛 Describe the bug `Qwen/Qwen3-30B-A3B-GPTQ-Int4` cannot start on ROCm. vLLM reaches model warmup, then the RDNA W4A16 kernel rejects the GPTQ zero-point layout: ```text AssertionError: zp shape mismatch: torch.Size([16, 640]) vs (5120, 16) ``` Is this checkpoint layout unsupported, or should the ROCm kernel transpose/normalize `qzeros` before the assertion? ## Environment - vLLM image: `vllm/vllm-openai-rocm@sha256:8f71438ee82c2022cdca92ffc8e7d93573e65f3bf59c2d5d915a5ed7d13db1e2` - vLLM: `0.26.1rc1.dev542+gb22afe45` - Model: `Qwen/Qwen3-30B-A3B-GPTQ-Int4` (`revision=main`) - Quantization detected: `auto_gptq` - GPU: AMD Radeon AI PRO R9700, `gfx1201` (reported by host) - Platform: ROCm Docker image, single GPU, tensor parallel size `1` - Context: `32768` - GPU memory utilization: `0.90` ## Reproduction ```bash docker --context compose \\ -f compose.yaml up vllm ``` compose.yaml defines the elevant server arguments: ```text Qwen/Qwen3-30B-A3B-GPTQ-Int4 --max-model-len 32768 --gpu-memory-utilization 0.90 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --default-chat-template-kwargs '{\"enable_thinking\":true}' ``` Relevant startup output: ```text Using RDNAHybridW4A16LinearKernel for AutoGPTQLinearMethod Layer 'model.layers.0.mlp.experts' is not supported by GPTQMoeMarlin. Falling back to Moe WNA16 kernels. Using 'TRITON' WNA16 MoE backend. Loading weights took 102.46 seconds Model loading took 15.65 GiB memory ``` Failure: ```text File .../vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py, line 228 assert zp.shape == (N, num_groups), ( AssertionError: zp shape mismatch: torch.Size([16, 640]) vs (5120, 16) ``` The failure occurs during dummy warmup/KV-cache profiling, before the API becomes healthy. ## Backend tests All tests used the same pinned nightly image, model, and runtime configuration. | `--moe-backend` | Result | |---|---| | omitted / `triton` | Reaches warmup, then fails with `zp shape mismatch: torch.Size([16, 640]) vs (5120, 16)` | | `emulation` | `WNA16 MoE backend 'EMULATION' does not support the deployment configuration since the MoeWNA16 checkpoint layout is not supported.` | | `triton_unfused` | `moe_backend='triton_unfused' is not supported for WNA16 MoE. Expected one of ['triton', 'marlin', 'humming', 'flashinfer_trtllm', 'emulation'].` | | `marlin` | `WNA16 MoE backend 'MARLIN' does not support the deployment configuration since the MoeWNA16 checkpoint layout is not supported.` | | `humming` | `WNA16 MoE backend 'HUMMING' does not support the deployment configuration since kernel does not support current device rocm.` | | `flashinfer_trtllm` | `WNA16 MoE backend 'FLASHINFER_TRTLLM' does not support the deployment configuration since kernel does not support current device rocm.` | ## Questions 1. Is `Qwen3-30B-A3B-GPTQ-Int4` expected to be supported by ROCm AutoGPTQ/WNA16 kernels on gfx1201? 2. Is `qzeros` shape `[16, 640]` a known checkpoint-layout variant requiring a transpose to `[5120, 16]`? 3. Is there a patch, conversion command, or supported quantization format we should use? 4. If this is fixed upstream, which commit/tag contains the fix? The same checkpoint starts loading and consumes approximately 15.65 GiB before the kernel assertion; this is not an OOM or model-download failure. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51971", "createdAt": "2026-08-12T09:16:10Z", "updatedAt": "2026-08-13T02:25:36Z", "timestamp": "2026-08-13T02:25:36Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "rocm", "quantization" ], "author": "MartyMcFlyInTheSky", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51975", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Bugs in kimi-k3 docker image", "text": "### Your current environment
The output of python collect_env.py ```text N/A - discussion about docker image ```
### 🐛 Describe the bug The [kimi-k3](https://hub.docker.com/layers/vllm/vllm-openai/kimi-k3/images/sha256-fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e) image and other [kimi k3 images](https://hub.docker.com/r/vllm/vllm-openai/tags?name=kimi) are outdates and still feature a bug that was fixed in vLLM commit beca88e59ea75a7aa1af72a5ae50188fa91d4e3d (\"[BugFix][K3] Skip moe_intermediate padding when EP is enabled\". ). The vllm [recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3?nodes=4) for kimi-k3 still refers to this image with a bug. The desired fix is to push the fixed kimi-k3 docker image to docker hub. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51975", "createdAt": "2026-08-12T09:41:02Z", "updatedAt": "2026-08-13T08:26:37Z", "timestamp": "2026-08-13T08:26:37Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "kimi", "k3" ], "author": "rwood-97", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51986", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: mnnvl allreduce workspace init hangs 30s and leaks GPU memory on IB-only multi-node", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-124-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA B300 SXM6 AC GPU 1: NVIDIA B300 SXM6 AC GPU 2: NVIDIA B300 SXM6 AC GPU 3: NVIDIA B300 SXM6 AC GPU 4: NVIDIA B300 SXM6 AC GPU 5: NVIDIA B300 SXM6 AC GPU 6: NVIDIA B300 SXM6 AC GPU 7: NVIDIA B300 SXM6 AC Nvidia driver version : 580.173.02 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) 6776P BIOS Model name: Intel(R) Xeon(R) 6776P CPU family: 6 Model: 173 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 Stepping: 1 CPU max MHz: 4600.0000 CPU min MHz: 800.0000 BogoMIPS: 4600.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 6 MiB (128 instances) L1i cache: 8 MiB (128 instances) L2 cache: 256 MiB (128 instances) L3 cache: 672 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-31,128-159 NUMA node1 CPU(s): 32-63,160-191 NUMA node2 CPU(s): 64-95,192-223 NUMA node3 CPU(s): 96-127,224-255 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS Not affected; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.1 vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: \u001b[4mGPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 CPU Affinity NUMA Affinity GPU NUMA ID\u001b[0m GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PXB NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS 0-31,128-159 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS 0-31,128-159 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS 32-63,160-191 1 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS 32-63,160-191 1 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PXB NODE NODE NODE SYS SYS 64-95,192-223 2 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE PXB NODE NODE SYS SYS 64-95,192-223 2 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB NODE 96-127,224-255 3 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PXB 96-127,224-255 3 N/A NIC0 PXB NODE SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NIC1 NODE PXB SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NIC2 NODE NODE SYS SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS SYS SYS SYS SYS SYS SYS NIC3 NODE NODE SYS SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS SYS SYS SYS SYS SYS SYS NIC4 SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NIC5 SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NIC6 SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS NIC7 SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS NIC8 SYS SYS SYS SYS NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS NIC9 SYS SYS SYS SYS NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS NIC10 SYS SYS SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE NIC11 SYS SYS SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_5 NIC2: mlx5_6 NIC3: mlx5_7 NIC4: mlx5_8 NIC5: mlx5_9 NIC6: mlx5_10 NIC7: mlx5_11 NIC8: mlx5_12 NIC9: mlx5_13 NIC10: mlx5_14 NIC11: mlx5_15 ============================== Environment Variables ============================== NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 CUDA_VERSION=13.0.2 LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 NVIDIA_VISIBLE_DEVICES=GPU-e713112b-69d2-1c97-3e3f-1d4928b14252,GPU-30ad23ee-a87d-946c-008c-0f642a30b03b,GPU-4be8db1a-84d2-ef7e-a378-908a70631412,GPU-328baf3a-8c49-14dd-9968-319bbb890442,GPU-d03d7060-7248-aeab-14d2-683c340e2be8,GPU-a9a9aff2-1e7b-b90d-9259-e5195fecde40,GPU-198f35fd-27f1-5bb4-f593-20bfb9cc5c4e,GPU-5278c0f8-77d8-3711-e6f8-399512c0d3cc NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_ENABLE_CUDA_COMPATIBILITY=0 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0 VLLM_USAGE_SOURCE=production-docker-image VLLM_BUILD_COMMIT=6e448d0ea9bf3d88d898b65449ca6dc2aec170ac VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/5006 VLLM_IMAGE_TAG=vllm/vllm-openai:v0.27.1 NCCL_SOCKET_IFNAME=bond0 NCCL_IB_PKEY=2 NCCL_IB_HCA=mlx5_0,mlx5_5,mlx5_8,mlx5_9,mlx5_10,mlx5_11,mlx5_14,mlx5_15 VLLM_ENGINE_READY_TIMEOUT_S=14400 NCCL_DMABUF_ENABLE=0 VLLM_ALLREDUCE_USE_FLASHINFER=0 VLLM_USE_V2_MODEL_RUNNER=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root VLLM_WORKER_MULTIPROC_METHOD=spawn ```
### 🐛 Describe the bug ## What happens Since the auto backend resolution started defaulting to `mnnvl` (#47219 / #47589), startup on non-fabric multi-node clusters attempts mnnvl workspace creation, which: 1. Stalls ~30s per workspace attempt (up to 2: non-quant + quant), then logs: - `Failed to initialize FlashInfer All Reduce workspace: [fd_exchange] timed out connecting to /tmp/cuda_fd_xchg_...` - `Failed to initialize FlashInfer Allreduce norm fusion workspace with backend=mnnvl` 2. I'm not sure why but we subsequently hit OOM on worker nodes. Disabling the pass avoids both: `--compilation-config '{\"pass_config\": {\"fuse_allreduce_rms\": false}}'` On \"vllm/vllm-openai:kimi-k3\" this did not occur because the mnnvl workspace was not attempted on this topology. ## Expected vLLM should pre-check fabric support before attempting multi-node mnnvl workspace creation, and skip cleanly on IB-only topologies instead of relying on a 30s timeout inside flashinfer. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51986", "createdAt": "2026-08-12T11:49:24Z", "updatedAt": "2026-08-13T12:16:30Z", "timestamp": "2026-08-13T12:16:30Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "kimi" ], "author": "appleparan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:51993", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: security: bump minimum setuptools version in requirements/common.txt", "text": "### Your current environment The output of python collect_env.py ```text OS: Linux (Termux / Android) Python Version: 3.11+ vLLM Repository: main branch audit ### 🐛 Describe the bug A security audit on `requirements/common.txt` identified known security advisories affecting the pinned version of `setuptools`: - **setuptools (`77.0.3` → Fix: `>=83.0.0`):** Affected by `PYSEC-2025-49` and `PYSEC-2026-3447`. ### Proposed Fix Update the lower bound in `requirements/common.txt`: ```text setuptools>=83.0.0 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51993", "createdAt": "2026-08-12T13:34:01Z", "updatedAt": "2026-08-13T00:00:45Z", "timestamp": "2026-08-13T00:00:45Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug" ], "author": "ev227166-spec", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52023", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: draft_model speculative decoding crashes at init under TP>1 when draft hidden_size > target (TRT-LLM fused allreduce+RMSNorm workspace sized from target only)", "text": "### Summary With `draft_model` speculative decoding under tensor parallelism (`TP>1`), the engine crashes at init when the **draft model's `hidden_size` is larger than the target model's**. The `fuse_allreduce_rms` compilation pass builds the FlashInfer TRT-LLM fused allreduce+RMSNorm workspace with a token budget derived from the **target** model's `hidden_size` only. When the draft model then runs its own forward under TP, its larger `hidden_size` overflows that workspace and `check_trtllm_allreduce_fusion_workspace_metadata` raises, aborting startup. `draft_hidden < target_hidden` and `TP=1` both work; the crash is specific to **`draft_hidden > target_hidden` AND `TP>1`**. ### Environment - vLLM `v0.27.1`, V1 engine, `dtype=torch.bfloat16` - 2×H100 PCIe (80GB), `tensor_parallel_size=2` - `compilation_config.pass_config.fuse_allreduce_rms: True` (default), FlashInfer allreduce backend - Full raw server logs for all arms (`collect_env` included): https://github.com/seongyun1104/depthchart/tree/39b1557/draft_model_tp_bug ### Reproduction | Arm | target (TP) | draft | draft_hidden vs target_hidden | result | |-----|-------------|-------|-------------------------------|--------| | A | Qwen3-30B-A3B (**TP=1**) | Qwen3-4B | **2560 > 2048** | OK (acceptance 83.8%, normal) | | B | Qwen3-30B-A3B (**TP=2**) | Qwen3-4B | **2560 > 2048** | **CRASH** at init | | C | Qwen3-30B-A3B (TP=2) | Qwen3-0.6B | 1024 < 2048 | OK (acceptance 73.1%, coherent) | Arm A vs B isolates the trigger to **`TP>1`**: same `draft_hidden > target_hidden` pair, only the parallelism differs. Arm C shows the same TP=2 target is fine when `draft_hidden < target_hidden`. Arm B command (greedy): ``` vllm serve Qwen/Qwen3-30B-A3B \\ --tensor-parallel-size 2 \\ --max-model-len 4096 --max-num-seqs 8 --gpu-memory-utilization 0.9 \\ --override-generation-config '{\"do_sample\": false}' \\ --speculative-config '{\"method\": \"draft_model\", \"model\": \"Qwen/Qwen3-4B\", \"num_speculative_tokens\": 1}' ``` (Qwen3-4B `hidden_size=2560` as draft; Qwen3-30B-A3B `hidden_size=2048` as target.) ### Traceback (arm B, verbatim) ``` File \".../vllm/compilation/passes/fusion/allreduce_rms_fusion.py\", line 265, in call_trtllm_fused_allreduce_norm ... check_trtllm_allreduce_fusion_workspace_metadata( File \".../flashinfer/comm/trtllm_ar.py\", line 1026, in check_trtllm_allreduce_fusion_workspace_metadata raise ValueError(error_msg) ValueError: Workspace validation failed: - token_num (8192) * hidden_dim (2560) exceeds workspace max_token_num (8192) * hidden_dim (2048). This may cause Illegal Memory Access. ``` `hidden_dim (2560)` is the draft (Qwen3-4B); the workspace budget `hidden_dim (2048)` is the target (Qwen3-30B-A3B). ### Where it comes from The fused-allreduce workspace token budget is `max_token_num = max_size // (hidden_dim * element_size)`, and `hidden_dim` is taken from the target model config in the pass. The draft model's forward under TP issues the same fused allreduce with its own (larger) `hidden_size`, exceeding the target-sized budget. Note (not a proposed fix): patching the pass `__init__` to use `max(target_hidden, draft_hidden)` did **not** resolve it — the run still reports `hidden_dim (2048)` in the validation error, so the workspace metadata that is actually checked is created upstream of that pass field. A correct fix has to reach the workspace **creation** site so its stored `hidden_dim` accounts for the draft model. ### Relationship to #50877 / #50932 Same theme (FlashInfer allreduce workspace/buffer under a draft model that adds allreduce traffic), but a **distinct backend**: - #50877 / PR #50932 — the **MNNVL** path (`trtllm_mnnvl_ar`, Lamport buffer rotation); #50932 touches `flashinfer_all_reduce.py` / `fused_allreduce_gemma_rms_norm.py` and adds an `is_buffer_size_sufficient` admission guard. - This report — the **TRT-LLM fused allreduce+RMSNorm** compilation-pass path (`allreduce_rms_fusion.py` → `flashinfer/comm/trtllm_ar.py`), which #50932 does not touch. Filing separately so the TRT-LLM path is tracked; happy to fold into #50877 if maintainers prefer a single tracking issue.", "url": "https://github.com/vllm-project/vllm/issues/52023", "createdAt": "2026-08-12T17:01:32Z", "updatedAt": "2026-08-13T17:41:35Z", "timestamp": "2026-08-13T17:41:35Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52025", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Rust frontend renders caller-supplied chat templates with no evaluation budget: a 116-byte request body costs 55 CPU-seconds", "text": "### Your current environment vllm/collect_env.py output is not included: this reproduces entirely against the Rust workspace under rust/, driven by cargo test and the in-tree mock engine. The vLLM Python package is not installed on the repro machine and there is no GPU, so collect_env.py would report N/A for essentially every field it exists to collect. Nothing in the affected code path touches torch, CUDA, or the Python runtime. Happy to provide a full collect_env.py dump from a GPU install if that is needed for triage.
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug ### Summary The Rust frontend renders a caller-supplied Jinja `chat_template` from the request body with **no evaluation budget of any kind**. Cost is `O(N^depth)` with both terms written into the template by the caller, so a 116-byte body can occupy a request-runtime worker thread for 55 seconds. The operator cannot turn the feature off: `--trust-request-chat-template=false`, the setting Python vLLM defaults to, is a hard startup error in the Rust frontend. This is not \"templates can be expensive\". vLLM already bounds this class of cost elsewhere, and minijinja already bounds the same primitive — **and both bounds are walked around by the same omission**. That is the substance of the report. **Component:** `rust/src/chat/src/renderer/hf/template.rs`, reached from `POST /v1/chat/completions`. Read at commit `5c4fe4b`. **Severity (our reading):** remote, unauthenticated, repeatable denial of service against the request-runtime pool; ### The defect `poc.rs` below reproduces all of this on a plain release build of minijinja with vLLM's exact feature set. **Cost is `O(N^depth)`, and depth costs about 40 bytes a level:** ``` depth range bytes iterations seconds 2 300 78 90,000 0.005 2 1000 80 1,000,000 0.057 3 200 116 8,000,000 0.473 3 400 116 64,000,000 3.659 3 999 116 997,002,999 55.6 4 100 154 100,000,000 5.9 ``` The 116-byte template is: in jinja: ```jinja {% for x0 in range(999) %}{% for x1 in range(999) %}{% for x2 in range(999) %}{% endfor %}{% endfor %}{% endfor %}ok ``` Every `range()` in it is individually legal and individually small. **Guard 1 — minijinja bounds a single `range()`, and nesting walks around it.** ``` range( 100,000) accepted range( 1,000,000) REFUSED -- invalid operation: range has too many elements ``` The engine's authors did consider a bound on this exact primitive. It is per-range, so composing three legal ranges multiplies past it. minijinja also ships a `fuel` feature for bounding total evaluation work; `grep -rn set_fuel` over `rust/` returns nothing. **Guard 2 — vLLM isolates the runtime, and does not bound the task on it.** `rust/src/server/src/runtime.rs:14-16` introduces a dedicated request runtime with the comment: > *\"routes so CPU-heavy request preparation does not monopolize the HTTP runtime's worker > queue\"* `/v1/chat/completions` is in `OFFLOADED_PATHS` (`middleware/offload.rs:25-35`) and is spawned onto it (`offload.rs:82`). **This works, and it bounds the blast radius**: `/health` and `/metrics` stay responsive, so this is not a whole-server outage and we do not claim one. But that runtime is a bounded pool — `worker_threads = min(available_parallelism, 32)` (`runtime.rs:9,21`) — and the render is a plain synchronous call on it with no timeout: ```rust // rust/src/chat/src/renderer/hf/template.rs:124-127 pub fn apply(&self, ctx: TemplateContext<'_>) -> Result { let tmpl = self.env.get_template(\"chat\")?; tmpl.render(ctx).map_err(TemplateError::from) } ``` The same codebase uses `tokio::task::spawn_blocking` for its other CPU-heavy request preparation — image, audio and video preprocessing (`chat/src/multimodal/{image.rs:67, audio.rs:62, video.rs:85}`). The pattern is applied deliberately elsewhere; offloading alone would not fix this anyway without a cost bound. **So: at most 32 concurrent hostile requests occupy every worker and the chat-completions plane stops serving, while the health check stays green.** A monitor watching `/health` sees nothing. **No effective ceiling on template size.** `DEFAULT_JSON_BODY_LIMIT_BYTES = 32 * 1024 * 1024` (`rust/src/server/src/routes.rs:33`). 116 bytes already buys 55 seconds. ### Why this is the same class you already bound elsewhere The Security Guide documents decode limits for exactly this failure mode: | control | default | |---|---| | `VLLM_MAX_IMAGE_PIXELS` | ~179M pixels | | `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` | 25 | | `VLLM_MAX_AUDIO_DECODE_DURATION_S` | 600 | with the note that setting any of them to `0` *\"removes the protection against resource-exhaustion attacks\"*. Multimodal decode is bounded; chat-template evaluation is not. ### Reproduction Needs no GPU and no model weights. ```toml # Cargo.toml — feature set copied from vLLM's rust/Cargo.toml [dependencies] # 2.18.0 is what vLLM's rust/Cargo.lock pins minijinja = { version = \"=2.18.0\", features = [ \"unstable_machinery\", \"json\", \"builtins\", \"loader\", \"loop_controls\", \"preserve_order\", ] } ``` ```rust use minijinja::{context, Environment}; use std::time::Instant; fn timed(tmpl: &str) -> (f64, Result) { let env = Environment::new(); let t = Instant::now(); let r = env.render_str(tmpl, context! {}); let dt = t.elapsed().as_secs_f64(); (dt, r.map(|s| s.len()).map_err(|e| e.to_string().chars().take(44).collect())) } fn nest(depth: usize, n: u64) -> String { let mut t = String::new(); for i in 0..depth { t.push_str(&format!(\"{{% for x{i} in range({n}) %}}\")); } for _ in 0..depth { t.push_str(\"{% endfor %}\"); } t.push_str(\"ok\"); t } fn main() { println!(\"== a single range() is bounded ==\"); for n in [100_000u64, 1_000_000] { let (_, r) = timed(&format!(\"{{% for i in range({n}) %}}{{% endfor %}}ok\")); println!(\" range({n:>9}) {}\", if r.is_ok() { \"accepted\".into() } else { format!(\"REFUSED -- {}\", r.unwrap_err()) }); } println!(\"\\n== nesting walks around it: O(N^depth) ==\"); println!(\" {:>5} {:>7} {:>7} {:>18} {:>10}\", \"depth\", \"range\", \"bytes\", \"iterations\", \"seconds\"); for (d, n) in [(2usize, 300u64), (2, 1000), (3, 200), (3, 400), (3, 999), (4, 100)] { let t = nest(d, n); let (dt, _) = timed(&t); println!(\" {d:>5} {n:>7} {:>7} {:>18} {dt:>10.3}\", t.len(), (n as u128).pow(d as u32)); } } ``` ### The operator cannot opt out Python vLLM gates this behind `trust_request_chat_template`, default **off**: ```python # vllm/entrypoints/chat_utils.py trust_request_chat_template: bool = False # \"Whether to trust the chat template provided in the request. If False, the server # will always use the chat template specified by --chat-template or the ones from # tokenizer.\" ``` The Rust frontend declares the same flag under `#[command(next_help_heading = \"Options not implemented in Rust frontend yet\")]` (`rust/src/cmd/src/cli/unsupported.rs`), where `Unsupported::from_str` always returns `Err`: ```console $ vllm-rs serve --no-trust-request-chat-template error: invalid value 'true' for '--trust-request-chat-template [<...>]': argument is not implemented in Rust frontend yet $ vllm-rs serve --trust-request-chat-template=false error: invalid value 'false' for '--trust-request-chat-template [<...>]': argument is not implemented in Rust frontend yet ``` Both spellings of the safe setting are hard startup errors, while the request field is accepted and reaches the renderer (`routes/openai/chat_completions/types.rs`, `convert.rs:160`). We recognise the flag is on a declared to-do list, and that \"implement the missing flag\" on its own would be a feature request. It is here because it is the reason the defect above cannot currently be mitigated by configuration — not as the defect. ### What we ruled out We expected a stronger primitive and did not find one. `build_environment` (`rust/src/chat/src/renderer/hf/template.rs:29-40`) is, in full: ```rust let mut env = Environment::new(); env.set_trim_blocks(true); env.set_lstrip_blocks(true); env.add_template_owned(\"chat\".to_owned(), template)?; env.set_unknown_method_callback(minijinja_contrib::pycompat::unknown_method_callback); env.add_filter(\"tojson\", hf_tojson_filter); ``` **No loader is registered.** We verified against minijinja directly that *with* a loader `{% include 'hostname' %}` renders that file's contents, and that without one `include`, `extends` and `import` all fail. vLLM enables minijinja's `loader` feature but never calls `set_loader`, so a request-supplied template **cannot read files**. The render context (`TemplateContext`, `template.rs:44-56`) carries messages, tools, documents, special tokens and template kwargs — nothing an unauthenticated caller did not already supply or know. Stated because it bounds the report, and so a reader need not wonder whether we looked. ### Suggested remediation 1. **Bound template evaluation.** Enable minijinja's `fuel`, or wrap `apply()` in a wall-clock budget. This closes the defect regardless of who supplied the template, and it also covers the **model-supplied** path, which no gate protects. 2. **Implement `trust_request_chat_template`, defaulting to `false`,** matching Python vLLM. That removes the request-supplied exposure and lets an operator who deliberately enables it keep the protection from (1). 3. Until either lands, consider rejecting `chat_template` in the request body rather than honouring a field the operator has no way to control. 4. Independently, `spawn_blocking` the render as the multimodal preprocessors already do. Hygiene, not a fix. ### Environment vLLM Rust frontend at `5c4fe4b`. **minijinja 2.18.0**, the version pinned in `rust/Cargo.lock`. rustc stable, release build, Linux x86-64, 32 cores, machine otherwise idle. Three repetitions of the 55.6 s row gave 55.382 / 55.436 / 55.397 — the figure is stable to about ±0.05 s. An earlier draft of this report quoted **86.7 s** for that row. That measurement was taken while the machine was running other builds and fuzzing campaigns, and overstated the cost by about 36%. It is corrected here rather than quietly restated. Absolute seconds will differ by machine in any case; the growth is the claim. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52025", "createdAt": "2026-08-12T17:11:53Z", "updatedAt": "2026-08-12T17:45:18Z", "timestamp": "2026-08-12T17:45:18Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "Yunzez", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:52027", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: logs from a parser loaded with --tool-parser-plugin bypass vLLM's logging config", "text": "### Your current environment
The output of python collect_env.py (trimmed to the relevant parts) ```text vLLM Version : 0.27.1 PyTorch version : 2.13.0+cu130 CUDA used to build PyTorch : 13.0 Python version : 3.12.13 (64-bit runtime) Python platform : Linux-7.0.0-generic-x86_64-with-glibc2.43 OS : Ubuntu 26.04 LTS (x86_64) Is CUDA available : True GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition Nvidia driver version : 595.84 ``` The bug is in the plugin import path and the logging config, so it does not depend on the model, the GPU or the driver.
### 🐛 Describe the bug A tool parser loaded through `--tool-parser-plugin` is imported under a module name taken from its file name: ```python # vllm/tool_parsers/abstract_tool_parser.py @classmethod def import_tool_parser(cls, plugin_path: str) -> None: \"\"\"Import a user-defined parser file from arbitrary path.\"\"\" module_name = os.path.splitext(os.path.basename(plugin_path))[0] import_from_path(module_name, plugin_path) ``` Plugin authors then do what every in-tree parser does - `logger = init_logger(__name__)` - and get a logger named `my_tool_parser`. `DEFAULT_LOGGING_CONFIG` configures only the `vllm` logger, so that logger has no handler and is not under a configured namespace: ```python # vllm/logger.py \"loggers\": { \"vllm\": { \"handlers\": [\"vllm\"], \"level\": envs.VLLM_LOGGING_LEVEL, \"propagate\": False, }, }, ``` Minimal reproduction - the same module, two loggers: ```python # /tmp/plugin.py from vllm.logger import init_logger by_module = init_logger(__name__) # what a plugin naturally writes under_vllm = init_logger(\"vllm.\" + __name__) def say(): by_module.info(\"A-info\") by_module.warning(\"A-warning\") under_vllm.info(\"B-info\") under_vllm.warning(\"B-warning\") ``` ```python # the call import_tool_parser makes from vllm.utils.import_utils import import_from_path import_from_path(\"plugin\", \"/tmp/plugin.py\").say() ``` Output: ``` A-warning INFO 08-12 19:01:36 [plugin.py:7] B-info WARNING 08-12 19:01:36 [plugin.py:8] B-warning ``` So from a plugin parser: * `logger.info` and `logger.debug` are dropped entirely - there is no handler, and Python's `logging.lastResort` fallback is WARNING-level; * `logger.warning` and above leave *outside* vLLM's configuration: unformatted (no timestamp, level or source) and unaffected by `VLLM_LOGGING_LEVEL` or `VLLM_LOGGING_CONFIG_PATH`. Whether they are seen at all then depends on what the deployment does with the root logger and stderr - on our server they never reached the journal, and three parser bugs stayed invisible for a day because of it. The parser had been reporting them the whole time. Nothing above is specific to one parser: it applies to every plugin loaded this way, and `--reasoning-parser-plugin` takes the same path. **Suggested fix.** Import the plugin under a namespaced module name, so `__name__` lands inside the configured namespace and plugin authors need to do nothing: ```python module_name = \"vllm.tool_parsers.plugins.\" + os.path.splitext( os.path.basename(plugin_path))[0] ``` Alternatively `init_logger` could prefix any name that is not already under `vllm.`. Fix in #52026, with a test that walks the logger chain from the name the plugin was imported under and asserts a record from it reaches the `vllm` handler. (Unrelated, noticed while collecting the environment: `collect_env.py` crashes with `AttributeError: 'NoneType' object has no attribute 'splitlines'` in a uv venv when `uv` is not on `PATH` - `run_and_read_all` returns `None` and `run_with_pip` does not check.) ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52027", "timestamp": "2026-08-12T17:27:22Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "honziik", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:52029", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Usage]: RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}", "text": "I was trying to run a standard vllm code ``` python import os from dotenv import load_dotenv load_dotenv() import huggingface_hub HF_token = os.getenv(\"HF_token\") huggingface_hub.login(token=HF_token) from vllm import LLM, SamplingParams prompts = [ \"Hello, my name is\", \"The president of the United States is\", \"The capital of France is\", \"The future of AI is\", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95) llm = LLM(model=\"facebook/opt-125m\") outputs = llm.generate(prompts, sampling_params) for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f\"Prompt: {prompt!r}, Generated text: {generated_text!r}\") ``` but got error similar Issue: [#2104](https://github.com/QwenLM/Qwen3-VL/issues/2104) I saw one of the vllm forms [推理时报错RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}](https://discuss.vllm.ai/t/runtimeerror-engine-core-initialization-failed-see-root-cause-above-failed-core-proc-s/2332) and add `gpu_memory_utilization=0.5` but didn't worked. **can anyone help to debug this.** terminal output ``` bash INFO 08-12 22:35:15 [api_utils.py:273] non-default args: {'gpu_memory_utilization': 0.5, 'disable_log_stats': True, 'model': 'facebook/opt-125m'} INFO 08-12 22:35:17 [model.py:645] Resolved architecture: OPTForCausalLM INFO 08-12 22:35:17 [model.py:1883] Using max model len 2048 INFO 08-12 22:35:17 [scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=8192. INFO 08-12 22:35:17 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) (EngineCore pid=36048) INFO 08-12 22:35:20 [core.py:121] Initializing a V1 LLM engine (v0.27.1) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') (EngineCore pid=36048) INFO 08-12 22:35:21 [parallel_state.py:1640] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.29.246:39473 backend=nccl (EngineCore pid=36048) INFO 08-12 22:35:21 [parallel_state.py:1977] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (EngineCore pid=36048) INFO 08-12 22:35:21 [gpu_worker.py:385] Using V2 Model Runner (EngineCore pid=36048) INFO 08-12 22:35:22 [model_runner.py:308] Loading model from scratch... (EngineCore pid=36048) INFO 08-12 22:35:22 [cuda.py:482] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (EngineCore pid=36048) INFO 08-12 22:35:22 [flash_attn.py:789] Using FlashAttention version 2 (EngineCore pid=36048) ERROR 08-12 22:35:23 [core.py:1349] EngineCore failed to start. ... ... raise RuntimeError( RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} ``` ```bash nvidia-smi Wed Aug 12 23:06:42 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 5050 ... Off | 00000000:01:00.0 Off | N/A | | N/A 40C P8 6W / 60W | 5MiB / 8151MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` [env_package_versions.txt](https://github.com/user-attachments/files/30990912/env_package_versions.txt) ### How would you like to use vllm I want to run inference of a [deepseek-ocr:latest](https://huggingface.co/deepseek-ai/DeepSeek-OCR). I tried to run it as per documentation but got error, so tried with basic vllm code to run facebook/opt-125m, but still got error ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52029", "timestamp": "2026-08-12T17:38:19Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "usage", "quantization" ], "author": "Raja-Vignesh7", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:issue:52034", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Mistral-Small-3.1 FP8 (Pixtral) returns NaN on image inputs with compilation enabled; works with --enforce-eager", "text": "### Your current environment
Environment - vLLM: v0.24.0 - Hardware: 1x NVIDIA H100 80GB (tensor-parallel-size 1) - Model: `RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic` (compressed-tensors FP8-dynamic; only the language-model `Linear` layers are quantized — `vision_tower`, `multi_modal_projector`, and `lm_head` are in the `ignore` list and stay bf16) - Served as `PixtralForConditionalGeneration` (via `--config-format mistral`)
### 🐛 Describe the bug With this FP8 Mistral-Small-3.1 (Pixtral) checkpoint, **text/chat/tool/structured requests work fine, but any request that includes an image returns `NaN`**. The engine does not crash; the API fails to serialize the response: ``` openai.BadRequestError: Error code: 400 - {'error': {'message': 'Out of range float values are not JSON compliant: nan', ...}} ``` Server-side, the request completes (HTTP 200 path) but the generated logits/logprobs contain `NaN`, so `json.dumps` in `create_chat_completion` raises `ValueError: Out of range float values are not JSON compliant: nan`. **Key finding: the NaN only occurs with compilation enabled. Adding `--enforce-eager` makes image requests produce valid output.** So this appears to be a torch.compile / cudagraph issue on the FP8 multimodal path rather than pure FP8 numerical instability (the quantized decoder is fine for text; it only goes NaN once image embeddings are in the sequence, and only when compiled). Text-only requests are unaffected in both modes. #### Reproduce Fails (image → NaN): ```bash vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic \\ --tokenizer-mode mistral --config-format mistral --max-model-len 32768 # then send any chat request containing an image_url -> response contains NaN -> 400 ``` Works (same request succeeds): ```bash vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic \\ --tokenizer-mode mistral --config-format mistral --max-model-len 32768 \\ --enforce-eager ``` #### Notes - The bf16 base model (`mistralai/Mistral-Small-3.1-24B-Instruct-2503`) does **not** exhibit this — only the FP8 checkpoint, and only with compilation on. - Loading this FP8 checkpoint via `--config-format hf` (`Mistral3ForConditionalGeneration`) is not a straightforward comparison point: with `--tokenizer-mode mistral` the HF `PixtralProcessor` and Mistral tokenizer disagree on the dummy `[IMG]` token at startup, and with `--tokenizer-mode auto` the Mistral tokenizer backend is selected and then `vllm/tokenizers/registry.py` calls `.is_fast` on it (`AttributeError: CachedMistralCommonBackend has no attribute is_fast`). Those look like separate issues; this report is specifically about the compiled FP8 multimodal NaN. ### Workaround `--enforce-eager` (disables torch.compile + cudagraphs), at a decode-throughput cost.", "url": "https://github.com/vllm-project/vllm/issues/52034", "createdAt": "2026-08-12T18:45:12Z", "updatedAt": "2026-08-12T20:44:23Z", "timestamp": "2026-08-12T20:44:23Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "quantization" ], "author": "wjhrdy", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:52038", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: LoRA adapter support for DFlash speculative decoding draft models", "text": "### Motivation. vLLM supports serving many LoRA adapters over a single base target model, but speculative decoding still requires a separate, fully-trained draft model per domain. When serving many domains, this is expensive: each domain needs its own drafter (e.g. 0.8B) held in memory, with corresponding I/O cost to load and swap. This RFC proposes applying LoRA adapters to the DFlash drafter, so a single base drafter can be specialized per-domain via small adapters instead of full drafters. In our experiments an R64 adapter is ~28x smaller than the 0.8B drafter it replaces, substantially reducing memory footprint and I/O latency when serving many domains, while keeping drafter quality within ~2% of a fully-trained per-domain drafter (measured by acceptance length behavior). This mirrors the efficiency win vLLM already provides for target-model LoRA, extended to the speculative-decoding drafter. ### Proposed Change. Allow a LoRA adapter to be applied to the DFlash drafter, independent of the target model: - The adapter wraps all drafter layers except `lm_head` and `embed_tokens`, loaded from a stage-2-trained LoRA checkpoint. - The drafter LoRA is specified via additional arguments in the speculative config, distinct from any target-model adapter. - The correct adapter is routed to the drafter and applied along its forward path. Initial support targets DFlash drafters specifically (tested with a GPT-OSS-120B target and a 0.8B DFlash drafter). This is distinct from target-side multi-adapter work (#49705, #48297) and from prior Eagle-style draft-model attempts. **Prior art / known issues** Closed PR #11966 (V0, Eagle-era) attempted LoRA + spec decode and hit a vocab-size mismatch (e.g. 768 vs 4096) from differing vocab padding on draft vs. target, and disabled draft-side LoRA \"until the API interface is changed to inject the corresponding LoRA adapter for the draft model.\" In our DFlash testing we did not encounter this mismatch — likely because the adapter excludes `lm_head` and `embed_tokens`, avoiding the padded-vocab layers. We'd welcome maintainer input on whether this holds across all DFlash configurations. **Alternatives considered** - Full per-domain drafters (status quo): simple but costly — one full drafter per domain in memory. - Offline-merging the LoRA into drafter weights: avoids runtime adapter logic but still stores a full merged drafter per domain, forfeiting the 28x saving and per-request swappability. Serving the adapter dynamically preserves both the memory saving and per-request swappability. **Testing & benchmarks** Acceptance behavior and throughput comparing base drafter, LoRA'd drafter, and a fully-trained per-domain drafter; memory-footprint comparison (adapter vs. full drafter) across N domains. ### Feedback Period. One week. ### CC List. @LucasWilkinson @WoosukKwon ### Any Other Things. There are some details I can't share about the experiment results because that involved classified data and domains while my time at Capital One AI Foundations where I worked on this project, but I will share what I can based on your questions. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52038", "createdAt": "2026-08-12T19:09:15Z", "updatedAt": "2026-08-12T19:09:15Z", "timestamp": "2026-08-12T19:09:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "RFC" ], "author": "anshkaggarwal22", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:52040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: MM input tensors are re-broadcast to all TP workers on every request, even when the prefix cache fully covers the image tokens (~19 ms per unique image)", "text": "## Environment - vLLM v0.25.1 (official `vllm/vllm-openai` Docker image), single API server process - TP=4 on one node (4 GPUs), `mm_processor_cache_type: lru` (behavior identical under `shm`) - Gemma-4-31B-IT with vision; `--mm-processor-kwargs {\"max_soft_tokens\": 560}` → `pixel_values` per image is `(1, 5040, 768)` fp32 ≈ **15.5 MB** - `--limit-mm-per-prompt {\"image\": 32}` ## Workload Assistant with a rolling visual-context window: each turn re-sends the same ~26 images (byte-identical, position-stable) plus one fresh frame. Prefix caching covers all history image tokens (this works great), and the mm processor cache hits 26/26 (also works great — with `lru`). Requests are therefore ~100% cached, and the encoder never runs for the history images. ## Problem Even in that fully-cached state, TTFT grows **linearly with the number of unique images attached**: ~23 ms per unique image, of which ~19 ms is the EngineCore re-serializing each unique tensor and re-copying it into the shm broadcast ring to the TP workers — on **every request**, although the workers will never consume those tensors (their soft-token spans are entirely within the prefix-cache-covered region, so no encoder run is ever scheduled). ## Measurements (all in-pod via localhost, byte-identical resends, all shapes warm) **1. Fully-cached TTFT floor vs unique image count** (queue = 0.0 ms, prefill ≈ 14 ms in all cases; `mm_cache_hits == queries` on every request): | unique images | TTFT (ms) | |---|---| | 2 | 69 | | 8 | 188 | | 13 | 310 | | 20 | 472 | | 26 | **634** | → slope ≈ **23 ms per unique image**, all CPU, none of it GPU. **2. Duplicate-image probe** — same request shape/token count, 26 image slots, but all 26 slots reference **one** image (byte-identical data URLs) vs 26 **distinct** images, everything cached in both cases: - 26 distinct cached images: **639 ms** - 26 copies of one cached image: **156 ms** → the cost is per-*unique* image, not per attached item. (Pickle's object memoization explains the collapse: the receiver cache returns the same tensor object per hash, so duplicates serialize once.) **3. py-spy on the EngineCore during a pure cache-hit request loop** (26 distinct cached images per request): of 29.4 s sampled, **6.4 s in `shm_broadcast.enqueue`** and **4.4 s in `torch.serialization._legacy_save`** (rest is idle `sched_yield`). The API server process is nearly idle under the same load (parse layer measured separately at ~0.9 ms/image). **4. Serializer baseline**: `MsgpackEncoder`/`MsgpackDecoder` (the path used API-server→EngineCore) round-trips the same 15.5 MB tensor in **0.01 ms** (zero-copy); `tensor.clone()` is 0.06 ms. The broadcast path's per-tensor cost is ~19 ms. ## Analysis (checked against current `main`) - API-server→EngineCore transport is already hash-only on hits — `MultiModalFeatureSpec.data`'s docstring says it explicitly: *\"Can be `None` if the item is cached, to skip IPC between API server and engine core processes\"* (`vllm/multimodal/inputs.py`). This half of the design works. - The EngineCore then re-populates the tensors from its receiver cache (`v1/engine/core.py:976`, `get_and_update_features`) and ships them to the TP workers inside `NewRequestData.mm_features` at admission (`v1/core/sched/output.py`) — per unique tensor, per request. Notably `NewRequestData` already carries `num_computed_tokens` in the same dataclass, so the information needed to skip is co-located with the thing to be skipped. - On v0.25.1 the broadcast serializes via pickle → `torch._legacy_save` (full byte rewrite of each 15.5 MB). Current `main` routes CPU tensors through `_reduce_tensor` / out-of-band buffers (`shm_broadcast.py:823-846`), which should remove most of the serialization half — but the ring copy remains, and more fundamentally **the tensors are shipped at all** for requests whose image placeholders are fully covered by the prefix cache. ## Proposal Attach an mm item's tensors to the **step where its encoder run is scheduled**, rather than to request admission: - At admission the scheduler already knows `num_computed_tokens` and each mm item's placeholder span; an item whose span lies entirely within the computed prefix needs no encoder run and therefore no tensor on the workers. The scheduler already plans encoder execution explicitly (`scheduled_encoder_inputs`, `v1/core/sched/scheduler.py:679`), so tensors could travel with that instead of with `NewRequestData`. - The receiver cache already retains tensors for the request lifetime, so preemption/recompute can still ship them lazily on demand. - For our workload this turns ~25 shipped tensors per request into 0–1, i.e. the fully-cached floor becomes flat in image count (~4 ms/image of unavoidable parse+hash remains). Happy to contribute a PR for this (and/or backport-friendly serializer improvements) with guidance from maintainers on the preferred shape. ## Reproduction sketch ```python # against any TP>1 vision deployment with prefix caching + processor cache enabled: # 1. build a chat request with N distinct images + fixed text; send twice (second send = fully cached) # 2. build the same request but with all N image slots referencing ONE image; send twice # 3. compare second-send TTFTs and watch vllm:mm_cache_* and queue/prefill counters: # distinct-N grows ~linearly in N; dup-N stays near the 1-image floor; queue/prefill flat in both. ```", "url": "https://github.com/vllm-project/vllm/issues/52040", "createdAt": "2026-08-12T19:44:46Z", "updatedAt": "2026-08-12T19:44:46Z", "timestamp": "2026-08-12T19:44:46Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "sseanliu", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:52042", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' / vllm 0.27.1 / DeepSeek-V4-Flash-0731", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (aarch64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-1059-nvidia-64k-aarch64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GH200 144G HBM3e GPU 1: NVIDIA GH200 144G HBM3e Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 144 On-line CPU(s) list: 0-143 Vendor ID: ARM Model name: Neoverse-V2 Model: 0 Thread(s) per core: 1 Core(s) per cluster: 72 Socket(s): - Cluster(s): 2 Stepping: r0p0 Frequency boost: disabled CPU max MHz: 3474.0000 CPU min MHz: 81.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti L1d cache: 9 MiB (144 instances) L1i cache: 9 MiB (144 instances) L2 cache: 144 MiB (144 instances) L3 cache: 228 MiB (2 instances) NUMA node(s): 18 NUMA node0 CPU(s): 0-71 NUMA node1 CPU(s): 72-143 NUMA node2 CPU(s): NUMA node3 CPU(s): NUMA node4 CPU(s): NUMA node5 CPU(s): NUMA node6 CPU(s): NUMA node7 CPU(s): NUMA node8 CPU(s): NUMA node9 CPU(s): NUMA node10 CPU(s): NUMA node11 CPU(s): NUMA node12 CPU(s): NUMA node13 CPU(s): NUMA node14 CPU(s): NUMA node15 CPU(s): NUMA node16 CPU(s): NUMA node17 CPU(s): Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.1 vLLM Build Flags: CUDA Archs: 8.0 8.7 8.9 9.0 10.0 11.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NODE 0-71 0 2 GPU1 NV18 X SYS 72-143 1 10 NIC0 NODE SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: roceP2p1s0f0 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=void VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/5006 NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=8.0 8.7 8.9 9.0 10.0 11.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_IMAGE_TAG=vllm/vllm-openai:v0.27.1 VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 VLLM_ENABLE_CUDA_COMPATIBILITY=0 VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/aarch64-linux-gnu VLLM_BUILD_COMMIT=6e448d0ea9bf3d88d898b65449ca6dc2aec170ac PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug `/vllm-workspace/benchmarks/kernels/benchmark_moe.py` used with DeepSeek-V4-Flash-0731 fails with `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'` To reproduce ; start v0.27.1 Docker image: ``` docker run --gpus all --ipc=host \\ --entrypoint /bin/bash \\ -it vllm/vllm-openai:v0.27.1-aarch64 ``` Run: ``` pip install ray cd /vllm-workspace/benchmarks/kernels python3 benchmark_moe.py -tp 2 --tune --trust-remote-code --model \"deepseek-ai/DeepSeek-V4-Flash-0731\" ``` Result: ``` Namespace(model='deepseek-ai/DeepSeek-V4-Flash-0731', tp_size=2, enable_expert_parallel=False, dtype='auto', use_deep_gemm=False, seed=0, batch_size=None, tune=True, trust_remote_code=True, model_prefix=None) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. INFO 08-12 20:16:45 [config.py:774] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM. Traceback (most recent call last): File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 1074, in main(args) File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 884, in main E, topk, intermediate_size, hidden_size = get_model_params(config) ^^^^^^^^^^^^^^^^^^^^^^^^ File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 822, in get_model_params E = config.num_local_experts ^^^^^^^^^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py\", line 482, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py\", line 312, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52042", "createdAt": "2026-08-12T20:25:18Z", "updatedAt": "2026-08-12T20:53:51Z", "timestamp": "2026-08-12T20:53:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "tomschelsen", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:issue:52049", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Gemma 4 31B MTP - Slower T/S at high context", "text": "### Your current environment ``` Collecting environment information... ============================== System Info ============================== OS : Ubuntu 26.04 LTS (x86_64) GCC version : (Ubuntu 15.2.0-16ubuntu1) 15.2.0 Clang version : Could not collect CMake version : version 4.2.3 Libc version : glibc-2.43 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu132 Is debug build : False CUDA used to build PyTorch : 13.2 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Aug 7 2026, 02:25:39) [Clang 22.1.3 ] (64-bit runtime) Python platform : Linux-7.0.0-29-generic-x86_64-with-glibc2.43 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.3.73 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 GPU 3: NVIDIA GeForce RTX 3090 Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD Ryzen 9 7950X 16-Core Processor CPU family: 25 Model: 97 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 Frequency boost: enabled CPU(s) scaling MHz: 62% CPU max MHz: 5883.1968 CPU min MHz: 425.2920 BogoMIPS: 8999.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze Virtualization: AMD-V L1d cache: 512 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 16 MiB (16 instances) L3 cache: 64 MiB (2 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.4.0.1 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.2.75 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.2.78 [pip3] nvidia-cuda-runtime==13.2.75 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.2.0.46 [pip3] nvidia-cufile==1.17.1.22 [pip3] nvidia-curand==10.4.2.55 [pip3] nvidia-cusolver==12.2.0.1 [pip3] nvidia-cusparse==12.7.10.1 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.2 [pip3] nvidia-cutlass-dsl-libs-base==4.6.2 [pip3] nvidia-cutlass-dsl-libs-core==4.6.2 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.2 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.2 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.2.75 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu132 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cpu [pip3] torchcodec==0.15.0+cu132 [pip3] torchvision==0.28.0+cu132 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.1rc1.dev640+g1ab2801dd (git sha: 1ab2801dd) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PXB PXB PXB 0-31 0 N/A GPU1 PXB X PXB PXB 0-31 0 N/A GPU2 PXB PXB X PIX 0-31 0 N/A GPU3 PXB PXB PIX X 0-31 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/cuda-13/lib64: PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_westfox35 ``` ### 🐛 Describe the bug I did a quick search but couldn't find anything. Not saying there isnt anything though. I've tried this on vllm, vllm nightly and ``` uv pip install \"git+https://github.com/vllm-project/vllm.git@1ab2801ddebe31b75dd6022c69113b610bbdc950\" --torch-backend=auto ``` Initially MTP is fast (ish). When at higher contexts (say 50k+) MTP on VLLM will be very slow. When disabling MTP, it's 3x faster or higher in gen T/S. Llama.cpp and ik_llama.cpp do not suffer from this fate. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52049", "createdAt": "2026-08-12T21:24:05Z", "updatedAt": "2026-08-12T21:28:12Z", "timestamp": "2026-08-12T21:28:12Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "frenzybiscuit", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52051", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: gpt-oss-120b MXFP4 + --enable-expert-parallel crashes at startup: modular MoE finalize allocates padded (3072) output for an unpadded (2880) expert result", "text": "**Before submitting a new issue...** - [x] I have searched for existing and past issues, at four breadths: the symptom (padded/unpadded shape mismatch in MoE finalize), the subsystem (`gpt-oss` MXFP4 expert-parallel MoE), the faulting file/class (`modular_kernel.py`, `FusedMoEPrepareAndFinalize`), and the exact symbols (`has_unpadded_output`, `hidden_dim_unpadded`). The closest existing item is PR #44452, which restructures this same call site but never mentions MXFP4 padding or DP+EP; #42118 is the same file/class with a different defect (unswizzled mxfp8 scales). --- ### Your current environment
The output of python collect_env.py ```text `collect_env.py` output is not available: this was found by an automated CI triage agent from a batch job log, and the container is not retained after the run, so the script could not be executed retroactively. The relevant values were taken verbatim from the engine's own startup log instead: vLLM version : 0.26.1rc1.dev602+g65b7662d3 PyTorch : 2.x + CUDA 13.0 wheel (from the vLLM nightly container built on this commit; exact torch build string not printed in the log) Python : 3.12 CUDA : 13.0 NCCL : 2.30.7 in the plain nightly container; 2.29.7 in the container that additionally installs FlashInfer nightlies (pynccl prints \"vLLM is using nccl==2.30.7\" and \"...==2.29.7\" respectively). The crash is byte-identical at both NCCL versions. FlashInfer : 0.6.18.dev20260811 (flashinfer-python / -cubin / -jit-cache+cu130 nightlies) GPU : 8 x NVIDIA B200 (sm100), single node Model : openai/gpt-oss-120b, MXFP4 checkpoint (hidden_size = 2880) ``` Selected engine config echoed by vLLM at startup: ```text dtype=torch.bfloat16, max_seq_len=4096, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=8, quantization=gpt_oss_mxfp4, enforce_eager=False, load_format=runai_streamer ``` Two log lines matter for backend selection: ```text INFO [mxfp4.py:528] Using 'FLASHINFER_TRTLLM_MXFP4_BF16' Mxfp4 MoE backend. INFO [mxfp4.py:1819] Using MoEPrepareAndFinalizeNaiveDPEPModular INFO [mxfp4.py:1820] Using TrtLlmMxfp4ExpertsModular ``` The naive DP/EP all-to-all path is selected under these plain default flags: ```text INFO [cuda_communicator.py:203] Using AgRsAll2AllManager all2all manager. ``` Note on DeepEP: one of the two containers additionally logs `DeepEP v2 requires NCCL >= 2.30.4 but found 2.29.7`, but **that is not what puts this configuration on the naive path.** The other container ships NCCL 2.30.7, prints no DeepEP-unavailable message at all, and still selects `MoEPrepareAndFinalizeNaiveDPEPModular` and still crashes identically. The naive DP/EP `prepare_finalize` is simply what `--enable-expert-parallel` resolves to here under default flags, so raising NCCL does not avoid the bug.
### 🐛 Describe the bug `gpt-oss-120b` (MXFP4) fails deterministically during V1 startup memory profiling when expert parallelism is enabled on Blackwell, before the server ever becomes ready. The same build serves the model fine without `--enable-expert-parallel`. #### Reproduce On a single 8xB200 node, with the `FLASHINFER_TRTLLM_MXFP4_BF16` MoE backend selected automatically (no MoE-backend or activation override): ```console vllm serve openai/gpt-oss-120b \\ --tensor-parallel-size 1 \\ --data-parallel-size 8 \\ --enable-expert-parallel \\ --max-model-len 4096 ``` The failing ingredients are: `hidden_size` that is **not** a multiple of 256 (gpt-oss-120b is 2880), the `FLASHINFER_TRTLLM_MXFP4_BF16` backend, and a modular `prepare_finalize` that copies into a caller-provided buffer — here `MoEPrepareAndFinalizeNaiveDPEPModular`, which is what `--enable-expert-parallel` resolves to here under default flags (it is reached at NCCL 2.30.7 as well as 2.29.7, so it is not a consequence of DeepEP being unavailable). Removing `--enable-expert-parallel` avoids it; so does forcing `--quantization-config.moe.activation mxfp8`, but that changes activation precision (see \"Why the mxfp8 workaround is not a fix\" below). #### Actual behavior Every DP rank aborts in `determine_available_memory` → `profile_run` → `_dummy_run` → `gpt_oss.py forward` → `torch.ops.vllm.moe_forward`: ``` RuntimeError: The size of tensor a (3072) must match the size of tensor b (2880) at non-singleton dimension 1 File \".../vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py\", line 207, in finalize output.copy_(get_ep_group().combine(...)) ``` 3072 is `round_up(2880, 256)`; 2880 is the model's true `hidden_size`. #### Expected behavior The engine starts and serves, as it does with EP disabled. #### Root cause The modular fused-MoE kernel allocates the finalize destination at the **padded** hidden dim, but this backend produces an **unpadded** expert output: 1. `moe_runner._maybe_pad_hidden_states` pads hidden states 2880 → 3072, because `GptOssMxfp4MoEMethod.skip_forward_padding` is `True` only for `FLASHINFER_TRTLLM_MXFP4_MXFP8` — so `..._MXFP4_BF16` gets padding. 2. `FusedMoEKernelModularImpl.apply` allocates `output = torch.empty_like(hidden_states)` → width **3072**. 3. `GptOssMxfp4MoEMethod.has_unpadded_output` is `True` for `..._MXFP4_BF16`, so `TrtLlmMxfp4ExpertsModular.workspace_shapes` returns `(M, hidden_dim_unpadded)` → `fused_out` is **2880** wide. `AgRsAll2AllManager.combine` reduce-scatters on dim 0 only, so the trailing dim stays 2880. 4. `finalize` is therefore asked to copy 2880 into 3072 → the `RuntimeError`. `FLASHINFER_TRTLLM_MXFP4_BF16` appears to be the only backend that sets padding **on** and unpadded output **on** at the same time, which is why nothing else trips this. `..._MXFP4_MXFP8` sets both `skip_forward_padding` and `has_unpadded_output`, so it never mismatches. The monolithic prepare/finalize sibling is immune because it *returns* its own tensor rather than copying into a caller buffer. Notably, `has_unpadded_output` / `hidden_dim_unpadded` do not appear anywhere in `modular_kernel.py` or under `fused_moe/prepare_finalize/`: the runner learned the padded/unpadded distinction, but the modular kernel's output allocation did not. There is even an existing `# TODO(bnell): move to MK/expert_class?` directly above the `has_unpadded_output` property in `mxfp4.py`. #### The copy is not the real defect — the allocation is Making only the `finalize` copy width-tolerant fixes the `RuntimeError` and then exposes a second, deeper failure in the compiled path: ``` File \".../torch/_inductor/output_code.py\", line 763, in __call__ return self.current_callable(inputs) File \".../inductor_cache/fi/cfi3kg....py\", line 994, in call assert_size_stride(buf8, (s72, 2880), (2880, 1), 'torch.ops.vllm.moe_forward.default') AssertionError: expected size 8192==8192, stride 3072==2880 at dim=0; expected size 3072==2880, stride 1==1 at dim=1 Error in op: torch.ops.vllm.moe_forward.default This error most often comes from a incorrect fake (aka meta) kernel for a custom op. ``` `_finalize` ends with `return output`, so the 3072-wide buffer **is** the custom op's return value, while `moe_runner._moe_forward_fake` declares the op returns `hidden_dim_unpadded` (2880). `_maybe_reduce_final_output` does strip the padding, but it runs *outside* the op, so the op boundary itself must already be 2880. The buffer must be allocated at the unpadded width, not merely copied into tolerantly. #### Suggested fix Recover the output width from the expert output once it is known, in `FusedMoEKernelModularImpl.apply` (before `self._finalize(...)`): ```python if fused_out.shape[-1] != output.shape[-1]: output = torch.empty( (*output.shape[:-1], fused_out.shape[-1]), dtype=output.dtype, device=output.device, ) ``` This appears safe: `output` reaches the experts only as `output_alias`, and `_fused_experts` ignores that alias unless `output_alias.shape == fused_out.shape` — which is false in exactly the mismatching case, so nothing has been written into the discarded buffer. It also fixes every `prepare_finalize` implementation at once rather than one `finalize()`. I have validated this change by patching it into the library in a nightly container and re-running the failing 8xB200 EP configuration: startup memory profiling completes and the server serves normally, while the non-EP configuration stays green. Happy to open a PR if maintainers prefer this shape of fix. PR #44452 (\"Delegate finalize output-buffer allocation to prepare_finalize\") restructures this exact site and recovers the width from `fused_expert_output.shape[-1]`, so it would likely fix this incidentally — but it is a broad refactor, currently open and needing a rebase, and does not mention MXFP4/padding/DP+EP. A targeted fix may be worth landing independently. #### Why the mxfp8 workaround is not a fix Setting `--quantization-config.moe.activation mxfp8` switches to `FLASHINFER_TRTLLM_MXFP4_MXFP8`, which sets `skip_forward_padding = True` and so never hits the mismatch. That masks the bug and changes activation precision; MXFP4 with BF16 activations under DP+EP remains broken. --- _This issue was drafted with assistance from the `opus` AI model._", "url": "https://github.com/vllm-project/vllm/issues/52051", "createdAt": "2026-08-12T21:33:42Z", "updatedAt": "2026-08-12T21:33:59Z", "timestamp": "2026-08-12T21:33:59Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "quantization" ], "author": "janbernloehr", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52053", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Evaluate STS for online DSpark confidence alignment", "text": "### 🚀 The feature, motivation and pitch The confidence scores from a DSpark head are not always perfectly calibrated. In particular, it seems like the deeper positions have higher variance and some bias in their expectation. The DSpark paper suggests [Sequential Temperature Scaling (STS)](https://arxiv.org/html/2607.05147v1) as an online solution to calibrating these estimates. I did not see much benefit when adding this feature, but we should experiment more thoroughly and assess the potential benefit of improved confidence estimates. A comprehensive study is warranted, which includes: - A detailed assessment of confidence head bias across domains (high entropy, low entropy), prediction depth (first draft token vs n-th draft token), and draft/target model choices (DSV4 Flash 0731, DSV4 Pro, and a community-trained DSpark such as for Kimi K3) - A draft implementation of online-STS - - A study of its effect on calibration bias and variance: can we reliably rescale confidence online? Does it drift over time as dataset changes? Can a sudden change in dataset degrade performance? How should we best tune the recency-bias-coefficient? - - A study of the downstream effect on performance: is additional confidence calibration useful? How much increase in AL / throughput do we get from \"oracle\"-level perfect confidence predictions? These are my preliminary results on DSV4-Flash-0731: \"Image\" \"Image\"", "url": "https://github.com/vllm-project/vllm/issues/52053", "createdAt": "2026-08-12T21:45:26Z", "updatedAt": "2026-08-12T21:45:34Z", "timestamp": "2026-08-12T21:45:34Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "kimi", "k3" ], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52057", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Improve Adaptive DSpark Online Profiling", "text": "### 🚀 The feature, motivation and pitch The initial version does profiling at startup to estimate the verification step time by replaying the recorded CUDA graphs. This is known to have some drift, especially at higher batch sizes. Here are my results from initial study: \"Image\" We want to extend this to fully online profiling. However, there are some challenges that make this tricky: - Online profiling may lead to \"dead spots\": one-time latency spikes make a certain batch size appear unattractive, leading to the adaptive verifier to never select that batch size, which leads to negative feedback and stale data limiting our range of profiled values. Generally there is an \"exploration vs exploitation\" problem here, as well as a requirement for some way to detect and eliminate outliers (such as those caused by JIT compilation). - Variable sequence lengths make it hard to measure a single reliable estimate of execution time for a given number of tokens. In practice we will want some way to account for this: either tracking measured points in 2D (num_tokens, max_seq_len or mean_seq_len), or estimating the overhead added per-token of KV reads We should do a more comprehensive study on the drift across models, batch sizes, and sequence lengths. Then, we should design a fully-online solution that addresses the shortcomings, and evaluate the performance gain. We should also evaluate the \"oracle\" speedup with perfect timing prediction (retrospectively) to assess how much performance can be gained", "url": "https://github.com/vllm-project/vllm/issues/52057", "createdAt": "2026-08-12T21:53:44Z", "updatedAt": "2026-08-12T21:53:44Z", "timestamp": "2026-08-12T21:53:44Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "benchislett", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52065", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeek-V4-Flash-0731 + DSpark fails to start on vLLM 0.27.0 / H100 sm90 (DeepGEMM CUDA_ERROR_ILLEGAL_ADDRESS); works on 0.26.0", "text": "### Your current environment - **GPU**: 8× NVIDIA H100 80GB HBM3 (sm90a), single node - **vLLM**: 0.27.0 (pip install into a clean venv) - **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` (local copy of the HF repo) - **Parallelism**: TP=8, expert-parallel, fp8 KV cache - **Also installed / compared against**: vLLM 0.26.0 in a separate venv on the *same* image and the *same* node Serve command (DSpark arm): ```bash vllm serve /models/DeepSeek-V4-Flash-0731 \\ --served-model-name DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --max-model-len 65536 \\ --gpu-memory-utilization 0.90 \\ --kv-cache-dtype fp8 \\ --max-num-batched-tokens 8192 \\ --max-num-seqs 128 \\ --seed 0 \\ --host 0.0.0.0 --port 8080 \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"greedy\"}' ``` ### 🐛 Describe the bug On vLLM **0.27.0**, `DeepSeek-V4-Flash-0731` with **DSpark speculative decoding** never reaches `Application startup complete` on sm90. The DSpark draft model loads successfully, and then roughly 8 minutes later (during engine init / graph capture) **all 8 workers** die with a DeepGEMM illegal memory access: ``` INFO [dspark.py:491] DSpark draft model loaded: 96 params ... ERROR [multiproc_executor.py:1018] WorkerProc hit an exception. RuntimeError: CUDA driver error (/workspace/.deps/deepgemm-src/csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:154): 700 (CUDA_ERROR_ILLEGAL_ADDRESS, an illegal memory access was encountered) ``` **The identical configuration works on 0.26.0.** Same node, same 8 GPUs, same container, same model files, same flags — only the venv (and therefore the vLLM version) differs. On 0.26.0 the server reaches `Application startup complete` and serves `/v1/chat/completions` correctly. Reproduced **3/3** on 0.27.0 across independent runs, with prefix caching both enabled and disabled (it makes no difference to this failure): | vLLM | DSpark | prefix caching | result | |---|---|---|---| | 0.27.0 | on | off | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.27.0 | on | on | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.27.0 | on | off | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.26.0 | on | off | serves OK | ### Possibly a broader 0.27.0 / sm90 problem, not only DSpark Worth flagging for triage: on 0.27.0 this model also fails to start **with DSpark disabled**, but with a *different* signature — a FlashMLA sparse **decode** tensormap assertion: ``` Assertion `res == CUresult::CUDA_SUCCESS` failed (/workspace/.deps/flashmla-src/csrc/sm90/decode/sparse_fp8/instantiations/../splitkv_mla.cuh:742) Engine core initialization failed ``` That arm also serves fine on 0.26.0. So on sm90, 0.27.0 appears unable to start this model in either configuration; DSpark just fails in a different subsystem than the non-DSpark path. I did not bisect between 0.26.0 and 0.27.0. Setting `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` does **not** help 0.27.0 (same `splitkv_mla.cuh:742` assertion). On 0.26.0 that env var is a valid workaround for an unrelated flashinfer JIT issue and the server still starts. ### Related issues - #51822 — same model + DSpark on 0.27.0, but H200 / TP=4 and a different failure: `device-side assert triggered` surfacing in `take_draft_token_ids` → `copy_event.synchronize()`. Possibly the same root cause manifesting asynchronously; filing separately because the signature, hardware and TP differ. - #50660 — same model on **0.26.0**: FlashMLA sparse **prefill** tensormap assertion (`sm90/prefill/sparse/.../phase1.cuh:614`) that I can reproduce 3/3 when DSpark is combined with prefix caching. Different version and different kernel, but the same `Assertion res == CUresult::CUDA_SUCCESS` family — these tensormap asserts guard `cuTensorMapEncodeTiled`, which returns `CUDA_ERROR_INVALID_VALUE` on a zero/degenerate tensor dimension. ### Before submitting a new issue... - [x] I searched existing issues (#51822, #50660, #50576, #51041) and believe this signature is not yet reported.", "url": "https://github.com/vllm-project/vllm/issues/52065", "createdAt": "2026-08-12T22:56:56Z", "updatedAt": "2026-08-13T01:26:31Z", "timestamp": "2026-08-13T01:26:31Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [], "author": "DiegoCao", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52069", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MTP speculative decoding cannot start under pipeline parallelism — SupportsPP demanded of the draft model", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Also reproduced on `dev678` and `dev686`. Code references below are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug Speculative decoding with an MTP head cannot start at all under pipeline parallelism. It fails during `create_engine_config`, before any weight is loaded: ``` vllm serve --pipeline-parallel-size 4 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ``` ``` NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface. ``` Traceback path: ``` vllm/config/speculative.py _verify_args -> self.draft_model_config.verify_with_parallel_config(self.draft_parallel_config) vllm/config/model.py verify_with_parallel_config ``` `speculative.py` L1411-1414 on main: ```python if self.draft_model_config: self.draft_model_config.verify_with_parallel_config( self.draft_parallel_config ) ``` ### Why the check does not apply to a draft model `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so verification then demands `SupportsPP` of the draft. But the drafter is only ever built on one rank — `v1/worker/gpu/model_runner.py`: ```python if self.speculative_config is not None: if self.is_last_pp_rank: self.speculator = init_speculator(self.vllm_config, self.device) ``` A draft model is therefore never split across pipeline stages, so whether it *could* be split is not a question that needs answering about it. ### Scope This is not specific to one model. `Glm4MoeMTP` and `DeepSeekMTP` are declared as `(nn.Module, <...>MixtureOfExperts)` — neither inherits `SupportsPP`, and neither do the other ~20 entries of `MTPModelTypes` (`deepseek_mtp`, `glm4_moe_mtp`, `ernie_mtp`, `qwen3_next_mtp`, `minimax_m3_mtp`, `longcat_flash_mtp`, …). Every MTP head is blocked under `pipeline_parallel_size > 1` by this check. ### Suggested fix Verify the draft as the single-stage model it actually is, and restore the config afterwards so rank and world-size bookkeeping downstream is untouched: ```python if self.draft_model_config: _dpc = self.draft_parallel_config if _dpc.pipeline_parallel_size > 1: _pp = _dpc.pipeline_parallel_size object.__setattr__(_dpc, \"pipeline_parallel_size\", 1) try: self.draft_model_config.verify_with_parallel_config(_dpc) finally: object.__setattr__(_dpc, \"pipeline_parallel_size\", _pp) else: self.draft_model_config.verify_with_parallel_config(_dpc) ``` ### Note on what comes next With this relaxed, MTP under PP gets further but does not yet work: the last rank then dies inside `compile_or_warm_up_model` → `v1/worker/gpu/warmup.py::warmup_kernels` → `_run_decode_step` → `get_pp_group().irecv_tensor_dict(...)`, and after that the output is wrong unless async scheduling is enabled. The scheduling half is filed separately; I am happy to open a third issue for the warmup receive if that is useful rather than noise. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52069", "createdAt": "2026-08-12T23:52:36Z", "updatedAt": "2026-08-13T07:22:14Z", "timestamp": "2026-08-13T07:22:14Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "nickus", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52071", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: speculative decoding under pipeline parallelism produces wrong output with --no-async-scheduling", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Reproduced at `pipeline_parallel_size` 2, 4 and 8, with two different speculative methods and two model families. Code references are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug With `--no-async-scheduling`, speculative decoding under pipeline parallelism produces wrong output — or a CUDA fault — instead of being rejected or working. Turning async scheduling back on fixes it. ### Root cause `next_decode_eligible_step` is assigned in exactly one place in the tree: ``` vllm/v1/core/sched/async_scheduler.py:49 request.next_decode_eligible_step = self.current_step + self.pp_size ``` initialised in one place: ``` vllm/v1/request.py:167 self.next_decode_eligible_step = 0 ``` and read in exactly one place: ``` vllm/v1/core/sched/scheduler.py:509 if self.current_step < request.next_decode_eligible_step: # V2+PP+async: enforce `pp_size` steps between same-req decodes # to match worker-side sampled-tokens broadcast slot ring cadence. ``` The base `Scheduler` — which is what `--no-async-scheduling` selects — reads that field and never sets it. It stays `0`, the guard never fires, and nothing keeps a request's decodes `pp_size` steps apart. The worker-side sampled-token broadcast ring is then read out of phase. ### What that produces, measured A probe at the point where a step's input ids are built (`combine_sampled_and_draft_tokens`), PP=2: ``` last_sampled=[16925] drafts=[16925, 16925, 16925] healthy last_sampled=[0] drafts=[0, 0, 0] ~25% of spec steps ``` Token id 0 is embedded as the anchor. On the model under test token 0 is `!`, and the generated text fills with `!`. The same skew has a second face — the step gets scheduled without the accepted token that its drafts hang off: ``` healthy: hidden=(4,1024) logits_indices=[0, 1, 2, 3] qlen=4 num_logits=4 ndraft=3 failing: hidden=(3,1024) logits_indices=[-1, 0, 1, 2] qlen=3 num_logits=4 ndraft=3 ``` `logits_start = query_end - num_logits = 3 - 4 = -1`, so `hidden_states[input_batch.logits_indices]` either wraps to the last row silently (torch semantics) or trips: ``` Assertion `ind >=0 && ind < ind_dim_size && \"vectorized gather kernel index out of bounds\"` failed ``` The same skew also reaches `_prepare_prefill_inputs_kernel` in `v1/worker/gpu/spec_decode/autoregressive/speculator.py`, where `query_len -= num_rejected` can reach 0 and the unmasked `tl.store(draft_input_ids_ptr + query_start - 1, ...)` writes out of bounds — Triton adds negative indices to the base pointer rather than wrapping. ### Reproduced with - `method: mtp` on GLM-4.5-Air (AWQ), PP=4 - `method: dspark` on Kimi-K3, PP=8 - PP=2 on a 16-layer synthetic target ### Controls Two controls that rule out the obvious alternatives, same tree, same session: ``` speculation OFF, async on vs async off : outputs IDENTICAL Qwen3-0.6B, PP=1 vs PP=2, no speculation : outputs IDENTICAL ``` So the pipeline path is greedy-stable on its own, and the scheduler mode alone changes nothing. The defect appears only when speculation and PP are combined with async scheduling disabled. Also worth noting: with async scheduling ON, plain decode throughput on the same config was ~1.9× better than with it off (94.6 → 48.8 ms/token TPOT, GLM-4.5-Air PP=4, concurrency 1), so the flag is expensive in its own right. ### Suggested fix Any of these beats the current behaviour, in descending order of preference: 1. Set the cadence in the base scheduler too, so the guard works regardless of scheduling mode. 2. Reject `speculative_config` + `pipeline_parallel_size > 1` + `--no-async-scheduling` at config time, with a message naming the cadence. 3. At minimum, document that speculative decoding under PP requires async scheduling. Silently emitting `!` is the worst of the available outcomes, and it is what happens today. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52071", "createdAt": "2026-08-12T23:52:42Z", "updatedAt": "2026-08-13T17:01:20Z", "timestamp": "2026-08-13T17:01:20Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "kimi" ], "author": "nickus", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52083", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Breaking change to DeepSeek-V4-Pro default reasoning effort", "text": "### Your current environment vllm main, bisected to https://github.com/vllm-project/vllm/pull/50580 ### 🐛 Describe the bug This PR (https://github.com/vllm-project/vllm/pull/50580), which changes the DSV4 encoder for DSV4-Flash-0731 support, changes the default behaviour for DSV4-Pro. Previously, only reasoning_effort == \"max\" adds the reasoning blurb. See here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-DSpark/blob/main/encoding/encoding_dsv4.py#L262 However, the new default behaviour is for reasoning effort \"high\" to set that blurb, and introduces a newer blurb for \"max\" thinking mode. This means that DSV4-Pro deployments will now use (what used to be) maximum reasoning effort by default. This caused a massive spike in truncations and a drop in AL in my serving setup. At the very least, we should issue a warning that the default behaviour has been changed. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52083", "createdAt": "2026-08-13T02:51:04Z", "updatedAt": "2026-08-13T13:15:25Z", "timestamp": "2026-08-13T13:15:25Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "benchislett", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52088", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[CI Failure][Anthropic] /v1/messages returns 500 when stop_sequences has more than 4 items", "text": "### Your current environment Observed in vLLM Buildkite CI while running PR #48215. ### Describe the bug The stateless OpenAPI Schemathesis test for `POST /v1/messages` can generate an Anthropic request containing more than four `stop_sequences`. The Anthropic request schema accepts the payload, but converting it to `ChatCompletionRequest` raises a Pydantic validation error because the OpenAI `stop` field permits at most four items. The validation exception is exposed as an HTTP 500 response, so the schema test reports a server error: ```text SUBFAILED entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/messages] schemathesis.core.failures.ServerError: Server error [500] Internal Server Error: 2 validation errors for ChatCompletionRequest stop.str Input should be a valid string [type=string_type, input_type=list] stop.list[str] List should have at most 4 items after validation, not 6 ``` The generated request contained six values: ```json { \"stop_sequences\": [\"sequence-1\", \"sequence-2\", \"\", \"sequence-4\", \"sequence-5\", \"sequence-6\"] } ``` This appears to be a mismatch between the `/v1/messages` OpenAPI request schema and the constraints applied during Anthropic-to-OpenAI request conversion. ### Expected behavior The OpenAPI schema should constrain `stop_sequences` to the supported maximum, or the conversion validation error should be returned as a client error (4xx). Invalid user input should not result in HTTP 500. ### Additional context - Failing test: `entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/messages]` - CI build: https://buildkite.com/vllm/ci/builds/83486 - Encountered while testing https://github.com/vllm-project/vllm/pull/48215; that PR changes Ultravox/Whisper model code and appears unrelated to the Anthropic entrypoint. ### Before submitting a new issue... - [x] I searched existing issues and PRs for `test_openapi_stateless`, `/v1/messages`, `stop_sequences`, and the Pydantic maximum-length error. I found related entrypoint/schema issues, but no issue covering this specific failure.", "url": "https://github.com/vllm-project/vllm/issues/52088", "createdAt": "2026-08-13T03:18:56Z", "updatedAt": "2026-08-13T06:00:13Z", "timestamp": "2026-08-13T06:00:13Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "arthurgao2003", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52089", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Continuous Host Memory Growth / Possible Memory Leak with V2 Runner on Qwen3-14B and Qwen3-Rerank-4B", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by Anaconda, Inc. | (main, Jul 9 2026, 14:38:16) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-6.8.0-31-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.2.51 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4090 GPU 1: NVIDIA GeForce RTX 4090 Nvidia driver version : 595.80 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD EPYC 7543 32-Core Processor CPU family: 25 Model: 1 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 1 BogoMIPS: 5589.49 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core intel_ppin ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd virt_ssbd arat umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache: 8 MiB (16 instances) L3 cache: 64 MiB (4 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.14 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.26.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.11.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchcodec==0.15.0 [pip3] torchvision==0.26.0 [pip3] transformers==5.14.1 [pip3] triton==3.6.0 [conda] flashinfer-python 0.6.14 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.0.3 pypi_0 pypi [conda] nvidia-cuda-cccl 13.3.3.4.1 pypi_0 pypi [conda] nvidia-cuda-crt 13.3.73 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvcc 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvdisasm 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.26.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-core 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu12 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu13 4.6.0 pypi_0 pypi [conda] nvidia-ml-py 13.610.43 pypi_0 pypi [conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] nvidia-nvvm 13.3.73 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] tokenspeed-triton 3.8.10.post20260721 pypi_0 pypi [conda] torch 2.11.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchcodec 0.15.0 pypi_0 pypi [conda] torchvision 0.26.0 pypi_0 pypi [conda] transformers 5.14.1 pypi_0 pypi [conda] triton 3.6.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-31 0 N/A GPU1 PHB X 0-31 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/cuda/lib64: CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_ubuntu ```
## Description I observed continuous and approximately linear **host memory growth** when running long-running data processing workloads with the **V2 Runner** and the following models: * Qwen3-14B-FP8 * Qwen3-Rerank-4B The memory usage keeps increasing over time and eventually consumes almost all available system memory. I reproduced the issue with both: * vLLM `v0.25.1` * vLLM `v0.27.0` In contrast, the same Qwen3-14B and Qwen3-Rerank-4B workloads remain stable when running with the **V1 Engine** on `vLLM==0.11.2`. I also tested `vLLM==0.25.1` with TranslateGemma. Its host memory usage remains stable during long-running inference, which suggests that this may be related to the V2 Runner and/or its interaction with these Qwen3 models rather than a general memory behavior of vLLM. ## Observed Behavior With V2 Runner: ```text Qwen3-14B-FP8 ↓ long-running data cleaning / inference workload ↓ host memory continuously increases ↓ memory usage grows approximately linearly ↓ eventually reaches almost 100% ↓ process/system becomes unstable or runs out of memory ``` The memory growth is not a short-term allocation spike. It continues accumulating throughout the long-running workload. The attached screenshots show the memory usage trend. In the long-term monitoring view, host memory usage increases from roughly 10% to nearly 100% over the observation period, with only a small drop/recovery when the process is restarted. ## Comparison ### V2 Runner | Model | vLLM Version | Engine | Prefix Caching | Long-running Memory | | --------------- | --------------: | ------ | -------------- | ------------------------ | | Qwen3-14B-FP8 | 0.25.1 | V2 | Enabled | ❌ Continuously increases | | Qwen3-14B-FP8 | 0.27.0 | V2 | Enabled | ❌ Continuously increases | | Qwen3-Rerank-4B | 0.25.1 / 0.27.0 | V2 | Enabled | ❌ Continuously increases | ### V1 Engine | Model | vLLM Version | Engine | Prefix Caching | Long-running Memory | | --------------- | -----------: | ------ | -------------- | ------------------- | | Qwen3-14B-FP8 | 0.11.2 | V1 | Enabled | ✅ Stable | | Qwen3-Rerank-4B | 0.11.2 | V1 | Enabled | ✅ Stable | ### Another V2 Model | Model | vLLM Version | Engine | Long-running Memory | | -------------- | -----------: | ------ | ------------------- | | TranslateGemma | 0.25.1 | V2 | Stable | This makes the regression particularly interesting because: 1. The issue occurs with multiple Qwen3 models. 2. It occurs across vLLM `0.25.1` and `0.27.0`. 3. The same Qwen3 models are stable under the V1 Engine. 4. Another model (TranslateGemma) is stable under V2. 5. Prefix caching is enabled in the tested deployments. ## Reproduction The following is the deployment command used for Qwen3-14B-FP8: ```bash gpu_id=${1:-0} export PYTORCH_ALLOC_CONF=\"expandable_segments:True\" CUDA_VISIBLE_DEVICES=${gpu_id} vllm serve /data/checkpoint/Qwen3-14B-FP8 \\ --gpu-memory-utilization 0.92 \\ --tensor-parallel-size 1 \\ --served-model-name Qwen3-14B-FP8 \\ --dtype auto \\ --max_model_len 2048 \\ --max-num-seqs 128 \\ --host 0.0.0.0 \\ --port $((8002 + gpu_id)) \\ --enable-prefix-caching \\ --enable-chunked-prefill ``` The workload continuously sends inference requests for a long-running data-cleaning pipeline. ## Environment * Model: `Qwen3-14B-FP8` * Additional affected model: `Qwen3-Rerank-4B` * vLLM: `0.25.1`, `0.27.0` * Stable comparison: `vLLM==0.11.2` (V1 Engine) * GPU: NVIDIA RTX 4090 * Engine: V2 for affected versions * `--enable-prefix-caching` * `--enable-chunked-prefill` * `--max_model_len 2048` * `--max-num-seqs 128` * `--gpu-memory-utilization 0.92` * `PYTORCH_ALLOC_CONF=expandable_segments:True` ## Expected Behavior Host memory usage should eventually reach a relatively stable level after the engine and model have warmed up. For a long-running inference service, memory usage should not continuously grow with the number of processed requests if the workload characteristics remain approximately stable. This is what I observe with: * Qwen3 models under V1 Engine (`v0.11.2`) * TranslateGemma under V2 Runner (`v0.25.1`) ## Actual Behavior With V2 Runner and Qwen3-14B / Qwen3-Rerank-4B, host memory continuously increases during the workload. The growth appears approximately linear with runtime/request volume and eventually exhausts available system memory. The behavior persists across vLLM `0.25.1` and `0.27.0`. ## Additional Observation The fact that the same Qwen3 models are stable with the V1 Engine is a strong indication that the issue may be related to a V2-specific component or code path. Possible areas that may be worth investigating include: * V2 request lifecycle / request state cleanup * prefix caching metadata or cache management * scheduler state * sequence/request objects retained after completion * async engine / output processing * KV cache bookkeeping * chunked prefill related state * Python-side objects associated with completed requests I have not yet determined which component is responsible for the retained memory. ## Evidence I have attached four memory-monitoring screenshots showing the observed behavior and comparison runs. The most obvious long-term behavior is the approximately linear growth of host memory usage under the affected V2 configuration, eventually reaching almost 100%. ## Request Could you please help investigate whether this is a known memory leak or memory retention issue in the V2 Runner? In particular, it would be useful to know whether there were any changes between V1 and V2 around: * request lifecycle management * prefix caching * scheduler state * sequence/request object cleanup * async output processing that could cause completed requests or related metadata to remain referenced over time. \"Image\"", "url": "https://github.com/vllm-project/vllm/issues/52089", "createdAt": "2026-08-13T03:33:00Z", "updatedAt": "2026-08-13T03:42:47Z", "timestamp": "2026-08-13T03:42:47Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "77wheat", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52095", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "RFC: Kimi K3 MoonEP integration roadmap", "text": "## Motivation Track the **MoonEP integration** item under Large-scale Serving in #50001 as a small set of independently reviewable PRs. MoonEP (https://github.com/MoonshotAI/MoonEP) is the balanced EP communication library used by Kimi K3: dynamic redundant experts keep every EP rank at exactly `S × K` tokens regardless of router skew, with static shapes and zero-copy dispatch/combine. ## Proposed change - [ ] **BF16 PoC: backend registration + dispatch/combine path + validation** — #52101 `\"moonep\"` `All2AllBackend`, `MoonEPAll2AllManager`, `MoonEPPrepareAndFinalize` (pad → dispatch → `prefetch_weight` → combine, plan stashed between prepare/finalize like DeepEP-HT's handle), replicated `[E+B, …]` BF16 weight-layout helper, reference expert runner over `cu_seqlens`, torchrun validation script. Validated on 4× GB300 (EP=4, NVLink): passes at default config, heavy router skew, and larger shapes (S=512, H=2048, top-8 of 64 experts); rel_err within BF16 tolerance. - [ ] **Grouped-GEMM expert implementation over `cu_seqlens`** — PR TBD Modular-kernel experts class consuming MoonEP's expert-grouped `[NvS, H]` activation format. Open question for maintainers: `torch._grouped_mm` vs Triton vs DeepGEMM contiguous-layout kernels. - [ ] **FusedMoE layer + engine wiring (BF16, eager)** — PR TBD Attach the weight layout after weight loading; config validation and docs. Exit criterion: a small real MoE (e.g. Qwen3-30B-A3B) serves end-to-end with `--all2all-backend moonep` with sane evals. - [ ] **Tests + benchmarks** — PR TBD Multi-GPU correctness test under `tests/kernels/moe/`; dispatch/combine benchmark vs `deepep_*` backends under `benchmarks/kernels/`, sweeping router imbalance. - [ ] **Sharded symmetric-memory expert weight ownership** — PR TBD Replace per-rank weight replication with MoonEP's production layout: one contiguous VMM tensor per projection where rows `[0, E)` alias each home rank's parameter memory. Memory-viability requirement for Kimi-K3 scale. Design question: allocate expert weights inside the MoonEP buffer before weight loading vs. post-load remap. - [ ] **CUDA graph support** — PR TBD MoonEP's fully static `NvS` shapes should make capture tractable. - [ ] **Quantized dispatch + weights (fp8 first)** — PR TBD Kimi-K3-scale serving will not be BF16; needs MoonEP-side support scoping as well. - [ ] **EPLB interaction** — PR TBD MoonEP's dynamic redundant experts overlap with the EPLB item in the same #50001 section; define compose-or-exclude semantics. - [ ] **Async / DBO support** — PR TBD `async_finish=True` + MoonEP's comm stream for prepare_async/finalize_async and dual-batch overlap. ## Scope This issue is a tracking RFC only. Each item should remain independently reviewable. The first four items establish a correctness-first BF16/eager path; the sharded weight-ownership and EPLB items carry the design decisions where maintainer input is requested before implementation.", "url": "https://github.com/vllm-project/vllm/issues/52095", "createdAt": "2026-08-13T04:22:08Z", "updatedAt": "2026-08-13T04:59:56Z", "timestamp": "2026-08-13T04:59:56Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization", "kimi", "k3" ], "author": "kaijunli-infr", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52104", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm_c fused_add_rms_norm rounds the residual sum before variance", "text": "### Summary The CUDA `vllm_c` implementation of `fused_add_rms_norm` and the native vLLM IR implementation use different rounding orders for BF16/FP16 inputs. At upstream commit [`903d2efe`](https://github.com/vllm-project/vllm/commit/903d2efe7eb6d045d03c0a8192b1ac7bec50d168): - The [native IR implementation](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/vllm/ir/ops/layernorm.py#L37-L55) computes `x.float() + residual.float()` and uses that unrounded FP32 sum to calculate the variance. - The [vectorized `vllm_c` CUDA kernel](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/csrc/libtorch_stable/layernorm_kernels.cu#L133-L169) stores the sum in an `_f16Vec` before `sum_squares()`. This rounds the residual sum to BF16/FP16 before calculating the variance. - The [generic CUDA kernel](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/csrc/libtorch_stable/layernorm_kernels.cu#L191-L215) has the same behavior because `z` is `scalar_t`. This matters because CUDA selects `[\"native\"]` with Inductor, but selects `[\"vllm_c\", \"native\"]` when compilation is disabled ([dispatch code](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/vllm/platforms/cuda.py#L686-L700)). Therefore, changing only `CompilationMode` can change numerical results and, after accumulation through many layers, greedy output tokens. I observed this with GLM-4.7-Flash in BF16: the `CompilationMode.NONE` path diverged from the compiled/native path. Making the CUDA kernel use the FP32 residual sum for variance removed the token divergence in that test. ### Minimal reproducer Run this against unmodified vLLM `main` on a CUDA GPU: ```python import torch from vllm import _custom_ops as ops torch.manual_seed(0) device = \"cuda\" dtype = torch.bfloat16 tokens = 4 hidden_size = 2048 epsilon = 1e-6 x = torch.randn(tokens, hidden_size, device=device, dtype=dtype) residual = torch.randn_like(x) weight = torch.randn(hidden_size, device=device, dtype=dtype) # Native IR semantics: keep input + residual in FP32 through variance. summed_fp32 = x.float() + residual.float() variance_fp32 = summed_fp32.square().mean(dim=-1, keepdim=True) native = summed_fp32 * torch.rsqrt(variance_fp32 + epsilon) native = (native.to(weight.dtype) * weight).to(dtype) native_residual = summed_fp32.to(dtype) # vllm_c custom op mutates both tensors in place. actual = x.clone() actual_residual = residual.clone() ops.fused_add_rms_norm(actual, actual_residual, weight, epsilon) torch.cuda.synchronize() diff = (actual.float() - native.float()).abs() print(\"output mismatches:\", torch.count_nonzero(actual != native).item()) print(\"max output diff:\", diff.max().item()) print(\"mean output diff:\", diff.mean().item()) print( \"residual mismatches:\", torch.count_nonzero(actual_residual != native_residual).item(), ) ``` The exact mismatch count can vary with the GPU/build, but the two implementations are structurally different: `vllm_c` calculates variance from the rounded low-precision sum, while native IR calculates it from the FP32 sum. For scale, simulating the current `vllm_c` arithmetic on an A100 with PyTorch 2.11.0/CUDA 13.0 and the seed/shapes above produced: ```text tokens=1: 368 / 2048 output elements differ; max abs diff 0.03125 tokens=4: 1465 / 8192 output elements differ; max abs diff 0.0625 tokens=16: 5791 / 32768 output elements differ; max abs diff 0.0625 ``` ### Expected behavior The `vllm_c` and native implementations should use the same defined arithmetic/rounding semantics so that selecting `CompilationMode.NONE` does not introduce a different residual-normalization calculation. ### Proposed fix For both CUDA kernel specializations: 1. Compute each residual sum as FP32: `float sum = float(input) + float(residual)`. 2. Accumulate variance from that FP32 `sum`. 3. Publish `residual = scalar_t(sum)`. 4. Normalize using the FP32 `sum`, while preserving the intended output/weight multiplication rounding policy. This requires recomputing the sum during the second pass or temporarily retaining it; reading the already-rounded `residual` in the second pass preserves the current mismatch. ### Related issues This appears distinct from: - #49616 / #49639, which concern rounding the normalized value before multiplying by the RMSNorm weight. - #42379, which concerns the dtype used for weight multiplication. - #48271, which concerns batch-size-dependent reduction behavior. - #43390, which concerns integer overflow. This report is specifically about rounding `input + residual` before variance. ### Environment used during investigation - GPU: NVIDIA A100-SXM4-80GB - PyTorch: 2.11.0+cu130 - CUDA: 13.0 - Driver: 580.105.08 - Dtype: BF16 - Model-level observation: GLM-4.7-Flash - Relevant configuration: `CompilationMode.NONE` ### Before submitting a new issue - [x] I searched existing issues and PRs for this specific residual-sum rounding mismatch. This issue was prepared with AI assistance; the source comparison and reported measurements were manually validated on the hardware above.", "url": "https://github.com/vllm-project/vllm/issues/52104", "createdAt": "2026-08-13T05:44:10Z", "updatedAt": "2026-08-13T09:02:42Z", "timestamp": "2026-08-13T09:02:42Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "hjzccc", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52105", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: sparse_mla_attention forward_mha falls through to unimplemented base class under low max-num-batched-tokens, surfaces as scheduler KeyError", "text": "## Environment - vLLM: v0.27.0, source-built with #51538's two commits (`47f6574`, `db39e67`) applied - Model: GLM-5.2 (compressed-tensors WNA16 Int4-Int8), `GlmMoeDsaForCausalLM` - Hardware: 4x DGX Spark GB10 (sm_121, capability 12), TP=4, `mp` executor - Attention backend: `FLASHINFER_MLA_SPARSE_SM120` (the only one that supports this hardware capability plus a sparse indexer, see below) - Speculative decoding: MTP, `num_speculative_tokens=2` - Relevant flags: `--max-model-len 64000 --kv-cache-memory-bytes 4294967296 --gpu-memory-utilization 0.70 --max-num-seqs 4 --max-num-batched-tokens 2048 --compilation-config '{\"cudagraph_mode\":\"NONE\"}'` Found while narrowing a separate concurrent-load hang (vllm-project/vllm#51921) under a reduced-memory diagnostic configuration. Not the same bug as #51921: confirmed by reverting only `--max-num-batched-tokens` from 2048 to production's 8192 with everything else identical, this crash disappears and the original silent hang (#51921) returns instead. ## Bug description Sending a long prompt (20000 tokens) under `--max-num-batched-tokens 2048` crashes the engine reproducibly (2/2 runs, exact same crash site both times): ``` File \"vllm/v1/core/sched/scheduler.py\", line 1761, in update_from_output req_index = model_runner_output.req_id_to_index[req_id] KeyError: 'chatcmpl-' ``` This is downstream of the real failure. All four TP ranks (the scheduler's own rank 0 worker included) hit an identical `NotImplementedError` at the same timestamp: ``` File \"vllm/v1/attention/backend.py\", line 1052, in forward_mha raise NotImplementedError ``` ## Root cause `vllm/v1/attention/backend.py`'s `MLAAttentionImpl.forward_mha` is a non-abstract base-class method that defaults to `raise NotImplementedError`, meant to be overridden by backends that support MHA-style prefill. `sparse_mla_attention.py`'s override contains: ```python if force_dense or (prefill_max_seq_len <= topk_tokens and not force_masked): return super().forward_mha(...) # the unimplemented base-class path ``` Under chunked prefill with `max-num-batched-tokens=2048`, a 20000-token prompt gets split into chunks no larger than 2048 tokens. If `topk_tokens` (the sparse indexer's per-request top-k budget) is >= 2048, every chunk satisfies `prefill_max_seq_len <= topk_tokens`, permanently routing into the unimplemented path instead of the working masked-MHA implementation a few lines below it. Since all four ranks are running the identical batch shape, they hit this simultaneously. Workers die before producing output; the scheduler's `req_id_to_index` `KeyError` on rank 0 is a consequence of that, not an independent bug, `req_id_to_index` has no guard anywhere against a request going missing because upstream compute failed. ## Reproduction 1. TP=4, GLM-5.2 (or any model on the `sparse_mla_attention.py` code path with a small enough `topk_tokens`), `--max-num-batched-tokens` set low enough that a long prompt's chunk size can be <= `topk_tokens`. 2. Send a prompt whose length exceeds `max-num-batched-tokens`, so chunked prefill kicks in. 3. Every rank raises `NotImplementedError` in `forward_mha` at the same moment; the API returns `500` for all in-flight requests. ## Suggested fix Either implement `forward_mha` in the base class (or raise a clear, catchable, user-facing error instead of a bare `NotImplementedError` that surfaces as an unrelated scheduler `KeyError`), or have `sparse_mla_attention.py`'s branch condition account for cases where the base class path is genuinely unimplemented rather than routing into it unconditionally. Happy to provide full four-rank tracebacks and the exact launch config if useful.", "url": "https://github.com/vllm-project/vllm/issues/52105", "createdAt": "2026-08-13T05:52:25Z", "updatedAt": "2026-08-13T05:52:25Z", "timestamp": "2026-08-13T05:52:25Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52107", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain", "text": "# NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain ## Summary Native MTP speculative decoding for Nemotron-3.5-Lightning-30B-A3B produces **0% draft acceptance** on Intel Arc Pro B70 (Xe2/Battlemage) via vLLM XPU nightly. After extensive debugging (9+ GPU cycles, 3 identified bugs patched, 7 instrumentation attempts), we found the V1 speculator's execution path **completely bypasses** the standard sampling/verification chain (`_process_engine_step`, `post_step`, `sample`, `RejectionSampler.__call__`). Drafts are proposed (spec counters increment) but never verified. ## Environment - **Hardware**: Intel Arc Pro B70 (32 GB GDDR6, Xe2/Battlemage, 256 XMX, `xe` driver) - **Image**: `vllm/vllm-openai-xpu@sha256:1da0a95485455f08588c11080b9718992fd7d434c6a965d74654903a9d999c57` - **vLLM**: `0.26.1rc1.dev668+g3ee2df303` - **vllm-xpu-kernels**: `0.1.12.3` (with grouped-GEMM `at::zeros` fix — see below) - **Model**: Nemotron-3.5-Lightning-30B-A3B, local symmetric GPTQ INT4 G64 conversion (base model from `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`; published at `SergiiioB/Nemotron-3.5-Lightning-30B-A3B-GPTQ-INT4-G64-sym`) - **Config**: `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`, `--enforce-eager`, `--no-enable-prefix-caching` ## Three identified bugs (all patched, still 0% acceptance) ### 1. Double-norming of hidden states `NemotronHModel.forward` applies `norm_f` (final RMSNorm) to hidden states BEFORE passing to the MTP draft. The draft's `NemotronHMTPAttentionDecoderLayer` applies its OWN `hnorm` → double-normalized → garbage draft attention input. **Fix**: `patch_nemotron_prenorm_mtp.py` — saves pre-norm state (`hidden_states + residual`) and exposes via `get_mtp_target_hidden_states()`. ### 2. Concatenation order mismatch The MTP fusion layer concatenates `[inputs_embeds, hidden_states]` but MTP/EAGLE architectures may train with reversed order. **Fix**: `patch_concat_order.py` — swaps to `[hidden_states, inputs_embeds]`. ### 3. Non-gated MoE w13 initialization NemotronH uses a non-gated MoE (no `gate_proj`). The fused w13 mapping in `routed_experts.py` warns `\"Unexpected gate/up projection names: up_proj, .\"` and skips fused mapping. The XPU WNA16 expert backend handles this correctly (its own loading path), but the torch FusedMoE path (used by unquantized drafts) leaves expert weights uninitialized. ## The critical finding: V1 execution path bypass After patching all three bugs and testing across 5 GPU-cycle combinations (INT4/BF16 × prenorm/concat/control — ALL 0% acceptance), we instrumented the V1 pipeline with file-based logging (baked into a custom Docker image, confirmed via grep): | Method | File | Reached? | |---|---|---| | `EngineCore.__init__` | `engine/core.py` | ✅ (file write confirmed) | | `_process_engine_step` | `engine/core.py:1442` | ❌ NEVER executed | | `post_step` | `engine/core.py:615` | ❌ NEVER executed | | `step` | `engine/core.py:583` | ❌ NEVER executed | | `model_runner.sample` | `model_runner.py:1227` | ❌ NEVER executed | | `RejectionSampler.__call__` | `rejection_sampler.py:233` | ❌ NEVER executed | | `rejection_sample` | `rejection_sampler_utils.py:922` | ❌ NEVER executed | | `DraftTokensHandler.get_draft_tokens` | `spec_decode/utils.py` | ❌ NEVER executed | **All patches confirmed present in source (grep verified) and the EngineCore subprocess loads from `/workspace/vllm/` (confirmed via `__init__` file write).** Yet NONE of the inference-time methods execute during MTP spec decode. The spec counters DO increment (`spec_decode_num_drafts_total: 31`, `spec_decode_num_draft_tokens_total: 63`, `spec_decode_num_accepted_tokens_total: 0`), confirming the speculator IS running — but through a completely non-standard execution path. ## Bonus: grouped-GEMM atomic race fix (already applied) During this investigation, we found and fixed a **one-word bug** in `vllm-xpu-kernels` that caused non-deterministic output in XPU graph mode: **File**: `csrc/xpu/grouped_gemm/xe_2/grouped_gemm_xe2_interface.hpp` ```diff - at::Tensor atomic_buffer = at::empty({static_cast(1)}, ...); + at::Tensor atomic_buffer = at::zeros({static_cast(1)}, ...); ``` The persistent block-scheduler's atomic counter was allocated with `at::empty` (uninitialized). SYCL doesn't guarantee `group_id == 0` launches first, so other groups `atomicAdd` into garbage → inflated counters → out-of-bounds tiles → non-deterministic MoE output. In XPU graph mode the dirty buffer is captured and guaranteed to poison every replay. **Impact**: This fix enabled deterministic XPU graph capture for NemotronH, raising decode from 21.8 → 92.65 t/s (4.25×) with byte-identical temperature-0 replay. The fix has zero performance cost. ## Request 1. **Investigate the V1 MTP speculator execution path** — why does the standard `_process_engine_step` → `step_fn` → `post_step` chain not execute during MTP spec decode? What alternative path does the speculator use? 2. **Verify the three identified bugs** — are they real issues on other platforms (CUDA)? 3. **Consider the `at::zeros` grouped-GEMM fix** for upstream merge (it affects all XPU users, not just NemotronH) ## Repro ```bash docker pull vllm/vllm-openai-xpu@sha256:1da0a95485455f08588c11080b9718992fd7d434c6a965d74654903a9d999c57 # Apply patches: patch_nemotron_prenorm_mtp.py, patch_xpu_grouped_topk_native_v2.py # (available at https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook/tree/main/patches) vllm serve /model --dtype bfloat16 --quantization gptq \\ --max-model-len 16384 --gpu-memory-utilization 0.90 --enforce-eager \\ --no-enable-prefix-caching \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' # Send any completion request → check /metrics: # spec_decode_num_accepted_tokens_total = 0 (should be >0) ```", "url": "https://github.com/vllm-project/vllm/issues/52107", "createdAt": "2026-08-13T06:29:39Z", "updatedAt": "2026-08-13T06:50:38Z", "timestamp": "2026-08-13T06:50:38Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "intel-gpu", "quantization" ], "author": "SergiioB", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52109", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][ROCm/gfx942]: DeepSeek-V4-Flash silent retrieval corruption for prompts ≥ ~4-5k tokens (AITER sparse indexer)", "text": "## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Image: `vllm/vllm-openai-rocm:nightly` (2026-08-12, v0.26.1rc1.dev668+g3ee2df303) - plus local backports of #51821 (merged 08-13) and #52058 / #51252 (open) — reproduces identically without the two open-PR backports - Model: `deepseek-ai/DeepSeek-V4-Flash-0731`, TP=8, `--kv-cache-dtype fp8_ds_mla`, `--max-num-batched-tokens 16384`, attention backend `DEEPSEEK_SPARSE_SWA` ## 🐛 Describe the bug Needle-in-haystack retrieval works perfectly for short prompts and collapses to 0/3 somewhere between **3,611 tokens (3/3 retrieved)** and **5,294 tokens (0/3)**. Above the threshold the model claims the needles do not exist and output quality degrades broadly (often rambling to `max_tokens`). No crash, no error — **healthy server, silently wrong results**. GSM8K-style short-prompt correctness is unaffected (facts probes pass). Measured (3 needles at 10%/50%/90% depth, `temperature=0`): | prompt tokens | retrieved | |---|---| | 1,913 | 3/3 | | 3,611 | 3/3 | | 5,294 | 0/3 | | 7,003 / 10,411 / 12,061 / 27k / 84k / 253k / 506k | 0/3 (10,411 once gave 2/3 with a corrupted digit string) | ## Ruled out experimentally - `max_model_len` (fails identically at 131,072 and 1,048,576) - chunked-prefill boundary (`max_num_batched_tokens=16384`; 12k prompts are single-chunk and still fail) - #51252 (sparse-indexer prefill buffer budget) — backported, no change on this path - AITER `fp8_mqa_logits` split-KV heuristic (`seq_len_kv >= 4096` → splits>1): forced `num_splits=1`, no change Looks like the gfx942 sibling of #40018 (gfx950 `ROCM_AITER_MLA_SPARSE` garbage for prompt_len > ~20K), with a lower threshold (~4-5k). ## Repro Self-contained script (generates haystack, inserts 3 needles, checks retrieval): expand below. Run against an OpenAI-compatible endpoint: `python3 needle_test.py 6000` → RETRIEVED 0/3; `... 4000` → 3/3.
needle_test.py ```python #!/usr/bin/env python3 \"\"\"長上下文 needle 測試:產生約 targetTok 的 haystack,插 3 根 needle(10%/50%/90% 深度),要求模型取回。\"\"\" import json, time, random, urllib.request, sys KEY = sys.argv[1] TARGET_TOK = int(sys.argv[2]) if len(sys.argv) > 2 else 600_000 MODEL = sys.argv[3] if len(sys.argv) > 3 else \"deepseek-v4-flash\" random.seed(42) subjects = [\"The quarterly report\", \"A municipal committee\", \"The research station\", \"An old lighthouse\", \"The logistics team\", \"A regional archive\", \"The observatory\", \"A harbor authority\", \"The botanical survey\", \"An engineering guild\"] verbs = [\"documented\", \"reviewed\", \"misplaced\", \"catalogued\", \"audited\", \"transferred\", \"digitized\", \"inspected\", \"renovated\", \"commissioned\"] objs = [\"seventeen ledgers\", \"a set of brass instruments\", \"the annual rainfall data\", \"three shipping manifests\", \"the訪客紀錄\", \"a collection of maps\", \"the maintenance schedule\", \"several personnel files\", \"the calibration records\", \"an inventory of spare parts\"] years = list(range(1951, 2026)) NEEDLES = { \"ZEPHYR-CODE\": \"738291\", \"MARLIN-KEY\": \"460517\", \"ONYX-TOKEN\": \"092384\", } def sentence(): return f\"{random.choice(subjects)} {random.choice(verbs)} {random.choice(objs)} in {random.choice(years)}. \" # 估 1 token ≈ 4 chars(英文混雜),目標字元數: target_chars = TARGET_TOK * 4 parts, n = [], 0 while n < target_chars: s = sentence() parts.append(s) n += len(s) keys = list(NEEDLES.items()) for frac, (k, v) in zip((0.10, 0.50, 0.90), keys): idx = int(len(parts) * frac) parts.insert(idx, f\"[IMPORTANT] The secret value of {k} is {v}. Remember it. \") haystack = \"\".join(parts) prompt = (haystack + \"\\n\\n---\\nFrom the document above, report the secret values of ZEPHYR-CODE, MARLIN-KEY, and ONYX-TOKEN. \" \"Answer in exactly this format:\\nZEPHYR-CODE=\\nMARLIN-KEY=\\nONYX-TOKEN=\") print(f\"haystack chars: {len(haystack):,}\", flush=True) body = {\"model\": MODEL, \"messages\": [{\"role\": \"user\", \"content\": prompt}], \"max_tokens\": 4000, \"temperature\": 0} req = urllib.request.Request(\"http://127.0.0.1:8000/v1/chat/completions\", json.dumps(body).encode(), {\"Authorization\": f\"Bearer {KEY}\", \"content-type\": \"application/json\"}) t0 = time.time() with urllib.request.urlopen(req, timeout=3600) as r: d = json.load(r) dt = time.time() - t0 u = d[\"usage\"] content = d[\"choices\"][0][\"message\"].get(\"content\") or \"\" print(f\"prompt_tokens={u['prompt_tokens']:,} completion_tokens={u['completion_tokens']} time={dt:.0f}s\", flush=True) print(\"--- model answer ---\") print(content[-500:]) ok = sum(1 for k, v in NEEDLES.items() if v in content) print(f\"RETRIEVED {ok}/3\") ```
", "url": "https://github.com/vllm-project/vllm/issues/52109", "createdAt": "2026-08-13T06:42:20Z", "updatedAt": "2026-08-13T10:42:48Z", "timestamp": "2026-08-13T10:42:48Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm" ], "author": "zzw09773", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52111", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeek-V4-Flash-0731 MTP speculative decoding fails to load: KeyError 'model.layers.43.mtp_block.main_norm.weight'", "text": "## Environment - 8× MI325X (gfx942), `vllm/vllm-openai-rocm:nightly` (2026-08-12, g3ee2df303) - `deepseek-ai/DeepSeek-V4-Flash-0731` (config has `num_nextn_predict_layers: 1`), TP=8 ## 🐛 Describe the bug Enabling self-speculative MTP: ``` --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' ``` Architecture resolves to `DeepSeekV4MTPModel`, then every worker dies during weight loading with: ``` KeyError: 'model.layers.43.mtp_block.main_norm.weight' ``` Looks like the MTP loader's expected parameter naming doesn't match this checkpoint's actual MTP tensor names. Serving without speculative config works fine (with #51821).", "url": "https://github.com/vllm-project/vllm/issues/52111", "createdAt": "2026-08-13T06:42:32Z", "updatedAt": "2026-08-13T10:42:23Z", "timestamp": "2026-08-13T10:42:23Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm" ], "author": "zzw09773", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52113", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Session-Aware KV Cache Hints for Agentic Workloads", "text": "### Motivation. Agentic applications often know more about future KV cache reuse than the inference engine: - a child agent is waiting for a tool and will probably resume soon; - several agents share a long system/tool prefix; - a branch has completed and its KV cache can be released; - an offloaded branch will be resumed and can be prefetched. Today, after a request finishes, these prefixes are ordinary free cached blocks managed by the same replacement policy. vLLM cannot distinguish a temporarily idle but valuable agent context from a prefix that will never be used again. This RFC proposes optional `agent_hint` metadata so an application can communicate session identity, a short retention TTL, and explicit cache management operations. The goal is to improve prefix-cache hit rate and TTFT for multi-turn and multi-agent workloads. The proposal addresses an information gap rather than replacing vLLM's cache policy. An agent runtime knows when a tool call is outstanding, a sub-agent has finished, a conversation has been suspended, or a checkpoint is about to be resumed. vLLM knows which KV blocks are resident and when capacity is needed. Neither side can make the best decision with only its own information. Hints provide a narrow, best-effort contract between those two layers. ### Workload scenarios and expected value The largest opportunity is not generic chat traffic, but workloads with long prefixes and explicit lifecycle transitions: | Application event | Hint/action | Expected system value | | --- | --- | --- | | A long-running agent waits for a tool or user response | Protect the request's cache for a short TTL, or offload it for a longer pause | Avoid an accidental LRU eviction and repeated prefill when the agent resumes | | A leader creates multiple workers from a common context | Associate parent and child sessions; protect the active branches | Preserve normal content-based sharing of the common prefix while retaining each branch only as long as it is useful | | A child agent completes or a failed branch is abandoned | Evict that session's cached context | Reclaim capacity earlier than an engine-only timeout or replacement policy can | | An application switches, resumes, or rewinds a session | Prefetch the target context before inference | Overlap KV movement with application state transition and reduce resume TTFT | | Context is compacted or archived | Evict obsolete cache, or offload context that may be restored | Align physical KV residency with the application's actual context lifecycle | | A session moves between instances in pooled/disaggregated serving | Retain or prefetch the corresponding remote KV entries | Improve cross-instance reuse and avoid recomputing a long prefix after routing changes | These actions target three concrete outcomes: - **Lower and more stable TTFT:** fewer long-prefix cache misses and an opportunity to hide remote-load latency behind tool execution, scheduling, or session switching. - **Higher effective KV capacity:** dead branches can be released promptly, while temporarily idle but expensive contexts receive bounded protection. The same HBM budget can therefore support a more useful working set, which may translate into more concurrent agent contexts or less capacity needed for the same service target. - **Less recomputation and data movement:** the application can distinguish a likely resume from a terminal state, allowing vLLM to spend prefill compute and connector bandwidth on contexts with known reuse value. The exact gain depends on prefix length, pause duration, cache pressure, and backend bandwidth. This RFC therefore treats these as benchmark hypotheses, not guaranteed improvements. The rollout section proposes measuring hit rate, recomputed prompt tokens, TTFT, throughput, and transfer volume. This is a cache-policy hint, not a new cache identity: - block hashes and prefix-cache matching remain unchanged; - identical content can still be reused across sessions; - `session_id` is not included in the cache key; - vLLM may ignore or override hints under memory pressure; - requests without hints behave exactly as they do today. ### Goals - Retain likely-to-be-reused cache blocks for a bounded time. - Release known-dead session context promptly. - Allow optional offload, prefetch, and evict operations. - Keep session policy separate from `BlockPool`. - Add negligible overhead to requests that do not use the feature. - Support local cache first and optional CPU/remote KV backends later. - Let applications express useful lifecycle intent without exposing vLLM block size, block tables, or physical block IDs. ### Proposed Change. ### 1. Initial request-level implementation The complete design can describe subranges, but the implementation currently being validated intentionally starts with a simpler profile: protection, offload, prefetch, and eviction operate on a whole request context. The agent runtime does not calculate or send block indices. For a normal inference request, vLLM associates `session_id` with the complete cacheable prefix materialized for that request. If `cache_control` is present, that complete association receives the requested TTL. For a pure management request, the operation applies to the complete context currently recorded for `session_id`. ```python class AgentHintParams(OpenAIBaseModel): session_id: str | None = None parent_session_id: str | None = None cache_control: CacheControlParams | None = None context_management: ContextManagementParams | None = None class CacheControlParams(OpenAIBaseModel): type: Literal[\"ephemeral\"] = \"ephemeral\" ttl: float = Field(default=300.0, ge=0, le=3600) # Optional fine-grained extension. None protects the whole request context. block_offset: int | None = Field(default=None, ge=0) class ContextManagementParams(OpenAIBaseModel): manage_request: bool = False edits: list[ContextEdit] = Field(default_factory=list) class ContextEdit(OpenAIBaseModel): type: Literal[\"offload\", \"prefetch\", \"evict\"] # Optional fine-grained extension. Both None target the whole request context. block_start: int | None = Field(default=None, ge=0) block_end: int | None = Field(default=None, ge=0) # exclusive ``` Protect a request context: ```json { \"model\": \"example-model\", \"messages\": [{\"role\": \"user\", \"content\": \"Inspect the repository.\"}], \"agent_hint\": { \"session_id\": \"worker-7\", \"parent_session_id\": \"orchestrator-1\", \"cache_control\": {\"type\": \"ephemeral\", \"ttl\": 300} } } ``` Prefetch a recorded session before it resumes: ```json { \"model\": \"example-model\", \"messages\": [{\"role\": \"user\", \"content\": \"\"}], \"agent_hint\": { \"session_id\": \"worker-7\", \"context_management\": { \"manage_request\": true, \"edits\": [{\"type\": \"prefetch\"}] } } } ``` This initial request-level implementation has several practical advantages for a vLLM integration: - the public API does not expose allocator-specific block indices; - all mapping work occurs at existing request allocation/completion boundaries; - validation can focus on lifecycle semantics and measurable benefit before adding range-selection complexity; - the implementation remains useful for the common cases of suspend, resume, branch completion, and session deletion. Internally vLLM still operates on blocks, but that is an implementation detail: SAM resolves the recorded request/session association to the current physical blocks and validates the expected hashes before changing metadata. In other words, the application is request-aware while vLLM remains block-aware. ### 2. Keep fine-grained fields, but defer their use in the initial implementation `block_offset`, `block_start`, and `block_end` remain in the request model as optional fields. They preserve the complete design and provide a compatible path to fine-grained management after the request-level workflow is validated. The current tests simply omit them: - `cache_control.block_offset=None` protects the whole cacheable request context; - `ContextEdit.block_start=None` and `block_end=None` target the whole recorded request context; - when fine-grained support is enabled, `block_start` and `block_end` must be supplied together and describe a session-relative half-open logical range; - a supplied `block_offset` protects the cacheable blocks from start to that logical offset of the request. These values are logical indices, never physical block IDs. Therefore retaining the fields does not require the simplified client workflow to understand vLLM's allocator or physical block table. When `context_management.manage_request` is true, vLLM skips model execution and applies the requested edits. With the default value `false`, edits run after the normal request completes. In the actual running process, message to token to block index translation is implemented by the upper-layer routing, or the parsing can be placed in the VLLM. ### 3. Add a small SessionAwareManager Add a `SessionAwareManager` (SAM) beside `KVCacheManager`. SAM owns: - session and optional parent/child relationships; - session-to-block and block-to-session indexes; - block-level TTL entries; - validation and execution of context-management edits. `BlockPool` continues to own physical block allocation, the free queue, and the prefix hash map. It does not store session IDs or session trees. ```mermaid flowchart LR API[\"Request + agent_hint\"] --> Scheduler[\"V1 Scheduler\"] Scheduler --> KVCM[\"KVCacheManager\"] Scheduler --> SAM[\"SessionAwareManager\"] KVCM --> Pool[\"Existing BlockPool\"] SAM -->|\"one metadata API\"| KVCM SAM -. \"optional lifecycle operations\" .-> Connector[\"KV connector capability\"] Connector -.-> Backend[\"CPU / remote KV store\"] ``` SAM records a logical block index and expected block hash with each association. ### 4. Add two aggregate fields to KVCacheBlock ```python @dataclass(slots=True) class KVCacheBlock: # Existing fields remain unchanged. _session_ref_cnt: int = 0 _ttl_expire_at: float = 0.0 ``` `_session_ref_cnt` is independent of the existing active-request `ref_cnt`. Detailed session IDs stay in SAM, so the additional per-block memory is constant and prefix-cache lookup does not need a session parameter. ### 5. Add one KVCacheManager metadata API SAM does not modify blocks directly. It uses one guarded method: ```python def update_block_meta( self, *, block_id: int, expected_block_hash: BlockHashWithGroupId | None = None, session_ref_delta: int = 0, ttl_expire_at: float | None = None, ) -> BlockMetaUpdateResult: ... ``` The method validates that: - the physical block still contains the expected cached content; - `_session_ref_cnt` cannot become negative; - SAM never changes active-request `ref_cnt`; - null blocks are not retained. If the block is free, the method also asks the free queue to reconsider its allocation priority. `KVCacheManager` only needs to notify SAM at existing request boundaries: 1. after cached/new blocks are assigned to a request; 2. when a request completes or a block is reassigned. ### 6. Best-effort TTL retention Free blocks are logically classified as: | Class | Condition | Allocation order | | --- | --- | --- | | A | TTL expired and no session reference | First | | B | TTL expired and has session references | Second | | C | TTL not expired | Last resort | This can be implemented by extending the existing free linked list with two boundaries; it does not require a new cache allocator. ```text [A: ordinary free] -> [B: session-associated] -> [C: TTL-protected] ``` TTL is deliberately soft: 1. allocate A blocks first; 2. then allocate B blocks; 3. if only C blocks remain and inference needs memory, break a C reservation. This guarantees that inaccurate hints cannot block scheduling or cause an avoidable out-of-memory failure. #### TTLManager and timer wheel SAM owns a `TTLManager` backed by a fixed-size timer wheel. Entries are keyed by `(block_id, session_id)`: ```python @dataclass class TTLBlockEntry: block_id: int session_id: str expire_at: float ``` `register()` is invoked when a block with a TTL is allocated or the TTL of a block is updated, and notifies the remote backend to keep the block alive of the corresponding session. `remove()` cancels an entry during evict or session cleanup, and notifies the remote backend to stop the block keepalive for the corresponding session. On every scheduler iteration, `tick(now)` advances elapsed wheel slots and calls `SAM.on_ttl_expired(block_id, session_id)` for truly expired entries. `on_ttl_expired` will call remove to delete the entry. The callback removes the session record, decrements the aggregate session reference, recomputes the block's latest remaining deadline, and calls `update_block_meta()`. Thus, expiration of one session does not remove a later TTL owned by another session. With a one-second tick and the proposed one-hour maximum TTL, the wheel has 3,600 slots. Registration, refresh, and removal are O(1); each tick processes only entries in elapsed slots. `time.monotonic()` is used throughout. Allocation also checks the block deadline, so scheduler delay or a skipped tick cannot prevent expired C-zone blocks from being reclaimed. ### 7. Session lifecycle and management operations Session registration is lazy and idempotent. Reusing a session refreshes its associations without double-incrementing `_session_ref_cnt`. `free_session(session_id)` removes that session's records and TTL entries. `free_session_tree(session_id)` optionally releases children before their parent. Neither operation invalidates blocks used by active requests or other sessions. In the initial implementation, management edits resolve to all blocks in the recorded request context for the target session. A later fine-grained extension may use session-relative half-open logical block ranges: - `evict`: cancel selected local and remote retention. - `offload`: persist selected KV data remotely, then release its local retention. - `prefetch`: load selected remote KV data into the local cache. Active request references always win. An edit never invalidates a block while its existing `ref_cnt` is positive. Offload and prefetch are capability based. If no backend supports an operation, vLLM returns `unsupported`; the local prefix-cache behavior remains valid. ### 8. Pooling and remote KV stores Local cache management is the first milestone. Pooled CPU or remote KV stores need the same lifecycle semantics, but the integration should not make vLLM core understand backend-specific PoolKeys, transfer threads, or eviction policies. #### Compatibility path for current connectors For connectors that only expose their existing lookup/save/load paths, an optional `SessionAwarePoolingManager` (SPM) can act as an adapter. It listens to SAM lifecycle events and maintains a temporary mapping: ```text session_id -> block hashes / connector keys ``` After a connector finishes saving KV data, SPM associates the saved keys with the session. It can then: - refresh active or TTL-protected keys at low frequency when the backend's `exists` operation also refreshes LRU state; - stop keep-alive after a session or key becomes orphaned, allowing the backend's existing LRU policy to reclaim it; - submit a bounded, low-priority load through the existing connector for prefetch, while reserving local capacity for normal inference; - confirm that offload/save completed before local retention is released. This adapter provides a way to test end-to-end semantics without requiring an immediate change to every connector or backend. It is deliberately optional and is not involved in local block allocation or prefix matching. ```mermaid flowchart LR Edit[\"offload / prefetch / evict\"] --> SAM[\"SAM: resolve logical blocks\"] SAM --> Local[\"KVCacheManager: local metadata\"] SAM --> SPM[\"SPM: resolve PoolKeys\"] SPM --> Tracker[\"SessionKeyTracker\"] SPM --> Queue[\"Bounded prefetch / eviction queue\"] Queue --> Store[\"CPU or remote KV store\"] ``` ### 9. Expected vLLM change surface The initial request-level implementation keeps the required vLLM core change small and localized: | Area | Change | | --- | --- | | Request protocol/internal request | Add optional request-level `agent_hint` fields | | New SAM module | Session indexes, TTL, edit controller | | `KVCacheBlock` | Add two aggregate metadata fields | | `KVCacheManager` | Add one metadata method and lifecycle notifications | | Free-block queue | Add metadata-based priority/reclassification | | Scheduler | Construct SAM, tick TTL, dispatch management requests | Fine-grained ranges and pooled-cache management are separate, optional layers: | Optional area | Change | | --- | --- | | Logical range support | Translate stable logical ranges only if the community chooses to expose them | | Compatibility SPM | PoolKey tracker, keep-alive, and remote operation queues for current connectors | | Connector capability | Propagate logical session/request identity and expose remote observation/control | | Remote backend | No required change; precise TTL/delete remains optional | The proposal does **not** change: - block hash calculation; - prefix-cache matching; - cache keys or `cache_salt`; - attention kernels; - model runners; - block tables sent to workers; - sampling or generated results; - existing connector behavior when no session-aware capability is enabled. SAM, range support, and pooled-cache integration should have separate experimental flags. With them disabled, the current cache and connector paths remain unchanged. In particular, connector support is not a prerequisite for reviewing or merging the local request-level path. ### Feedback Period. _No response_ ### CC List. @JuneHM @fc-liu @luokui183 @wangxiaochao6 @LookAround0301 ### Any Other Things. ### Co-Authors @HiC4Sh1e @JiahongZhang-Work @Xavier-Zeng @socrahow @liudi60 ### Results Based on the SWE-bench Verified data set and Qwen3-235B model, openJiuwen did a series of comparative tests: covering typical scenarios such as Bug fixes, feature development, and code refactoring, simulating 10 users concurrently using JiuwenSwarm to perform tasks, comparing the effect with and without computing affinity. The results show that the first token delay (TTFT) is reduced by 57.46% , the model request end-to-end (E2E) delay is reduced by 27.61% , the Prefix Cache hit rate is increased by 33% , and the peak Cache usage is reduced by 25.24% . \"Image\" Additionally, we also conducted a simple test without the agent framework. Experimental configuration: 20 sessions, 5 concurrent requests, request length 10-30K. By sending requests directly to vLLM. We observed that the TTFT was optimized by approximately 21%. | Metrics | Agent Hint OFF | Agent Hint ON | Agent Hint ON + Evict | | --- | --- | --- | --- | | TTFT (average) | 4.52 s | 3.58 s | 3.57 s | | TTFT (P50) | 4.57 s | 3.47 s | 3.57 s | | TPOT (average) | 0.030 s | 0.034 s | 0.032 s | | TPOT (P50) | 0.029 s | 0.034 s | 0.032 s | ### Backward compatibility and safety Requests without `agent_hint` behave exactly as before, and hints cannot affect generated tokens. `session_id` is not authorization: management operations must be tenant-scoped, while `cache_salt` continues to provide cache isolation. TTL, sessions, records, edit ranges, and prefetch work must be bounded; session IDs must not be metric labels. ### Performance and observability The target overhead is two scalars per block, O(1) reclassification, and work proportional to associated blocks at request boundaries. There is no session lookup during prefix matching or per-token decode work. Remote tracking and control run only when both a connector and session hints are enabled. Suggested metrics include: - sessions, records, protected blocks, and pressure overrides; - edit/prefetch results by operation and status; - tracked/shared/orphaned keys, keep-alive, and remote eviction counts; - connector transfer bytes, queue delay, completion latency, and failures; - prefix-cache hit rate and TTFT for hinted versus unhinted requests. ### Rollout and testing The change can be split into small PRs: 1. request schema and shadow SAM accounting; 2. local TTL retention and session cleanup; 3. local evict and structured management responses; 4. SPM with PoolKey tracking and keep-alive; 5. optional connector offload/prefetch/explicit-evict capabilities. Benchmarks should compare prefix-cache hit rate, recomputed prompt tokens, TTFT, throughput, scheduler CPU time, and metadata memory overhead against the current LRU baseline. ### Future direction #### Planned direct connector integration The longer-term integration should make session-aware management an optional connector capability instead of relying on SPM to infer remote state from side-effects. A capable connector is the component that already knows how local block hashes map to remote keys, whether data is resident, and when an asynchronous transfer has completed. It is therefore the right boundary for both observing and controlling remote KV state. The exact interface should be developed with connector maintainers, but the capability would cover operations equivalent to: ```python class SupportsSessionAwareKVManagement(Protocol): def register_cache(self, session_id, request_id, block_hashes) -> None: ... def offload(self, session_id, request_id=None) -> OperationHandle: ... def prefetch(self, session_id, request_id=None) -> OperationHandle: ... def evict(self, session_id, request_id=None) -> OperationHandle: ... def release(self, session_id, request_id=None) -> None: ... def query_residency(self, session_id, request_id=None) -> Residency: ... ``` The names and argument types above are illustrative, not a proposed final API. The important properties are: - **direct visibility:** connectors report save/load completion, remote residency, misses, and backend pressure through structured results or events; - **direct control:** offload, prefetch, retain/release, and explicit eviction use connector methods when supported, rather than scheduler-created virtual requests or assumed `exists()` side-effects; - **logical inputs:** core passes a session/request association and content hashes; connector-specific keys remain private to the connector; - **capability negotiation:** unsupported operations return `unsupported` and never change normal request execution; - **backend freedom:** a connector may implement exact TTL/delete, map release to stopped keep-alive, or rely on backend LRU according to its capabilities. ```mermaid flowchart LR Hint[\"request-level hint\"] --> SAM[\"SAM: logical lifecycle\"] SAM --> Local[\"KVCacheManager: local policy\"] SAM --> API[\"optional connector capability\"] API --> Observe[\"residency and completion events\"] API --> Control[\"offload / prefetch / evict / release\"] Observe --> Store[\"pooled CPU or remote KV store\"] Control --> Store ``` AscendStore can be an initial integration by propagating optional `session_id` and request identity in connector metadata, reporting saved block hashes after put completion, and reusing its lookup/save/load paths. As connector capability methods mature, its adapter-specific tracking can move behind the connector boundary. Other KV connectors can opt in independently; none of these remote capabilities are required for the initial local implementation. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52113", "createdAt": "2026-08-13T06:54:55Z", "updatedAt": "2026-08-13T06:54:55Z", "timestamp": "2026-08-13T06:54:55Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "RFC" ], "author": "FermatGo", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52125", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: vLLM Native Reusable Engine Snapshot Sleep (L3)", "text": "## Motivation. A vLLM cold start includes model weight loading, EngineCore initialization, CUDA context and allocator creation, `torch.compile`, CUDA Graph capture, KV cache profiling, and warmup. These steps are repeated for every new instance. This impacts responsiveness in elastic scaling and multi-model deployment scenarios. When an instance is released and re-launched, or when a new replica is scaled out from an existing one, the full initialization pipeline runs again. During model switching, previously completed compilation and runtime state are also difficult to reuse directly. [vLLM #34303](https://github.com/vllm-project/vllm/issues/34303) framed this as CUDA Checkpoint/Restore for Near-Zero Cold Starts and proposed combining CRIU to persist full process state. This RFC continues along that CRIU-based persistence path, using CRIU and CUDA C/R to save an already-initialized vLLM Engine. vLLM's existing L1/L2 sleep modes primarily manage GPU allocations such as weights and KV cache, while the EngineCore process remains alive. They are suitable for short-term HBM release, but cannot cause EngineCore to exit, nor produce a reusable artifact that can be restored multiple times. Once the process is destroyed, the next launch must redo Engine initialization from scratch. This RFC proposes adding a reusable Engine Snapshot as an L3-level sleep mode in vLLM: save the initialized Engine and CUDA state, allow the source process to exit after the snapshot is committed, and restore directly back to a serving state on wake. ## Core Goals and Design Philosophy Container-level snapshot solutions such as Dynamo have already demonstrated that CRIU and CUDA Checkpoint can jointly save and restore inference processes containing GPU state. These solutions treat the entire container as the snapshot boundary, and the restore process depends on coordination among operators, node agents, and CRI runtimes — tightly coupled with the Kubernetes control plane and runtime environment, making them suitable for scenarios where Kubernetes uniformly manages instance lifecycles. This proposal defines the snapshot boundary at the EngineCore level within vLLM and integrates with the existing sleep/wake lifecycle. Before entering deep sleep, vLLM drains requests and brings workers and communicators into a saveable state. After the snapshot is saved, the EngineCore process exits. On wake, EngineCore is restored, necessary runtime connections are re-established, and the Engine is confirmed functional before accepting requests again. When to save, what internal state to prepare, and at what stage of restore the Engine is considered ready — all of these are controlled by vLLM itself. This proposal provides deeper sleep capability than L2, and users can invoke it according to their own scheduling policies. Within this framework, generated snapshots can be restored repeatedly, and the approach can evolve to support cross-machine restore, 1→N, multi-GPU, and distributed scenarios. From the user's perspective, sleep depth is progressively layered: | Mode | State Retained After Sleep | |----|--------------------------| | L1 | EngineCore keeps running, weights retained as host-side backup, KV cache released | | L2 | EngineCore keeps running, weights and KV cache released, reloaded on wake | | Snapshot sleep (L3) | EngineCore saved as snapshot then exits, frontend and launcher continue running | L3 exposes two snapshot modes. `full` preserves weights, KV cache, and runtime state in the CUDA process image. `minimized` removes weights and KV cache before capture, then reloads weights from model files and rebuilds KV cache after restore. ## Design Phase 1 targets same-machine, single-GPU, TP=1 runtime sleep/wake. Deep sleep occurs while the service is running. After EngineCore exits, the service must still accept wake requests, maintain sleep state, and preserve the original HTTP listener. Based on this requirement, this proposal decouples the API frontend and EngineCore lifecycles: the frontend continues running during sleep, while `EngineCoreProc` exits and is restored as an independent snapshot object. ### Phase 1 Architecture ```text Before ====== ┌──────────────────────────────────────────┐ │ vllm serve main process │ │ │ │ HTTP / API │ │ MPClient │ │ CoreEngineProcManager │ └──────────────┬────────────────┬──────────┘ │ business ZMQ │ create / monitor │ │ ▼ ▼ ┌──────────────────────────────────────────┐ │ EngineCoreProc │ │ │ │ EngineCore │ │ UniProcExecutor + GPUWorker │ │ weights / KV / CUDA context / HBM │ └──────────────────────────────────────────┘ After Phase 1 ============= ┌──────────────────────────────────────────┐ │ API frontend │ │ │ │ HTTP listener / API │ │ admission / readiness │ │ MPClient │ └──────────────┬────────────────┬──────────┘ │ control │ business ZMQ │ │ ▼ │ ┌───────────────────────────────┼──────────┐ │ launcher │ │ │ │ │ │ CoreEngineProcManager │ │ │ · create / monitor │ │ │ · adopt restored process │ │ │ │ │ │ EngineSnapshotManager [new] │ │ │ · lifecycle state │ │ │ · artifact / provider │ │ └──────────────┬────────────────┘ │ │ │ │ manage + CRIU / CUDA C/R │ │ │ ▼ ▼ ┌──────────────────────────────────────────┐ │ EngineCoreProc subtree │ │ │ │ EngineCore │ │ UniProcExecutor + GPUWorker │ │ weights / KV / CUDA context / HBM │ │ │ │ entire subtree captured / restored │ └──────────────────────────────────────────┘ After deep sleep: launcher continues running API frontend continues running, retains HTTP listener EngineCoreProc saved as snapshot, then exits ``` `HTTP/API` and `MPClient` are split from the original main process into the persistent frontend. EngineCore process management stays in the launcher, with added capability to adopt restored processes. `EngineSnapshotManager` is a new component responsible for snapshot state and provider invocation. The internal structure of `EngineCoreProc` remains unchanged; the entire subtree is saved and restored as a unit. A control channel between the API frontend and launcher handles sleep, wake, and status queries. `MPClient` and EngineCore continue using business ZMQ. The business connection is torn down when EngineCore exits and re-established after restore. ### Phase 1 Lifecycle ```text READY → DRAINING → PREPARING → SNAPSHOTTING → HIBERNATED HIBERNATED → RESTORING → ATTACHING → VERIFYING → READY ``` During the sleep phase, the frontend first closes the request admission gate and confirms EngineCore is idle. EngineCore prepares the state to be saved according to the resource policy, then the provider executes CUDA checkpoint and CRIU dump. After the snapshot is committed, the source `EngineCoreProc` exits, and the system enters `HIBERNATED`. During the wake phase, the system first validates the snapshot and runtime environment, then restores `EngineCoreProc` and CUDA state. The launcher adopts the restored process, and the frontend re-establishes the business connection. After Engine state and inference result checks pass, the service returns to `READY`. ### Future Evolution Phase 1 delivers same-machine, TP=1 EngineCore snapshot and restore. Phase 2 addresses cross-machine snapshot usage. Phase 3 extends to multi-GPU, distributed topologies, and elastic restore. #### Phase 2: Single-Machine, Single-GPU Cross-Machine Restore (with 1→N Support) When moving across machines, state elements carrying source-machine identity — the original filesystem, `/dev/shm`, GPU, and communication addresses — become invalid and must be handled. 1. **vLLM runtime files.** a. **Compilation artifacts.** vLLM uses TorchInductor, AOTAutograd, and Triton. `VLLM_CACHE_ROOT/torch_compile_cache` contains generated Python modules, JIT `.so` files, Triton kernels, `cubin`, and compilation metadata. FlashInfer additionally produces JIT `.so` files, Ninja build files, and autotune results. b. **Shared memory.**`/dev/shm` may contain `psm_*` objects created by Python `SharedMemory`, `torch_*` objects used by PyTorch tensor IPC, and shared files that have been unlinked from the directory but are still held by the process. c. **Hardcoded paths and open files.** The model directory must be available in the target container. EngineCore may also retain absolute paths for cache, working directories, logs, temporary files, and Unix sockets. 2. **Restore fixup.** a. **Environment variables and in-process caches.** Values such as `VLLM_HOST_IP`, `CUDA_VISIBLE_DEVICES`, and `VLLM_RPC_BASE_PATH` may differ on the target instance. Some of these have already been absorbed into `VllmConfig`, worker configuration, or other runtime objects. Modifying `os.environ` alone will not synchronize already-created state; fixup must cover this in-process data. b. **Instance and network identity.** The target instance acquires a new IP, port, GPU UUID, and IPC paths. The ZMQ address between the API frontend and EngineCore, the worker's `distributed_init_method`, and cached device information in the process all need to be rechecked.c. **Prefill-Decode disaggregation connections.** KV Connector handshake information, peer addresses, and transport-layer state all originate from the source instance. After restoring to a new machine, Prefill and Decode instances must rediscover peers and complete the handshake. Connectors such as NIXL, Mooncake, LMCache, and P2P NCCL must also determine which registration states can be preserved and which must be re-established. 3. **(External dependency) Environment compatibility.** a. **Software environment.** File paths, Python, PyTorch, CUDA user-space libraries, Triton, and FlashInfer can be made compatible by using the same container image. b. **Hardware environment.** The CPU must have compatible architecture and instruction sets. The GPU must use the same GPU model, compute capability, and memory specifications. #### Phase 3: Single-Machine Multi-GPU With TP\\>1, a single EngineCore manages multiple workers through the Executor. Each rank has its own process, CUDA context, and GPU state, while being linked together through communication resources such as NCCL and custom all-reduce. The snapshot boundary must expand from a single CUDA process to the entire EngineCore worker group. 1. **Multi-GPU boundaries for CUDA Checkpoint.** a. **Multiple CUDA processes.** vLLM's multiprocessing executor maps each TP rank to one worker process and one GPU. The full set of CUDA PIDs must be identified per rank. b. **Cross-process CUDA resources.** Current CUDA Checkpoint has good support for ordinary CUDA allocations, but CUDA IPC memory still has limitations. vLLM's NCCL P2P, cuMem/NVLS, symmetric memory, and custom all-reduce may all use cross-process GPU mappings. 2. **Communication resources.** a. **NCCL communicator.** Determine whether the communicator is preserved as part of process state, suspended before checkpoint and restored or rebuilt after restore. NCCL internals may also hold P2P mappings, shared memory, and network transport state; different transports have different restorability characteristics. b. **vLLM communicator lifecycle.** PR #46877 has already added `checkpoint_prepare` and `checkpoint_restore` hooks to the communicator, providing a unified entry point for each communication backend. PR #51485 is using `ncclCommSuspend`/`ncclCommResume` to handle NCCL buffers during ordinary sleep. c. **CUDA Graph.** May record collective kernels, communication buffer addresses, and communicator handles. If communication resources change after restore, the original graph may become invalid. #### Phase 4: Multi-Node Multi-GPU In multi-node deployments, workers may be launched by Ray or an external launcher and are no longer part of a single local process tree on the machine where EngineCore resides. The snapshot object must be extended into a cross-node snapshot group. 1. **Cross-node snapshot generation.** a. **Multi-rank synchronization.** When producing a snapshot, all workers must enter the same checkpoint barrier. On restore, each rank must complete process restore, CUDA restore, and communication state handling before EngineCore can resume scheduling requests. b. **Cross-node processes.** Each node can only save its own worker processes and CUDA state. The management model for EngineCore, driver worker, remote workers, and DP coordinator must be clearly defined. 2. **Post-restore topology and identity correction.** a. **Rank placement.** On restore, the original TP/PP/DP ranks must be mapped to target nodes and GPUs, and `CUDA_VISIBLE_DEVICES`, `LOCAL_RANK`, and the rank-to-logical-device mapping must be regenerated. b. **Distributed addresses.**`MASTER_ADDR`, `MASTER_PORT`, `VLLM_HOST_IP`, `distributed_init_method`, DP coordinator addresses, and various ZMQ endpoints may all change. Ray actors are also associated with node IDs, GCS, and actor runtime; processes managed by an external launcher depend on external rendezvous. c. **NCCL network state.** Cross-node NCCL communicators hold bootstrap addresses, sockets, InfiniBand/RDMA queue pairs, GIDs, and memory registrations. ## Relationship to PR #51360 We started this work before PR #51360 came to our attention. Both efforts follow the same basic approach: using CRIU and CUDA C/R to capture an initialized Engine and restoring it from a snapshot to skip Engine initialization. Both currently target the same scope: same-host, single-GPU, TP=1. The main difference lies in when the snapshot is used and the semantics exposed to users. This leads to different snapshot boundaries: 1. **PR #51360 targets instance startup.** It launches a snapshot server from the command line and captures the entire vLLM process tree after Engine initialization but before the HTTP listener is bound. A new instance restores the frontend and EngineCore together, then creates the HTTP listener. Its broader snapshot boundary makes the restore path more direct, since the EngineCore does not need to reconnect to an existing frontend. 2. **This proposal targets runtime deep sleep.** A service can run normally and enter sleep through an HTTP API when it becomes idle. The frontend and HTTP listener remain alive during sleep to handle status and wake requests. The EngineCore is captured and exits, then is restored and reattached to the existing frontend on wake-up. The direct benefit of this proposal is that Engine Snapshot can be exposed as a native vLLM sleep level. A higher-level system can decide when to put a running instance to sleep based on actual load, regardless of how long the service has already been running. The HTTP listener and service address remain unchanged across sleep/wake cycles, so there is no need to restart the service process or recreate the control endpoint. These semantics are better suited to reclaiming resources from running instances while reusing vLLM’s existing sleep/wake API and state management. Both approaches restore an initialized Engine, so we do not expect a fundamental performance difference. PR #51360 benefits from a broader process boundary and a simpler restore path. This proposal handles EngineCore detachment, restoration, and reattachment in exchange for allowing a running service to sleep and wake at any time. We have discussed the overlap with the author of PR #51360 and reached a shared understanding of the first-phase scope. The subsequent work—cross-host restore, GPU relocation, 1→N, multi-GPU, and distributed restore—focuses on snapshot migration and coordinated restore, and does not directly depend on which frontend boundary is chosen in the first phase. ## 5\\. Related Issues, RFCs, and PRs - [#34303](https://github.com/vllm-project/vllm/issues/34303) is the RFC this document directly continues. It proposed using CUDA Checkpoint/Restore to shorten cold starts and listed CRIU-based full-process state persistence as a future direction. This document advances the full-process snapshot path within that framework. - [#51360](https://github.com/vllm-project/vllm/pull/51360) has implemented same-machine, TP=1 initialized-engine snapshot, directly overlapping with Phase 1 of this proposal. The two efforts differ in snapshot boundary and runtime lifecycle; see the previous section for details. - [#35934](https://github.com/vllm-project/vllm/pull/35934), [#37921](https://github.com/vllm-project/vllm/pull/37921), and [#37925](https://github.com/vllm-project/vllm/pull/37925) are earlier CRIU-safe suspend/resume and CUDA checkpoint prototypes, covering scheduler, worker, communicator, and HTTP API prepare and restore. These PRs are older but their lifecycle handling remains a useful reference. - [#44074](https://github.com/vllm-project/vllm/pull/44074) introduced a pluggable `SleepModeBackend`. [#46877](https://github.com/vllm-project/vllm/pull/46877) added communicator checkpoint prepare/restore hooks. When subsequently integrating CUDA checkpoint backends and handling NCCL and other communication resources, these common interfaces should continue to be used. - [Dynamo Snapshot](https://github.com/ai-dynamo/dynamo/tree/main/deploy/snapshot) provides a complete implementation for Kubernetes scenarios, including operator, node agent, CRIU/CUDA C/R, container environment recording, artifact storage, and restore cleanup. It serves as an important reference for external orchestration and can directly invoke vLLM's Engine Snapshot primitives in the future. ## Current Progress and Experimental Results The Phase 1 prototype has completed same-machine, single-GPU, TP=1 sleep/wake validation. Both public modes, `full` and `minimized`, have passed real CRIU/CUDA capture and restore. Two additional internal resource ablations were tested earlier for engineering comparison only and are not part of the user-facing interface. ### Test Environment - GPU: NVIDIA H20, single GPU exposed inside the container - Model: Qwen3-8B, BF16 - Topology: TP=1, PP=1, DP=1, single API server - Engine: non-eager, `max_model_len=8192`, `max_num_seqs=16`, `gpu_memory_utilization=0.50` - Software: PyTorch `2.13.0+cu130`, CUDA 13.0, NVIDIA driver/cuda-checkpoint `580.105.08`, CRIU 4.1 `20c6014e1` - Storage: model, compilation cache, CRIU images, and snapshots all on tmpfs - Snapshot config: `page_cache + optimistic`, skip SHA-256 computation for large files on restore - Sampling: 6 runs per group, first run as warmup, last 5 runs used for statistics. Times reported in the table are medians. `minimized` was run independently in three groups, totaling 15 steady-state runs. L3 restore time is measured from the `/wake_up` request to the first non-empty token received in a fixed streaming request, with final response content confirmed correct. ### Performance Results | L3 Policy | Runs | Checkpoint Time | Restore Time | Artifact Size | | --- | ---: | ---: | ---: | ---: | | `minimized` | 15 | 2.837 s | **5.948 s** | 4.251 GiB | | `full` | 5 | 27.242 s | 22.990 s | 49.659 GiB | Baseline results from the same batch: | Path | Runs | Sleep | Wake / Start | First Inference | |----|----|-----|------------|---------------| | Fresh Engine | 5 | — | 35.396 s to `/ready` | 0.054 s | | CuMem L1 | 5 | 0.340 s | 0.407 s | 0.044 s | | CuMem L2 | 5 | 0.047 s | 2.907 s | 0.049 s | ### Restore Breakdown Phase-by-phase timing for `minimized`: | Phase | Median | |-----|------| | CRIU restore | 1.861 s | | CUDA restore-TID discovery | 0.335 s | | CUDA restore action | 0.553 s | | CUDA unlock | 0.021 s | | Provider total | 2.810 s | | Worker restore | 3.147 s | | Post-restore TTFT | 0.014 s | ### Correctness and Resource Results - All 30 formal L3 runs restored successfully and passed fixed semantic checks. - Model page residency rate was 100% before every restore. - Engine `read_bytes` delta was 0 for all runs. - Across 30 `HIBERNATED` checks, project process HBM and full-GPU HBM were both 0 MiB. - Ruff and Python 3.10 mypy passed. - Snapshot tests: `75 passed`. - CLI parser tests: `3 passed, 30 deselected`. ### Feedback Period. _No response_ ### CC List. _No response_ ### Any Other Things. ## Discussion Topics 1. **Snapshot boundary.** This proposal preserves the frontend and only saves/restores EngineCore. #51360 saves the entire initialized process tree. We would like to discuss which scenarios each approach is best suited for, and the trade-offs around HTTP lifecycle, restore adoption, and persistent overhead. 2. **User interface.** Should Engine Snapshot be integrated as a new sleep level within `sleep/wake`, or exposed as a standalone `snapshot create/restore` interface? Which lifecycle and state definitions can be shared between the two entry points? 3. **Future evolution.** This proposal plans to progress through cross-machine 1→N, single-machine multi-GPU, and multi-node multi-GPU. We would like to discuss whether this roadmap aligns with the community's plans, which scenarios should be prioritized, and what additional considerations are needed for cross-machine and distributed restore. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52125", "createdAt": "2026-08-13T08:25:08Z", "updatedAt": "2026-08-13T13:26:59Z", "timestamp": "2026-08-13T13:26:59Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "RFC" ], "author": "fuhefei", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52137", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: split local/external prefix-cache hits in `prompt_tokens_details`", "text": "### 🚀 The feature, motivation and pitch ## Background With distributed prefix caching (KV connectors such as Mooncake / lmcache), cache hits in a single prefill can come from two different sources: - **local**: hits on this instance's GPU prefix cache; - **external**: hits fetched from a remote KV store / another instance via the KV connector. The scheduler already tracks `num_local_cached_tokens` / `num_external_cached_tokens` separately in `PrefillStats`, but `OutputProcessor` only forwards their sum (`num_cached_tokens`), so the breakdown is lost before it reaches the OpenAI serving layer — users cannot tell, per request, where cache hits came from. Exposing this information per-request may also be handy for pricing purposes. ## Current behavior With `--enable-prompt-tokens-details` enabled, the response currently looks like: ```json \"prompt_tokens_details\": { \"cached_tokens\": 3627, \"created_cache_tokens\": 0, \"multimodal_tokens\": null } ``` ## Expected behavior Add two optional keys, `local_cached_tokens` / `external_cached_tokens`, alongside the existing fields: ```json \"prompt_tokens_details\": { \"cached_tokens\": 2320, \"created_cache_tokens\": 0, \"multimodal_tokens\": null, \"local_cached_tokens\": 1216, \"external_cached_tokens\": 1104 } ``` Invariant: `cached_tokens == local_cached_tokens + external_cached_tokens`. ## Requirements - Strictly additive change: existing fields (`cached_tokens` / `created_cache_tokens` / `multimodal_tokens`) must be preserved untouched; - New fields default to `None` and must not affect existing output when the flag is off or the values carry no meaning; - Cover both streaming and non-streaming paths for chat and completion endpoints, and also benefit offline `LLM()` users. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52137", "createdAt": "2026-08-13T09:18:49Z", "updatedAt": "2026-08-13T17:17:15Z", "timestamp": "2026-08-13T17:17:15Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "fataswellassad", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52146", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] xgrammar backend ignores the model's other EOS ids, so guided decoding can stop mid-string", "text": "`XgrammarBackend.__post_init__` builds the `TokenizerInfo` from `tokenizer.eos_token_id` alone: ```python tokenizer_info = xgr.TokenizerInfo.from_huggingface( self.tokenizer, vocab_size=self.vocab_size, ) ``` `from_huggingface` accepts `stop_token_ids`; omitted, it derives a single id. A model whose `generation_config.json` declares several EOS ids ends up with a grammar that knows one of them. xgrammar holds a *stop* token back until the grammar may terminate — but an id it was never told about is ordinary vocabulary, legal wherever its text is legal, which inside a JSON string is essentially everywhere. The engine, which does know the full set, then stops on it: `finish_reason=\"stop\"` on a guided request whose JSON is cut inside a string. Ours declares `\"eos_token_id\": [200001, 200008]` while `tokenizer.eos_token_id` is `200001`. Measured at a prefix inside a property-name string: ``` from_huggingface(tok, vocab_size=len(tok)) -> 200008 allowed: True from_huggingface(tok, vocab_size=len(tok), stop_token_ids=[200001, 200008]) -> 200008 allowed: False ``` ### Fix Union the ids the engine can stop on and pass them. `model_config.try_get_generation_config()` already documents itself as the place where \"information about special tokens\" lives. The patch on this branch adds `model_stop_token_ids()` and wires it into **both** `TokenizerInfo` construction sites — the mistral branch made the same single-id assumption — with unit tests covering the multi-id, single-id, absent, duplicate and malformed cases.", "url": "https://github.com/vllm-project/vllm/issues/52146", "createdAt": "2026-08-13T10:06:35Z", "updatedAt": "2026-08-13T10:37:25Z", "timestamp": "2026-08-13T10:37:25Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "y-edy-os", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52150", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][ROCm/gfx942]: GLM-5.2-FP8 — first request after GPU idle emits garbage; piecewise CUDA graph cold replay corrupts the request's own prefill (workaround: cudagraph_mode=FULL_DECODE_ONLY)", "text": "## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Images: `vllm/vllm-openai-rocm:nightly` 2026-08-12 (v0.26.1rc1.dev668+g3ee2df303) **and** 2026-08-13 (`8f515d1d`) — both reproduce - Model: `zai-org/GLM-5.2-FP8`, TP=8, `--kv-cache-dtype auto`, `--max-num-batched-tokens 4096`, MoE backend AITER blockscale-FP8 (`fmoe_bf16_a16_blockscaleFp8...`) - Reproduces identically at `--max-model-len` 131072 and 1048576 ## 🐛 Describe the bug After the engine sits idle for ≥ ~20–50 s, the **next** request returns garbage from its very first sampled token, e.g. `'!0.1.5,0.1 and1.1.1.0 21.1.'` (token-0/digit noise). Requests sent immediately afterwards are correct. Reproduction probability rises with idle time (~50–100 % at ≥ 50 s idle; back-to-back requests never fail, dozens of samples). Key observations: 1. **The corrupt request never self-heals**: with `max_tokens=300` the output is noise from token 1 through token 300, while a request issued ~0.5 s later is clean — i.e. the corrupt request's *own prefill* wrote garbage into its KV/indexer cache, and every decode step then consumes that poisoned context. 2. `temperature=0`; prompts are ~6–10 tokens (below the 16-token block size, so prefix caching is not involved). 3. No errors in the server log, container healthy, RAS/ECC counters unchanged. 4. Not JIT/autotune related: Triton JIT warnings (`_cp_gather_indexer_quant_cache_kernel`, `_fp8_mqa_logits_kernel`) appear only on the very first request after startup; later corrupt events have no JIT activity. 5. A 25 s-interval keepalive (unique prompts forcing real prefills, all HTTP 200) does **not** reliably prevent it. ## Bisection | config | result | |---|---| | defaults (FULL_AND_PIECEWISE graphs), 131k & 1M, nightly 08-12 & 08-13 | ❌ corrupt | | `--enforce-eager` | ✅ clean 8/8 (startup burst + 60 s idle ×3 + 25 s idle ×2) | | `--compilation-config='{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}'` | ✅ clean 8/8 at 1M | → The corruption tracks **piecewise CUDA/HIP graph replay in the prefill path** after a cold window. With prefill graph-free (eager or `FULL_DECODE_ONLY`) the model is fully healthy: fact probes pass, tool calling OK, needle-in-haystack 3/3 at 2k/6k/12k/100k/900k tokens, and a 5-minute cold soak followed by a fresh request is correct. Other models on the same host/stack (Kimi-K3 with eager prefill + FULL decode graphs, DeepSeek-V4-Flash-0731) do not show this — consistent with the piecewise-prefill-graph attribution: decode-side graphs essentially never replay cold, because the preceding prefill has just warmed the GPU. This smells like a ROCm/HIP graph runtime issue (queue/doorbell state after idle → first graph replay misfires) rather than a model bug, but filing here since the practical impact is \"GlmMoeDsa default config silently corrupts on gfx942\" and the workaround is a vLLM flag. Related (same class — HIP graph replay silently wrong on AMD — but none match the idle-triggered gfx942 manifestation): - ROCm/hip#3887 — HIP graph replay reads stale device-pointer values (gfx1201/RDNA4, ~200+ nodes) - ROCm/aiter#3632 — W4A4 fused MoE kernel not HIP-graph-safe (gfx950) - pytorch/pytorch#155684 — HIP graph misuse yields silent wrong results where CUDA raises ## Workaround `--compilation-config='{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}'` — no measurable decode-speed loss (decode keeps FULL graphs; only prefill runs graph-free). Single-stream decode on this host: 44.5 tok/s (prose) / 61.9 tok/s (code) at 1M `max_model_len`. ## Repro 1. Serve `zai-org/GLM-5.2-FP8` on 8×gfx942 with the default compilation config. 2. `sleep 60` 3. ```bash curl -s localhost:8000/v1/completions -H 'content-type: application/json' \\ -d '{\"model\":\"glm\",\"prompt\":\"The capital of France is\",\"max_tokens\":16,\"temperature\":0}' ``` → garbage (`!0.1...`). Repeat immediately → ` Paris.` (correct). Vary the prompt each round to rule out prefix-cache effects; alternate idle/burst to see the pattern.", "url": "https://github.com/vllm-project/vllm/issues/52150", "createdAt": "2026-08-13T10:37:11Z", "updatedAt": "2026-08-13T10:42:04Z", "timestamp": "2026-08-13T10:42:04Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm", "kimi" ], "author": "zzw09773", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52154", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Standalone torch.compile cache uses stale artifact path after relocation", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 4.4.2 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cpu Is debug build : False CUDA used to build PyTorch : Could not collect ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.4.239-1.el7.elrepo.x86_64-x86_64-with-glibc2.39 ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 2 Stepping: 6 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 3 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 80 MiB (64 instances) L3 cache: 96 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-31,64-95 NUMA node1 CPU(s): 32-63,96-127 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Vulnerable Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Vulnerable Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] numpy==2.3.5 [pip3] pyzmq==27.1.0 [pip3] torch==2.13.0+cpu [pip3] torchaudio==2.11.0+cpu [pip3] torchcodec==0.15.0 [pip3] torchvision==0.28.0+cpu [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.2rc1.dev29+g50ba4bc6b (git sha: 50ba4bc6b) vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 SYS SYS PXB PXB SYS SYS SYS SYS SYS 0-31,64-95 0 N/A GPU1 NV12 X SYS SYS SYS SYS SYS PXB PXB SYS SYS 32-63,96-127 1 N/A NIC0 SYS SYS X PIX SYS SYS SYS SYS SYS SYS SYS NIC1 SYS SYS PIX X SYS SYS SYS SYS SYS SYS SYS NIC2 PXB SYS SYS SYS X PIX SYS SYS SYS SYS SYS NIC3 PXB SYS SYS SYS PIX X SYS SYS SYS SYS SYS NIC4 SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS NIC5 SYS PXB SYS SYS SYS SYS SYS X PIX SYS SYS NIC6 SYS PXB SYS SYS SYS SYS SYS PIX X SYS SYS NIC7 SYS SYS SYS SYS SYS SYS SYS SYS SYS X PXB NIC8 SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=, CUBLAS_VERSION=13.1.0.3 NVIDIA_REQUIRE_CUDA=cuda>=9.0 TORCHINDUCTOR_LOOP_ORDERING_AFTER_FUSION=0 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX NCCL_VERSION=2.28.8 NVIDIA_DRIVER_CAPABILITIES=compute,utility,video TORCH_NCCL_USE_COMM_NONBLOCKING=0 CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0 NVIDIA_PRODUCT_NAME=PyTorch CUDA_VERSION=13.0.2.006 PYTORCH_VERSION=2.10.0a0+b558c98 PYTORCH_BUILD_NUMBER=0 CUBLASMP_VERSION=0.6.0.84 CUDNN_FRONTEND_VERSION=1.15.0 CUDA_COMPONENT_LIST=cccl crt nvrtc driver-dev culibos-dev cudart cudart-dev nvcc CUDNN_VERSION=9.15.0.58 PYTORCH_HOME=/opt/pytorch/pytorch NVIDIA_BUILD_ID=231036167 CUDA_DRIVER_VERSION=580.95.05 PYTORCH_BUILD_VERSION=2.10.0a0+b558c98 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda CUDA_MODULE_LOADING=LAZY NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS= NVIDIA_PYTORCH_VERSION=25.11 TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/mnt/tmp/solar-serving-study/tmp/torchinductor_nsml ```
### 🐛 Describe the bug vLLM's [`torch.compile` compilation cache documentation](https://docs.vllm.ai/en/stable/design/torch_compile/#compilation-cache) documents that the `torch_compile_cache` directory can be copied in deployment scenarios to reuse compilation artifacts. For standalone Inductor compilation, an artifact handle is persisted as: ```text (artifact_key, absolute_path) ``` `InductorStandaloneAdaptor.load()` currently resolves the artifact directly from the persisted absolute path: ```python path = handle[1] ``` If the compile cache is relocated to a different cache root, the artifact moves with the cache, but the persisted handle still points to its original location. Once that location is unavailable, loading through the persisted handle attempts to access the stale path. ### Minimal reproducer The following reproduces the path-resolution behavior on vLLM main at commit `50ba4bc6b` without requiring a model or an existing compiled artifact: ```python from unittest.mock import MagicMock, patch from vllm.compilation.compiler_interface import InductorStandaloneAdaptor old_cache_dir = \"/tmp/vllm-cache-old\" new_cache_dir = \"/tmp/vllm-cache-new\" artifact_key = \"artifact_shape_None_subgraph_0\" adaptor = InductorStandaloneAdaptor(save_format=\"binary\") adaptor.initialize_cache(new_cache_dir) handle = ( artifact_key, f\"{old_cache_dir}/{artifact_key}\", ) with ( patch( \"torch._inductor.CompiledArtifact.load\", return_value=MagicMock(), ) as load_mock, patch( \"torch._inductor.compile_fx.graph_returns_tuple\", return_value=True, ), ): adaptor.load( handle=handle, graph=MagicMock(), example_inputs=[], graph_index=0, compile_range=MagicMock(), ) print(\"current cache dir:\", new_cache_dir) print(\"persisted path: \", handle[1]) print(\"path used by load:\", load_mock.call_args.kwargs[\"path\"]) ``` Observed output: ```text current cache dir: /tmp/vllm-cache-new persisted path: /tmp/vllm-cache-old/artifact_shape_None_subgraph_0 path used by load: /tmp/vllm-cache-old/artifact_shape_None_subgraph_0 ``` Although the adaptor is initialized with the relocated cache directory, `CompiledArtifact.load()` is still called with the absolute path from the original cache location. I also verified the failure with an existing standalone compiled artifact by relocating the same artifact from cache root A to cache root B: ```text ORIGINAL_PATH_LOAD = PASS STALE_PATH_LOAD = FAIL (FileNotFoundError) REBASED_PATH_LOAD = PASS ``` The artifact itself remains loadable from B after relocation. Loading through the persisted handle fails when it still references A, which isolates the stale absolute path as the failure condition. ### Expected behavior A standalone compile cache should remain loadable when copied to a different cache root for reuse and the cached artifacts are otherwise unchanged. Artifact loading should resolve the corresponding artifact from the current cache directory rather than depend on the cache's original absolute path. A fix with regression coverage for both `binary` and `unpacked` standalone artifact formats is available in #52142. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52154", "createdAt": "2026-08-13T10:51:36Z", "updatedAt": "2026-08-13T10:51:36Z", "timestamp": "2026-08-13T10:51:36Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "jungjiyu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52155", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: VLLM_BATCH_INVARIANT does not cover convolution, used in VAE blocks", "text": "### Your current environment OS : Ubuntu 22.04.5 LTS (x86_64) Clang version : 22.0.0git (roc-7.2.3 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) Libc version : glibc-2.35 PyTorch version : 2.11.0+gitd0c8b1f ROCM used to build PyTorch : 7.2.53211 Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.15.0-131-generic-x86_64-with-glibc2.35 Is CUDA available : True GPU models and configuration : (gfx942:sramecc+:xnack-) # 8x AMD Instinct MI300X HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 CPU : Intel(R) Xeon(R) Platinum 8480C, 224 CPU(s), 2 sockets, 2 NUMA nodes ### 🐛 Describe the bug `init_batch_invariance()` does not address convolution, so a model whose forward contains one keeps a nondeterministic reduction while `VLLM_BATCH_INVARIANT=1` is on. The `torch.library` override set covers `mm`, `addmm`, `matmul`, `linear`, `softmax`, `mean.dim`, `bmm` and `rms_norm`. The only line touching conv is ```python # vllm/model_executor/layers/batch_invariant.py, init_batch_invariance() torch.backends.cudnn.conv.fp32_precision = \"ieee\" ``` which constrains precision, not reduction order — and `torch.backends.cudnn.deterministic` is never set. Affected models are the ones with a convolution in the forward path: multimodal vision patch-embeds (Qwen2.5-VL's is a `conv3d`) and diffusion VAEs. **This is not ROCm-specific**: that flag exists because cuDNN also has nondeterministic convolution algorithms, and it is unset on both backends. Possibly what [#27059](https://github.com/vllm-project/vllm/issues/27059) hit before it was closed as stale, though I have not verified that on CUDA. **Suggested fix.** Set it alongside the existing `fp32_precision` lines — already inside `if envs.VLLM_BATCH_INVARIANT:`, so the default path is untouched, and it has no effect on models without convolutions. On ROCm the same flag maps to MIOpen, no backend branch. **Scope.** The missing conv coverage is a code fact and platform-independent. The measurement is run-to-run at fixed shape on ROCm, not batch invariance, and not a claim about SM80/SM90. `cudnn.deterministic` also does not pin *which* algorithm is selected, and the heuristics read batch size as part of the shape — deterministic conv and batch-invariant conv stay different claims. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52155", "createdAt": "2026-08-13T10:57:25Z", "updatedAt": "2026-08-13T11:13:10Z", "timestamp": "2026-08-13T11:13:10Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "ZJLi2013", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52167", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Extended online quantization roadmap", "text": "### Motivation. Continuation of https://github.com/vllm-project/vllm/issues/32029. Online quantization API is now relatively stable at https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online and documented at https://docs.vllm.ai/en/stable/features/quantization/online/. This RFC summarizes and motivates proposed features that would be helpful to make online quantization to be more flexible and more widely compatible with models, including pre-quantized ones. ### Proposed Change. ### 1. Support activation quant key override Implemented in https://github.com/vllm-project/vllm/pull/51808. This is necessary to dispatch on specific linear/moe backends that can be picked only through the `activation_quant_key` (see e.g. https://github.com/vllm-project/vllm/blob/c4e969294ecab9ffefb995b734303f13f62b723f/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L441-L445) ### 2. Targeted online quantization configuration based on user patterns Implemented in https://github.com/vllm-project/vllm/pull/51285, RFC in https://github.com/vllm-project/vllm/issues/50281. The goal is simply to extend the UI to support e.g. `--quantization-config.targets '{\"re:.*o_proj.*\":\"mxfp8\"}'`, e.g. to target a single layer similar to e.g. https://github.com/vllm-project/vllm/blob/dc9f845ddc54c1df38fdbce5afe03f9fd15813bd/vllm/model_executor/layers/attention/mla_attention.py#L935-L948 (that is currently hard-coded and may be causing accuracy issues if done silently https://github.com/vllm-project/vllm/pull/48051), or e.g. `--quantization-config.targets '{\"re:.*shared_expert.*\":\"fp8_per_block\"}'` to enable fused shared expert. ### 3. Support online quantization with partially pre-quantized checkpoints Implemented in https://github.com/vllm-project/vllm/pull/51392. This is necessary to combine online quantization with already partially quantized checkpoints, which is notoriously the case recently. This is e.g. useful to test the accuracy/speed tradeoff of quantizing certain additional layers, or useful to e.g. release a single model compatible with fused shared expert / unfused shared expert. This has e.g. been requested in separate PRs: - https://github.com/vllm-project/vllm/pull/48983 - https://github.com/vllm-project/vllm/pull/49313 ### 4. Requantize already-quantized layers to different precision Not implemented yet, closest prototype is https://github.com/vllm-project/vllm/pull/48427. This is similar to https://github.com/sgl-project/sglang/pull/28291 & https://github.com/sgl-project/sglang/pull/29328 on sglang side. This idea here would be to allow loading through a certain quantization method, dequantize and override the quantization method on request. This has e.g. been requested in: - https://github.com/vllm-project/vllm/pull/48427 - https://github.com/vllm-project/vllm/pull/51274 ### Feedback Period. One month ### CC List. @BowenBao @mgoin @kylesayrs @vkuzo @maeehart @tanpinsiang @fanxingran @access2rohit ### Any Other Things. _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52167", "createdAt": "2026-08-13T13:25:24Z", "updatedAt": "2026-08-13T15:09:22Z", "timestamp": "2026-08-13T15:09:22Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "RFC", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52170", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "OffloadingConnector AssertionError in _build_store_jobs under MultiConnector with high concurrency multi-turn", "text": "## Summary The OffloadingConnector's scheduler crashes with an `AssertionError` in `_build_store_jobs()` when running under a MultiConnector (NixlConnector + OffloadingConnector) with high-concurrency multi-turn agentic workloads. ## Environment - **vLLM**: v0.26.0 (`vllm/vllm-openai:v0.26.0`) - **Model**: RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block (Mamba-hybrid MoE, TP8) - **Hardware**: 4× H200 nodes (140GB each), 8 GPUs per node, InfiniBand RDMA - **Architecture**: Prefill/Decode disaggregation (3 prefill + 1 decode, TP8) - **KV transfer config**: MultiConnector with NixlConnector (kv_consumer) + OffloadingConnector (200GB CPU, lazy_offload=true) - **Workload**: 30 concurrent multi-turn conversations, 540 turns per conversation, 160K first-prompt tokens, 15s inter-turn delay, FP8 KV cache, prefix caching enabled ## Error ``` (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] EngineCore encountered a fatal error. (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] Traceback (most recent call last): File \".../vllm/v1/engine/core.py\", line 1323, in run_engine_core engine_core.run_busy_loop() File \".../vllm/v1/engine/core.py\", line 1364, in run_busy_loop self._process_engine_step() File \".../vllm/v1/engine/core.py\", line 1403, in _process_engine_step outputs, model_executed = self.step_fn() File \".../vllm/v1/engine/core.py\", line 645, in step_with_batch_queue scheduler_output = self.scheduler.schedule(self._should_throttle_prefills()) File \".../vllm/v1/core/sched/scheduler.py\", line 1174, in schedule meta = self._build_kv_connector_meta(self.connector, scheduler_output) File \".../vllm/v1/core/sched/scheduler.py\", line 1196, in _build_kv_connector_meta return connector.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 422, in build_connector_meta metadata=tuple( File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 423, in c.build_connector_meta(scheduler_output) for c in self._connectors File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py\", line 157, in build_connector_meta return self.connector_scheduler.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 1157, in build_connector_meta store_jobs=self._build_store_jobs(scheduler_output), File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 975, in _build_store_jobs assert len(offload_keys) == len(offload_block_ids) AssertionError ``` ## Reproduction The crash occurs on the **decode** pod after several minutes of sustained multi-turn traffic. It does not happen immediately on startup — the model loads and serves requests successfully before the offloading scheduler's internal state diverges. KV transfer config used: ```json { \"kv_connector\": \"MultiConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"connectors\": [ { \"kv_connector\": \"NixlConnector\", \"kv_role\": \"kv_consumer\", \"kv_connector_extra_config\": {\"bidirectional_kv_xfer\": true} }, { \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"cpu_bytes_to_use\": 214748364800, \"lazy_offload\": true } } ] } } ``` vLLM flags: `--kv-cache-dtype fp8 --enable-prefix-caching --enable-expert-parallel --max-model-len 262144` ## Analysis The assertion at `scheduler.py:975` expects `offload_keys` and `offload_block_ids` to have the same length when building CPU offload store jobs. Under heavy multi-turn load with 30 concurrent conversations accumulating context over hundreds of turns, the offloading scheduler's block tracking appears to get out of sync — likely a race or bookkeeping error when blocks are evicted and reassigned rapidly. This is replicating the workload from the [llm-d Nemotron agentic serving guide](https://github.com/llm-d/llm-d/blob/main/guides/agentic-serving/nemotron-3-ultra-550b-h200.md) with the same MultiConnector configuration.", "url": "https://github.com/vllm-project/vllm/issues/52170", "createdAt": "2026-08-13T13:46:21Z", "updatedAt": "2026-08-13T13:46:21Z", "timestamp": "2026-08-13T13:46:21Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "bbenshab", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52180", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Improve Pixtral vision attention scaling for batched images", "text": "### Proposal to improve performance Pixtral concatenates the patch sequences for all images in an encoder batch, while attention remains semantically independent per image. On the default installation, where xFormers is unavailable, the model constructs a dense block-diagonal mask and applies SDPA to the combined sequence. The resulting attention work and mask storage scale poorly as more images are batched. This issue proposes routing both Pixtral vision implementations through vLLM's `MMEncoderAttention`, using cumulative sequence lengths to preserve per-image boundaries. This would: - use vLLM's existing multimodal attention backend selection, including FlashAttention, FlashInfer, Torch SDPA, Triton attention, and ROCm AITER; - avoid materializing the dense combined-sequence mask; - remove Pixtral's model-specific optional xFormers path; and - preserve checkpoint compatibility, preprocessing, output ordering, and token counts. Backend-specific packed metadata still needs to be respected. In particular, FlashInfer uses padded sequence-length metadata and transformed cumulative offsets, so the implementation delegates metadata construction to `MMEncoderAttention` rather than assuming the FlashAttention layout. ### Report of performance regression Controlled local benchmark on an NVIDIA DGX Spark (GB10), using BF16, the 24-layer Pixtral HF vision encoder, 840 x 840 images, three warmup iterations, and ten measured iterations. Both variants used the same branch parent, container, model configuration, and identically seeded random weights. | Encoder batch | Unpatched base | Packed FlashAttention | Speedup | | ---: | ---: | ---: | ---: | | 1 | 119.4 ms / 8.37 images/s | 95.9 ms / 10.43 images/s | 1.25x | | 2 | 371.1 ms / 5.39 images/s | 213.4 ms / 9.37 images/s | 1.74x | | 4 | 1187.4 ms / 3.37 images/s | 430.9 ms / 9.28 images/s | 2.76x | Numerical validation against Torch SDPA used the same 24-layer, identically initialized encoder on mixed-size images: | Backend | Max absolute difference | Cosine similarity | | --- | ---: | ---: | | FlashAttention | 0.00220 | 0.999985 | | FlashInfer | 0.00195 | 0.999980 | A two-layer pretrained-weight check produced identical features when the same image was encoded alone or alongside a differently sized image. A full Mistral3 vLLM smoke test also loaded a real checkpoint, selected the requested FlashAttention vision backend, processed two differently sized images, and generated successfully. ### Misc discussion on performance ## Related work and design implications - #40703 migrates Pixtral's current xFormers imports to `mslk` while intentionally preserving the existing attention design. This proposal is materially different and would supersede the need for that model-specific path if accepted. - #38175 tracks full ViT CUDA graph support, and Pixtral/Mistral3 work has recently been claimed there. This proposal does not add encoder CUDA graph support, but metadata preparation may eventually move outside `forward` as part of that work. - #43052 addresses a separate Pixtral/Mistral3 graph-mode failure. Would maintainers prefer Pixtral to adopt the standard `MMEncoderAttention` path, or retain a model-specific `mslk` path? If `MMEncoderAttention` is preferred, should graph-friendly metadata preparation be included here or remain scoped to the CUDA graph work in #38175? ### Your current environment (if you think it is necessary) ```text The output of `python collect_env.py` ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52180", "createdAt": "2026-08-13T14:51:49Z", "updatedAt": "2026-08-13T15:07:31Z", "timestamp": "2026-08-13T15:07:31Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance" ], "author": "oliverholworthy", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:issue:52181", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: FA2 is only supported on devices with compute capability >= 8", "text": "### 🚀 The feature, motivation and pitch I am trying to use the vllm for loading [QWEN3.6 model](https://huggingface.co/Qwen/Qwen3.6-27B) Operating system= rocky linux 9 GPU = 2X Quadro RTX 8000 48 GB As soon as I try to load with the command `vllm serve ./Qwen3.6-27B` there is error below. `(EngineCore pid=1673840) ERROR 08-13 16:42:32 [fa_utils.py:273] Cannot use FA version 2 is not supported due to FA2 is only supported on devices with compute capability >= 8` No matter i use quantized model or the simple, I cannot load. This is bottle neck and this must have some solution to load via some flag or additional parameter. Kindly I need your support and help to use this model on the mentioned GPU. ### Alternatives Tell me the way how to make a quantized version of this above model with vllm which I can load and use on the cuda compute capability 7.5 as my GPU is Quadro RTX 8000 which has power less than 8.0 cuda compute capability ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52181", "createdAt": "2026-08-13T14:52:13Z", "updatedAt": "2026-08-13T16:45:03Z", "timestamp": "2026-08-13T16:45:03Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "feature request", "quantization" ], "author": "Tortoise17", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:24235", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Change the default value of truncate_prompt_tokens in the embedding/rerank/pooling model to -1", "text": "Change the default value of truncate_prompt_tokens in the embedding model to -1，By default, the model is truncated according to its maximum length. ## Purpose The client no longer needs to worry about the maximum length supported by the model and will not report an error if the input text is too long ## Test Plan work wll on bge-m3 ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/24235", "createdAt": "2025-09-04T08:56:46Z", "updatedAt": "2026-08-13T02:16:09Z", "timestamp": "2026-08-13T02:16:09Z", "metrics": { "reactions": 1, "comments": 18 }, "labels": [ "frontend", "needs-rebase", "stale" ], "author": "zhaotyer", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:25135", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Llamas 3.1 405B fp4 changes upstreaming from 355_wip", "text": "Perf is the same: upstream tp1 ``` ============ Serving Benchmark Result ============ Successful requests: 320 Maximum request concurrency: 64 Benchmark duration (s): 413.07 Total input tokens: 326905 Total generated tokens: 327680 Request throughput (req/s): 0.77 Output token throughput (tok/s): 793.27 Total Token throughput (tok/s): 1584.66 ---------------Time to First Token---------------- Mean TTFT (ms): 5240.27 Median TTFT (ms): 5150.59 P99 TTFT (ms): 13542.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.58 Median TPOT (ms): 76.05 P99 TPOT (ms): 79.35 ---------------Inter-token Latency---------------- Mean ITL (ms): 75.58 Median ITL (ms): 66.27 P99 ITL (ms): 73.04 ----------------End-to-end Latency---------------- Mean E2EL (ms): 82558.95 Median E2EL (ms): 82994.12 P99 E2EL (ms): 89413.31 ================================================== ``` 355_wip tp1 ``` ============ Serving Benchmark Result ============ Successful requests: 320 Maximum request concurrency: 64 Benchmark duration (s): 415.51 Total input tokens: 326905 Total generated tokens: 327680 Request throughput (req/s): 0.77 Output token throughput (tok/s): 788.62 Total Token throughput (tok/s): 1575.37 ---------------Time to First Token---------------- Mean TTFT (ms): 5262.51 Median TTFT (ms): 5156.42 P99 TTFT (ms): 14143.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 76.03 Median TPOT (ms): 76.44 P99 TPOT (ms): 79.86 ---------------Inter-token Latency---------------- Mean ITL (ms): 76.03 Median ITL (ms): 66.64 P99 ITL (ms): 69.13 ----------------End-to-end Latency---------------- Mean E2EL (ms): 83044.74 Median E2EL (ms): 83387.81 P99 E2EL (ms): 90413.53 ================================================== ``` Command: ``` HIP_VISIBLE_DEVICES=7 \\ VLLM_DISABLE_COMPILE_CACHE=1 \\ USE_FASTSAFETENSOR=1 \\ SAFETENSORS_FAST_GPU=1 \\ VLLM_USE_V1=1 \\ AMDGCN_USE_BUFFER_OPS=1 \\ TRITON_HIP_ASYNC_COPY_BYPASS_PERMUTE=1 \\ TRITON_HIP_USE_ASYNC_COPY=1 \\ TRITON_HIP_USE_BLOCK_PINGPONG=1 \\ TRITON_HIP_ASYNC_FAST_SWIZZLE=1 \\ VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_MHA=0 \\ VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 \\ VLLM_USE_AITER_UNIFIED_ATTENTION=0 \\ VLLM_ROCM_USE_TRITON_ROPE=1 \\ VLLM_ROCM_USE_AITER_RMSNORM=1 \\ VLLM_ROCM_USE_AITER_FP4_ASM_GEMM=1 \\ vllm serve /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --host localhost \\ --port 30000 \\ --swap-space 64 \\ --disable-log-requests \\ --dtype auto \\ --max-model-len 8192 \\ --tensor-parallel-size 1 \\ --max-num-seqs 64 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.85 \\ --max-seq-len-to-capture 8192 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --max-num-batched-tokens 8192 \\ --compilation-config='{\"pass_config\":{\"enable_attn_fusion\":true,\"enable_noop\":true,\"enable_fusion\":true},\"cudagraph_mode\":\"FULL\",\"custom_ops\":[\"+rms_norm\",\"+silu_and_mul\",\"+quant_fp8\"],\"splitting_ops\":[]}' ``` # Run the client benchmark ``` vllm bench serve \\ --host localhost \\ --port 30000 \\ --model /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --max-concurrency 64 \\ --num-prompts 320 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --ignore-eos ``` Correctness - shows reasonable answers for command: ``` HIP_VISIBLE_DEVICES=7 \\ VLLM_DISABLE_COMPILE_CACHE=1 \\ USE_FASTSAFETENSOR=1 \\ SAFETENSORS_FAST_GPU=1 \\ VLLM_USE_V1=1 \\ VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 \\ AMDGCN_USE_BUFFER_OPS=1 \\ VLLM_USE_AITER_TRITON_ROPE=1 \\ TRITON_HIP_ASYNC_COPY_BYPASS_PERMUTE=1 \\ TRITON_HIP_USE_ASYNC_COPY=1 \\ TRITON_HIP_USE_BLOCK_PINGPONG=1 \\ TRITON_HIP_ASYNC_FAST_SWIZZLE=1 \\ VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_MHA=0 \\ VLLM_ROCM_USE_AITER_RMSNORM=1 \\ VLLM_TRITON_FP4_GEMM_USE_ASM=1 \\ python /data/vllm-scripts/llm_test.py \\ --model /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --dataset-path /data/models/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json \\ --batch-size 32 \\ --swap-space 64 \\ --dtype auto \\ --max-model-len 8192 \\ --tensor-parallel-size 1 \\ --max-num-seqs 1024 \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.92 \\ --max-seq-len-to-capture 8192 \\ --no-enable-prefix-caching \\ --max-num-batched-tokens 8192 ```", "url": "https://github.com/vllm-project/vllm/pull/25135", "createdAt": "2025-09-18T04:18:56Z", "updatedAt": "2026-08-12T17:59:11Z", "timestamp": "2026-08-12T17:59:11Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [ "rocm", "ready", "v1", "llama" ], "author": "maleksan85", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:33763", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add vllm_enable_compile_cache config flag with backward compatibility", "text": "- Add vllm_enable_compile_cache boolean flag to CompilationConfig (defaults to True) - Keep VLLM_DISABLE_COMPILE_CACHE environment variable for backward compatibility - Environment variable overrides config flag when set via model_validator - Update is_compile_cache_enabled() to accept config flag parameter - Update all test files to use config flag instead of environment variable - Update documentation to mention both config flag (preferred) and env var (backward compatibility) This change provides a cleaner API via config while maintaining backward compatibility with existing code using the environment variable. For https://github.com/vllm-project/vllm/issues/29917 ---
- [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/33763", "createdAt": "2026-02-04T04:50:40Z", "updatedAt": "2026-08-13T14:03:27Z", "timestamp": "2026-08-13T14:03:27Z", "metrics": { "reactions": 2, "comments": 19 }, "labels": [ "documentation", "frontend", "ready", "llama", "qwen" ], "author": "elizabetht", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:34143", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Fix] Bump lmcache minimum version to 0.3.11", "text": "The `get_kv_events()` API used in `LMCacheConnectorV1` is only available starting from lmcache 0.3.11. Bump the minimum version requirement to prevent runtime AttributeError. ## Purpose - Bumps the minimum `lmcache` version from `>= 0.3.9` to `>= 0.3.11` in `requirements/kv_connectors.txt` - The `get_kv_events()` API used in `LMCacheConnectorV1.get_kv_connector_kv_cache_events()` is only available starting from lmcache 0.3.11, causing a runtime `AttributeError` with older versions Fixes https://github.com/vllm-project/vllm/issues/34136 ## Test Plan N/A ## Test Result N/A ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/34143", "createdAt": "2026-02-09T14:56:35Z", "updatedAt": "2026-08-13T02:15:13Z", "timestamp": "2026-08-13T02:15:13Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "needs-rebase", "ci/build", "stale", "kv-connector" ], "author": "MohanKumar21", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:34354", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix step3p5 tool parser and unnecessary unstreamed tool args in serving.", "text": "## Purpose The step3.5 tool parser and test cases have been modified, resulting in a more stable parser. In `vllm/entrypoints/openai/chat_completion/serving.py`, some parsers do not require checking unstreamed tool arguments. For example: **qwen3coder_tool_parser**: does not maintain variables for checking unstreamed tool arguments (`streamed_args_for_tool`), may cause out of index error. **qwen3xml_tool_parser**: has some bugs during maintenance, causing duplicate parameter sending. **step3p5_tool_parser**: has the same problem as qwen3xml. The added skip logic only applies to some parsers that already have the problem and will not affect other parsers. ## Test Plan add test `/data/open/vllm/tests/tool_parsers/test_step3p5_tool_parser.py` ## Test Result test_step3p5_tool_parser: All passed. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/34354", "createdAt": "2026-02-11T16:26:45Z", "updatedAt": "2026-08-13T02:15:08Z", "timestamp": "2026-08-13T02:15:08Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "frontend", "needs-rebase", "stale", "tool-calling", "qwen" ], "author": "mariohong128", "state": "closed", "assignees": [ "chaunceyjiang" ] }, { "id": "github:vllm-project/vllm:pull_request:35305", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Fix MoE g_idx params causing ValueError with actorder=null AWQ models", "text": "## Summary - Fix `CompressedTensorsWNA16MarlinMoEMethod.create_weights()` to only register `g_idx` parameters when `self.actorder` is set, preventing `ValueError` from `track_weights_loading()` when loading compressed-tensors AWQ MoE models with `actorder=null` - Fix `CompressedTensorsWNA16MoEMethod.create_weights()` to remove unreachable `g_idx` registration (class asserts `actorder != \"group\"`) - Fix `process_weights_after_loading()` to derive `num_experts`/`device` from `w13_weight_packed` instead of `w13_weight_g_idx` Fixes #35303 ## Motivation `CompressedTensorsWNA16MarlinMoEMethod.create_weights()` unconditionally registers `w13_weight_g_idx`, `w2_weight_g_idx`, `w13_g_idx_sort_indices`, and `w2_g_idx_sort_indices` as `nn.Parameter`. When `actorder` is `null` (no activation ordering), the checkpoint does not contain these tensors. The strict weight validation in `DefaultModelLoader.track_weights_loading()` then raises: ``` ValueError: Following weights were not initialized from checkpoint: { 'language_model.model.layers.0.mlp.experts.w13_weight_g_idx', 'language_model.model.layers.0.mlp.experts.w2_weight_g_idx', ... (all MoE layers) } ``` This affects any compressed-tensors AWQ MoE model with `actorder=null`, such as `cpatonn/Qwen3-VL-30B-A3B-Instruct-AWQ-4bit`. These models load successfully on v0.15.1 but crash on the v0.16.0 nightlies. The `process_weights_after_loading()` method already handles `actorder != \"group\"` by replacing these with empty tensors — but validation runs *before* that method, so the error fires first. A similar bug was previously fixed for non-MoE models (Issue #5088 / PR #5108 for GPTQ Marlin `g_idx_sort_indices`), but the MoE code path was not covered. ## Changes ### `CompressedTensorsWNA16MarlinMoEMethod` (Marlin backend) 1. **`create_weights()`**: Wrap the four `g_idx` `register_parameter()` calls in `if self.actorder:` so they are only registered when the checkpoint actually contains them. 2. **`process_weights_after_loading()`**: - Get `num_experts`/`device` from `layer.w13_weight_packed` (always present) instead of `layer.w13_weight_g_idx` (absent when actorder is null). - In the `else` branch (no actorder), use plain `torch.empty()` tensors instead of `nn.Parameter` for the empty g_idx placeholders, since they don't need gradient tracking or checkpoint loading. ### `CompressedTensorsWNA16MoEMethod` (non-Marlin backend) 3. **`create_weights()`**: Remove g_idx parameter registration entirely. This class asserts `weight_quant.actorder != \"group\"` in `__init__`, so g_idx tensors are never present in the checkpoint and should never be registered. ## Test plan - [x] Verified fix loads and runs `cpatonn/Qwen3-VL-30B-A3B-Instruct-AWQ-4bit` successfully on v0.16.0rc2.dev472 (CUDA 13.0, RTX 5090) - [ ] Existing compressed-tensors MoE tests should still pass (actorder=group path unchanged) - [ ] Models with `actorder=\"group\"` should still correctly load g_idx from checkpoint 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/35305", "createdAt": "2026-02-25T15:52:21Z", "updatedAt": "2026-08-13T02:14:55Z", "timestamp": "2026-08-13T02:14:55Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "jhsmith409", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:36951", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Add persistent cache mounts and fix test download paths", "text": "This PR avoids cold network downloads during AMD CI test collection and execution. The immediate motivation was a Buildkite multimodal shard spending time downloading before the selected tests even ran: https://buildkite.com/vllm/ci/builds/76781/canvas?jid=019f3b96-c3dd-4f37-98b2-0599cfe73733&tab=output. The biggest issue was `test_phi4mm.py` resolving the 12.9 GB Phi-4 multimodal snapshot at module import time, even when pytest later deselected that file. This PR moves that setup into lazy fixtures, shares the Phi-4 model and LoRA paths across the relevant tests, and removes other top-level tokenizer/config loads that made collection expensive or network-dependent. Concretely, this adds persistent AMD CI cache mounts, routes direct test downloads through `VLLM_TEST_CACHE`, respects `HF_HOME` for cache-sensitive tests, and removes explicit cache paths that bypass the shared Hugging Face cache. It also covers files that do not naturally live in the HF cache, such as public vLLM assets, small direct-download fixtures, and the `tiktoken-rs` runtime vocab cache. AMD CI sets `VLLM_TEST_CACHE_ONLY=1`, so these helpers fail clearly on a cache miss instead of silently reaching out to the network during a test job. Direct URL helpers only treat non-empty files as cache hits and use atomic writes/copies so partially populated cache entries are not reused.", "url": "https://github.com/vllm-project/vllm/pull/36951", "createdAt": "2026-03-13T04:56:43Z", "updatedAt": "2026-08-13T07:16:28Z", "timestamp": "2026-08-13T07:16:28Z", "metrics": { "reactions": 4, "comments": 9 }, "labels": [ "rocm", "ci/build", "v1", "multi-modality", "tool-calling", "qwen", "gpt-oss", "kv-connector" ], "author": "AndreasKaratzas", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:36979", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[refactor] Refactor SpeculativeConfig for speculative method extensibility", "text": "## Purpose Refactor `SpeculativeConfig:__post_init__` from a monolithic method into a dispatcher to family specific init methods. The current `__post_init__` interleaves the config logic for all speculative methods into a single nested if/elif chain. This makes it hard to trace a single method's setup without reading the entire function, and adding a new speculative method requires changes in multiple branches. The goal of this refactor is for each speculative 'family' to have its own method, allowing us to more easily add new speculative methods in the future: `_init_ngram_family` -> ngram/ngram_gpu (no model loading) `_init_suffix_family` -> suffix (no model loading) `_init_mtp_family` -> mtp (reuse of target model weights) `_init_eagle_family` -> eagle/eagle3/extract_hidden_states `_init_draft_model_family` -> draft_model/medusa/mlp_speculator The code for normalizing method names, autodetecting based on model name and the shared configuration for model-loading families have also been split into seperate methods. There are some behavioural changes that are all either minor or bug fixes: 1. deepseek_v32 enforce_eager=True Old: enforce_eager=True only ran when model=True New: always gets set in _init_mtp_family() 2. MTP with method='mtp', model=None and num_speculative_tokens=None Old: MTP init skipped on line 363 New: Use n_predict if available in `_init_mtp_family` -> `_init_model_config_tail` 3. Medusa method but mismatched hf_config Old: If user set method='medusa' but hf_config.model_type was not 'medusa', NotImplementedError is raised New: Gets accepted with warning (trusting user input) ## Test Plan `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/test_config.py -v` `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/config/test_model_arch_config.py -v` `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/v1/e2e/test_spec_decode.py -v` ## Test Result test_config.py -> All passed. test_model_arch_config.py -> All passed. test_spec_decode.py -> All passed (multi-GPU, unsupported backends and flaky tests automatically skipped. PR doesn't affect runtime so should not be an issue)", "url": "https://github.com/vllm-project/vllm/pull/36979", "createdAt": "2026-03-13T12:59:39Z", "updatedAt": "2026-08-13T02:14:25Z", "timestamp": "2026-08-13T02:14:25Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "needs-rebase", "stale", "verified" ], "author": "TQCB", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:37227", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Use list.extend() over append loops in FlatLogprobs + minor hot-path cleanups", "text": "Minor Python-level performance optimizations in hot paths: 1. **FlatLogprobs.append_fast**: Replace per-item append() loop with list.extend() — ~1.37x speedup on the per-token logprobs path. 2. **common_prefix**: Compare elements by index instead of creating slice copies — ~9.4x speedup on long shared prefixes. 3. **FlatLogprobs.__iter__**: Use self[i] instead of self.__getitem__(i), remove redundant range(0, ...). 4. **OutputProcessor.propagate_error**: Use dict.values() instead of dict.items() when key is unused — ~1.3x speedup. All changes are semantically identical refactors. Existing tests pass.", "url": "https://github.com/vllm-project/vllm/pull/37227", "createdAt": "2026-03-16T20:49:32Z", "updatedAt": "2026-08-13T02:14:16Z", "timestamp": "2026-08-13T02:14:16Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "stale", "v1" ], "author": "vaibhavhariram", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:37383", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Add comprehensive --speculative-config documentation", "text": "## Summary - Adds a new `--speculative-config` Reference section to the speculative decoding documentation (`docs/features/speculative_decoding/README.md`) that enumerates all user-facing configuration keys accepted by `--speculative-config` (CLI) and the `speculative_config` Python dict - Documents every key's type, default value, valid ranges/values, and which speculative method each key applies to - Includes usage examples for CLI and Python API, plus practical examples for draft model with quantization, EAGLE with tree speculation, n-gram with custom window, and suffix decoding with tuned parameters - All information is derived directly from `vllm/config/speculative.py` (`SpeculativeConfig` dataclass) - Fixed `enforce_eager` description: it controls CUDA graph capture for the speculative model (not the draft model's `ModelConfig.enforce_eager`), and defaults to allowing CUDA graphs when `None` Fixes #35573 ## Test plan - [x] Cross-reference each documented key against `vllm/config/speculative.py` to confirm accuracy - [x] Cross-reference `enforce_eager` usage in `vllm/v1/worker/gpu_model_runner.py` and `vllm/v1/spec_decode/eagle.py` to confirm description - [ ] Verify the documentation renders correctly with `mkdocs serve` - [ ] Confirm no regressions in existing documentation sections (Lossless guarantees, Resources, etc.)", "url": "https://github.com/vllm-project/vllm/pull/37383", "createdAt": "2026-03-18T04:23:51Z", "updatedAt": "2026-08-13T08:24:17Z", "timestamp": "2026-08-13T08:24:17Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "documentation", "needs-rebase" ], "author": "karanb192", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:37390", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Quark OCP-MX W4A6 support: dequant dtype + apply_weights", "text": " Fixes two issues preventing Quark W4A6 models (MXFP4 weights / MXFP6 activations, e.g. `ziliangpeng/DeepSeek-V3-Quark-MXFP4-v4-w4a6`) from running on ROCm: 1. **MoE QuantType crash** (`rocm_aiter_moe.py`): W4A6 models set `weight_quant_dtype=\"mxfp4\"` but not the `use_mxfp4_w4a4/w4a16` flags, causing AITER MoE to receive `QuantType.NO` and crash with \"Unsupported kernel config for moe heuristic dispatch\". Fixed by adding a fallback check on `weight_quant_dtype`. This is the caller-side fix suggested by @valarLip in ROCm/aiter#2457. 2. **Emulate path byte tensor crash** (`quark_ocp_mx.py`): In the emulation path (used when native MXFP6 kernels are not available), input `x` can arrive as `uint8` from upstream quantized tensor flow. This causes `F.linear(uint8, uint8)` to crash and the Quark dequant kernel to reject the dtype. Fixed with defensive dtype handling: non-floating-point inputs are cast to `bfloat16` before dequantization. The activation QDQ path (`quant_dequant_func`) is preserved for accuracy. ### Test Plan **Hardware:** AMD Instinct MI350X / MI355X (gfx950), 8xGPU **Model:** `ziliangpeng/DeepSeek-V3-Quark-MXFP4-v4-w4a6` **1. Build and install vLLM with this PR:** ```bash # Start ROCm container docker run -it --network=host --device=/dev/kfd --device=/dev/dri \\ --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \\ -v /path/to/models:/workspace/models \\ vllm/vllm-openai-rocm:latest bash # Inside container: clone and install the PR branch git clone https://github.com/vecheruk-amd/vllm.git -b fix/quark-w4a6-mxfp4-compat cd vllm && pip install -e . ``` **2. Set environment variables:** ```bash export VLLM_ROCM_USE_AITER=1 ``` **3. Serve the model (TP=8, eager mode):** ```bash vllm serve /workspace/models/DeepSeek-V3-Quark-MXFP4-v4-w4a6 \\ --tensor-parallel-size 8 \\ --block-size 1 \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.90 \\ --max-model-len 32000 \\ --enforce-eager \\ --port 8000 ``` **4. Test inference:** ```bash curl http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{\"model\": \"/workspace/models/DeepSeek-V3-Quark-MXFP4-v4-w4a6\", \"messages\": [{\"role\":\"user\",\"content\":\"Explain quantum computing\"}], \"max_tokens\": 256}' ``` **Without this PR:** Step 3 crashes with \"Unsupported kernel config for moe heuristic dispatch\" (QuantType issue). **With this PR:** Both short and long generation produce correct, coherent output. ### Test Result Both short and long generation produce correct, coherent output with `--enforce-eager`. ### Known Issue CUDA graph mode segfaults during decode with the W4A6 emulation path on PyTorch 2.10+ / ROCm 7.2+ (works on v0.18.0 / PyTorch 2.9 / ROCm 7.0). `--enforce-eager` is the workaround. A separate issue will be filed. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/37390", "createdAt": "2026-03-18T06:05:13Z", "updatedAt": "2026-08-13T02:14:07Z", "timestamp": "2026-08-13T02:14:07Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "stale" ], "author": "vecheruk-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:37925", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] CUDA Checkpoint/Restore — Phase 2: Engine/Executor/API Integration", "text": "## Summary Phase 2 of CUDA checkpoint/restore (RFC #34303). Wires `suspend`/`resume` through the full engine stack and exposes REST endpoints. **Depends on:** #37921 (Phase 1: C extension, Python wrapper, worker methods) ### Changes - **Executor** (`vllm/v1/executor/abstract.py`): `suspend()`/`resume()` via `collective_rpc`, `is_suspended` state tracking - **Engine Core** (`vllm/v1/engine/core.py`): `suspend(mode)`/`resume()`/`is_checkpoint_suspended()` with scheduler pausing - **Protocol** (`vllm/engine/protocol.py`): Abstract methods for checkpoint operations - **AsyncLLM** (`vllm/v1/engine/async_llm.py`): Async implementations - **Core Client** (`vllm/v1/engine/core_client.py`): Methods in all 4 client classes (base, InprocClient, SyncMPClient, AsyncMPClient) - **REST API** (`vllm/entrypoints/serve/checkpoint/`): `POST /suspend`, `POST /resume`, `GET /is_suspended` (dev mode only) - **Router registration** (`vllm/entrypoints/serve/__init__.py`) ### Architecture Follows the exact same wiring pattern as existing `sleep`/`wake_up`/`is_sleeping`: ``` API Router → AsyncLLM → CoreClient → EngineCore → Executor → Worker ``` ## Test plan - REST endpoints gated behind `VLLM_SERVER_DEV_MODE` (same as sleep) - Integration tests to be added in follow-up - `ruff check` and `mypy` pass on all changed files ## Notes - AI assistance was used (Claude). All changes reviewed by human submitter. - Not duplicating any existing PR — this is new functionality per RFC #34303. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/37925", "createdAt": "2026-03-23T20:24:34Z", "updatedAt": "2026-08-13T13:11:47Z", "timestamp": "2026-08-13T13:11:47Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "frontend", "v1", "nvidia" ], "author": "elizabetht", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:38484", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Add SM121 (DGX Spark / GB10) to published build targets", "text": "## Summary - Add compute capability 12.1 (SM121) to `TORCH_CUDA_ARCH_LIST` in all build configurations: release wheel CI, Dockerfile, docker-bake.hcl, versions.json, and FlashInfer AOT build - `CMakeLists.txt` already lists 12.1 in `CUDA_SUPPORTED_ARCHS` for CUDA 12.8+ and 13.0+, and all kernel-specific arch lists (CUTLASS, Marlin FP8, MoE, NVFP4) already include 12.1 — but the build scripts never passed 12.1 in `TORCH_CUDA_ARCH_LIST`, so the intersection filtering excluded it from all published artifacts - This is the minimal change needed so that published wheels and Docker images include SM121 kernels for DGX Spark / GB10 users out of the box ## Context The NVIDIA GB10 GPU in DGX Spark reports compute capability 12.1 (SM121). Without native kernels in published builds, users must either build from source or rely on PTX JIT compilation via the 12.0+PTX forward-compat path (slower first-run, no guarantee of correctness for all kernels). **Upstream PyTorch fix:** The CMake auto-detection bug that corrupts `12.1` → `12.1(2.0)` when building PyTorch from source on SM121 hardware has fixes in review: [pytorch/pytorch#174065](https://github.com/pytorch/pytorch/pull/174065) and [pytorch/pytorch#173754](https://github.com/pytorch/pytorch/pull/173754). I independently reproduced and verified both fixes on DGX Spark hardware (see comments on those PRs). PyTorch cu130 aarch64 nightly wheels already ship with `12.0+PTX` which covers SM121 via JIT. **Relationship to existing PRs:** [#31740](https://github.com/vllm-project/vllm/pull/31740) includes these build config changes among 27 files of broader SM121 platform support, but has been `CONFLICTING` since January. This PR extracts just the build target additions — 5 files, 6 lines changed — to unblock SM121 in published artifacts independently. Addresses #36821. ## Test evidence Tested on NVIDIA DGX Spark: | Component | Value | |-----------|-------| | GPU | NVIDIA GB10 | | Compute capability | 12.1 (SM121) | | Architecture | aarch64 | | OS | Ubuntu 24.04.4 LTS | | CUDA | 13.0, V13.0.88 | | Driver | 590.48.01 | Confirmed that `CMakeLists.txt` already supports 12.1 in `CUDA_SUPPORTED_ARCHS` and all kernel-specific arch lists (`SCALED_MM_ARCHS`, `FP4_ARCHS`, `MLA_ARCHS`, `CUTLASS_MOE_DATA_ARCHS`, `MARLIN_FP8_ARCHS`, `MARLIN_MOE_FP8_ARCHS`). The only gap was the `TORCH_CUDA_ARCH_LIST` in build scripts. ## Files changed | File | Change | |------|--------| | `.github/workflows/scripts/build.sh` | Add `10.0 12.0 12.1+PTX` to release wheel arch list | | `docker/Dockerfile` | Add `12.1` to both `torch_cuda_arch_list` ARGs (lines 191, 804) | | `docker/docker-bake.hcl` | Add `12.0 12.1` to default arch list | | `docker/versions.json` | Match Dockerfile ARG update | | `tools/flashinfer-build.sh` | Add `12.1` to CUDA 13.0+ FlashInfer AOT arch list | ## Test plan - [ ] Verify `docker buildx bake --print` shows 12.1 in resolved `torch_cuda_arch_list` - [ ] Verify wheel build with CUDA 13.0 includes SM121 kernels (`cuobjdump` check) - [ ] Run vLLM inference on DGX Spark with published image (no `--enforce-eager` needed)", "url": "https://github.com/vllm-project/vllm/pull/38484", "createdAt": "2026-03-29T18:15:26Z", "updatedAt": "2026-08-13T17:28:39Z", "timestamp": "2026-08-13T17:28:39Z", "metrics": { "reactions": 1, "comments": 8 }, "labels": [ "needs-rebase", "ci/build", "nvidia" ], "author": "JCorners68", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:39413", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Responses API] Fix truncation: auto returning 400 instead of truncating", "text": "## Summary Fixes #38132 When using the Responses API with `truncation: \"auto\"`, sending input that exceeds the model's context window returns a 400 error instead of truncating the input to fit. **Root cause**: The Harmony code path (used by `gpt-oss` models) bypasses the standard renderer/`TokenizeParams` pipeline and directly tokenizes via `render_for_completion()`, which never applied truncation. The full oversized prompt was sent to the engine validation, triggering a 400 error. **Fix**: Added `_apply_harmony_truncation()` to perform left-side truncation on the Harmony path, keeping the most recent conversation tokens and dropping the oldest to fit within `max_model_len - max_output_tokens`. This is applied in: - Initial request processing (`_make_request_with_harmony`) - Multi-turn tool-call loops (`_generate_with_builtin_tools`) The non-Harmony path already works correctly through `build_tok_params` setting `truncate_prompt_tokens=-1`. ## Changes - **`vllm/entrypoints/openai/responses/serving.py`**: Added `_apply_harmony_truncation()` method and integrated it into both the initial Harmony request path and the multi-turn Harmony generation loop. Passed `request` to `_generate_with_builtin_tools` to enable truncation in multi-turn. - **`tests/entrypoints/openai/responses/test_serving_responses.py`**: Added `TestApplyHarmonyTruncation` test class with 6 test cases covering disabled truncation, auto truncation from left, no-truncation-needed, null max_output_tokens, exact boundary, and one-over-boundary. - **`tests/entrypoints/openai/responses/test_sampling_params.py`**: Added `TestResponsesRequestTruncation` test class with 4 test cases verifying `build_tok_params` correctly maps the `truncation` parameter. ## Test plan - [x] Unit tests for `_apply_harmony_truncation()` covering all edge cases - [x] Unit tests for `build_tok_params` truncation parameter mapping - [ ] Manual test with oversized input on gpt-oss model with `truncation: \"auto\"` (requires GPU) - [ ] Verify non-Harmony models still work with `truncation: \"auto\"` (already handled by existing `TokenizeParams` pipeline) 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/39413", "timestamp": "2026-08-12T16:39:29Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "frontend" ], "author": "Ricardo-M-L", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:39448", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "AMD remove sync visible devices", "text": "## Purpose Use HIP_VISIBLE_DEVICES and device control environment variable on ROCm platform. Aligning with Pytorch and Ray frameworks. Syncing with CUDA_VISIBLE_DEVICES is no longer necessary ## Test Plan vllm serve with ray backend `vllm serve Qwen/Qwen2-7B-Instruct --tensor-parallel-size 4 --pipeline-parallel-size 2 --distributed-executor-backend ray --gpu-memory-utilization .9 --max-model-len 8192 --max-num-seqs 2000 --max-num-batched-tokens 131072` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/39448", "createdAt": "2026-04-09T19:53:00Z", "updatedAt": "2026-08-12T19:35:41Z", "timestamp": "2026-08-12T19:35:41Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "rocm", "verified" ], "author": "vickytsang", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:39452", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Use physical device id for get_device_uuid", "text": "This PR updates the get_device_uuid to translate the device_id parameter to a physical gpu id before retrieving the gpu handle. `get_device_uuid` is used in the Verl application PPO and Fully Async uses cases which executes the code below when the vllm backend is being used: def get_device_uuid(device_id: int) -> str: from vllm.platforms import current_platform # Convert torch.npu.current_device to its corresponding ASCEND_RT_VISIBLE_DEVICES. if is_npu_available: if os.getenv(\"ASCEND_RT_VISIBLE_DEVICES\") is not None: npu_visible_devices = os.environ[\"ASCEND_RT_VISIBLE_DEVICES\"].split(\",\") assert device_id < len(npu_visible_devices), f\"device_id {device_id} must less than {npu_visible_devices}\" return \"NPU-\" + npu_visible_devices[device_id] else: return f\"NPU-{device_id}\" else: return current_platform.get_device_uuid(device_id)", "url": "https://github.com/vllm-project/vllm/pull/39452", "createdAt": "2026-04-09T21:01:17Z", "updatedAt": "2026-08-13T02:13:53Z", "timestamp": "2026-08-13T02:13:53Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "rocm", "stale" ], "author": "tmm77", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:40337", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring", "text": "## Summary Replaces the vanilla padded-bmm MaxSim path (introduced in #35330, re-enabled on GPU in #38620) with vendored flash-maxsim Triton kernels for ColBERT / ColPali document scoring. Addresses feature request #38282. ## Why this isn't a duplicate - Issue #38282 explicitly asks for this integration; no other open PR addresses it (searched `is:pr is:open flash maxsim`). - PR #38620 re-enabled GPU MaxSim infrastructure but explicitly deferred the kernel integration (\"only cleans up the API to make integration easier\") — this PR is the promised follow-up. - PR #35330 introduced the vanilla `compute_maxsim_score_batched` that this PR replaces. ## Approach Three scoring paths with automatic fallback: 1. **Zero-copy** (default for GPU, d ≥ 16, no chunked prefill, default pooling params): - Project the full `hidden_states` once via `TokenEmbeddingPoolerHead.project_batch()`. - Each doc request's pooler output is a `[L, d]` view into `projected_batch` — no copy, no extra allocation. - `flash_maxsim_rerank_direct` reads each doc at `(offset, length)` directly from the projected tensor. - The `[B, Lq, Ld]` score matrix is never materialized; the fused kernel tiles through SRAM. 2. **Flash-packed** (fallback when zerocopy disabled or pooling params use matryoshka / `use_activation=False`): - `torch.cat` all docs once, call `flash_maxsim_packed` with `cu_seqlens`. - No per-doc padding; kernel skips padding tokens via `cu_seqlens`. 3. **Vanilla** (fallback for CPU tensors, embedding dim < 16 (Triton `tl.dot` minimum), or `VLLM_FORCE_VANILLA_MAXSIM=1`): - Original sub-batched padded-bmm — unchanged. Triton autotune keys are bucketed (`{32, 64, 128, 256, 512, 1024, 2048, 4096}` for Lq/Ld, `next_pow2(d)` for embed dim). `LateInteractionRunner.__init__` pre-compiles 72 bucket combinations so no autotune fires on first request; failures during warmup are logged and cleanup is in `finally`. ## Files **New** (vendored Triton kernels — `vllm/v1/pool/flash_maxsim/`): - `flash_maxsim.py` — main forward kernel (hardware-specific autotune configs) - `flash_maxsim_rerank.py` — zero-copy rerank kernel (reads scattered offsets) - `flash_maxsim_varlen.py` — packed cu_seqlens kernel - `flash_maxsim_advanced.py` — q-reuse / split-K variants - `__init__.py` **Modified**: - `vllm/v1/pool/late_interaction.py` — `compute_maxsim_score_batched` dispatches to flash when available - `vllm/v1/worker/gpu/pool/late_interaction_runner.py` — kernel warmup, `_score_zerocopy`, `has_pending_docs` - `vllm/v1/worker/gpu_model_runner.py` — zerocopy branch in `_pool()` with fallback on chunked prefill / matryoshka / `use_activation=False` - `vllm/model_executor/layers/pooler/tokwise/heads.py` — `project_batch()` method (fp16 matmul, fp32 cast of small output) **Tests / demos**: - `tests/v1/worker/test_late_interaction_runner.py` — updated to d=32 (Triton requires K≥16) - `tests/v1/worker/bench_flash_maxsim.py` — detailed kernel benchmark - `tests/v1/worker/demo_flash_maxsim.py` — copy-paste kernel/memory comparison - `tests/v1/worker/demo_oom_resilience.py` — live server OOM demo ## Environment toggles - `VLLM_DISABLE_ZEROCOPY=1` — disable zero-copy, use flash-packed - `VLLM_FORCE_VANILLA_MAXSIM=1` — disable flash entirely, use vanilla bmm - `VLLM_FLASH_MAXSIM_WARMUP_D=` — extend warmup to an extra embedding dim ## Results (A100 80GB, ColBERT) ### Kernel-level on variable-length docs | Workload | Vanilla | Flash-Packed | Zero-Copy | |----------|---------|-------------|-----------| | B=1K, Ld=10-100 | 42.6ms | 0.58ms | **0.05ms** (~788×) | | B=5K, Ld=10-100 | 220.3ms | 2.16ms | **0.08ms** (~2,830×) | | B=10K, Ld=10-100 | 437.6ms | 4.04ms | **0.14ms** (~3,034×) | | B=10K, Ld=50-500 | 472.9ms | 4.22ms | **0.53ms** (~892×) | | ColPali B=1K, Ld≈1030 | 40.4ms | 0.72ms | **0.22ms** (~184×) | Most of vanilla's time (>90%) on varlen workloads is Python-side padding / fp32 cast / allocation, not bmm. The fused kernel eliminates that bookkeeping. ### E2E `/v1/score` (real embeddings, concurrent clients, best-of-3) | Docs/req | Conc | Flash TPS | Vanilla TPS | ΔTput | ΔP95 | |----------|------|-----------|-------------|-------|------| | 500 | 8 | 6.2/s | 5.2/s | **+19%** | **+19%** | | 500 | 16 | 6.0/s | 5.2/s | **+17%** | +7% | | 1000 | 4 | 3.0/s | 2.6/s | **+16%** | +13% | | 2000 | 4 | 1.6/s | 1.3/s | **+23%** | **+24%** | | 5000 | 1 | 0.5/s | 0.4/s | **+22%** | +18% | | 5000 | 2 | 0.6/s | 0.5/s | **+15%** | +19% | ### Correctness - Kernel vs fp32 bmm: `max_abs_err = 4e-6` - E2E 5K real docs: `max_abs_diff < 0.001`, zero pairs > 0.001 off, **top-3 rankings identical** Small tail-ranking noise at positions 4-5 reflects fp16 tensor-core nondeterminism on scores within 5e-4 of each other. ## Test plan - [x] `pytest tests/v1/worker/test_late_interaction_runner.py -v` → 4 passed - [x] `python tests/v1/worker/demo_flash_maxsim.py` → kernel speedups & memory savings displayed - [x] `python tests/v1/worker/bench_flash_maxsim.py` → full kernel benchmark - [x] 5K real ColBERT docs via `/v1/score` with `VLLM_FORCE_VANILLA_MAXSIM` vs flash → score parity verified - [x] `pre-commit run` ruff clean on all changed files - [ ] CI on `main` ## Notes for reviewers - The zerocopy path falls back to the normal pooler when `pooling_params.dimensions` is set (matryoshka) or `use_activation=False`, because `project_batch` normalizes before truncation — matryoshka would produce non-unit vectors. Falls back on chunked prefill for the same reason (partial cache). - `flash_maxsim_advanced.py` (q-reuse / split-K variants) is vendored but not dispatched from the default path. Kept for potential follow-ups. - Warmup takes ~160s cold (first launch on a host, no Triton disk cache) and ~15s warm; model load is ~80s either way. ## AI assistance Development of this PR used AI assistance (Claude). Every changed line was reviewed by the submitter; benchmarks were run end-to-end by the submitter against both paths on real A100 hardware with real ColBERT embeddings.", "url": "https://github.com/vllm-project/vllm/pull/40337", "createdAt": "2026-04-20T08:50:30Z", "updatedAt": "2026-08-13T17:17:58Z", "timestamp": "2026-08-13T17:17:58Z", "metrics": { "reactions": 3, "comments": 30 }, "labels": [ "frontend", "v1", "mrv2", "verified" ], "author": "roipony", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:40704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunner V2] Speculative Decoding NGram GPU Implementations", "text": "## Purpose 1. Added a new NGram GPU speculator. The main feature is a new implementation at: `vllm/v1/worker/gpu/spec_decode/ngram/speculator.py`, similiar to https://github.com/vllm-project/vllm/pull/29184. 2. Updated request state storage for NGram GPU. `vllm/v1/worker/gpu/states.py` `vllm/v1/worker/gpu/model_runner.py` This changes how RequestState is initialized so that all_token_ids can stay densely resident on GPU instead of defaulting to UVA when ngram_gpu is active. As discussed in #29184, the new n-gram speculator repeatedly scans active request token history, doing that from GPU-resident dense storage is much more appropriate than pulling through UVA-backed memory, this is a performance-oriented architectural change supporting the new feature. The `model_runner.py` also injects req_states into speculators that need direct access to the persistent token store. 3. Added variable-length draft token plumbing Several files were updated to support draft proposals where different requests may have different numbers of valid draft tokens: `vllm/v1/outputs.py` `vllm/v1/worker/gpu/spec_decode/utils.py` `vllm/v1/core/sched/scheduler.py` `vllm/v1/engine/core.py` `vllm/v1/worker/gpu/model_runner.py` DraftTokenIds now includes: `num_valid_draft_tokens: list[int] | None`. Scheduler logic now truncates speculative tokens based on `num_valid_draft_tokens`. `EngineCore adds _maybe_update_async_draft_token_ids()` to consume draft metadata from async execution and update scheduler state at the right time. ## Test Plan vllm bench cmd: ``` vllm bench serve \\ --port 8000 \\ --backend vllm \\ --model Qwen3-8B \\ --endpoint /v1/completions \\ --dataset-name sonnet \\ --dataset-path sonnet.txt \\ --max-concurrency 128 \\ --sonnet-input-len 128 \\ --sonnet-output-len 100 \\ --sonnet-prefix-len 10 \\ --num-prompts 256 \\ --ignore-eos \\ --percentile-metrics \"ttft,tpot,itl,e2el\" \\ --seed 1234 ``` ## Test Result Async NGram GPU V1 results: ``` ============ Serving Benchmark Result ============ Successful requests: 256 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 10.42 Total input tokens: 31693 Total generated tokens: 25600 Request throughput (req/s): 24.56 Output token throughput (tok/s): 2456.33 Peak output token throughput (tok/s): 4277.00 Peak concurrent requests: 186.00 Total token throughput (tok/s): 5497.29 ---------------Time to First Token---------------- Mean TTFT (ms): 1348.49 Median TTFT (ms): 1083.36 P99 TTFT (ms): 2778.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 34.86 Median TPOT (ms): 34.45 P99 TPOT (ms): 51.03 ---------------Inter-token Latency---------------- Mean ITL (ms): 40.09 Median ITL (ms): 30.44 P99 ITL (ms): 236.84 ----------------End-to-end Latency---------------- Mean E2EL (ms): 4799.33 Median E2EL (ms): 4419.59 P99 E2EL (ms): 7136.50 ---------------Speculative Decoding--------------- Acceptance rate (%): 5.16 Acceptance length: 1.15 Drafts: 22038 Draft tokens: 66114 Accepted tokens: 3409 Per-position acceptance (%): Position 0: 6.27 Position 1: 5.34 Position 2: 3.86 ================================================== ``` Async NGram GPU V2 results: ``` ============ Serving Benchmark Result ============ Successful requests: 256 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 8.41 Total input tokens: 31693 Total generated tokens: 25600 Request throughput (req/s): 30.43 Output token throughput (tok/s): 3042.78 Peak output token throughput (tok/s): 4471.00 Peak concurrent requests: 187.00 Total token throughput (tok/s): 6809.77 ---------------Time to First Token---------------- Mean TTFT (ms): 980.36 Median TTFT (ms): 713.87 P99 TTFT (ms): 1685.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 31.17 Median TPOT (ms): 31.34 P99 TPOT (ms): 42.63 ---------------Inter-token Latency---------------- Mean ITL (ms): 35.77 Median ITL (ms): 28.55 P99 ITL (ms): 154.67 ----------------End-to-end Latency---------------- Mean E2EL (ms): 4065.85 Median E2EL (ms): 3954.04 P99 E2EL (ms): 5862.69 ---------------Speculative Decoding--------------- Acceptance rate (%): 52.64 Acceptance length: 2.58 Drafts: 2136 Draft tokens: 6408 Accepted tokens: 3373 Per-position acceptance (%): Position 0: 64.04 Position 1: 54.63 Position 2: 39.23 ================================================== ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/40704", "createdAt": "2026-04-23T12:42:36Z", "updatedAt": "2026-08-13T03:58:56Z", "timestamp": "2026-08-13T03:58:56Z", "metrics": { "reactions": 3, "comments": 21 }, "labels": [ "speculative-decoding", "ready", "ci/build", "v1", "mrv2" ], "author": "PatchouliTIS", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:41412", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add MixtureOfExperts support for PhiMoE", "text": "This PR adds EPLB (Expert-Parallel Load Balancing) support for PhiMoE by implementing the `MixtureOfExperts` protocol. ## Changes - Add `PhiMoEMixtureOfExperts` mixin class with `extract_moe_parameters()` and `update_physical_experts_metadata()` - Thread `enable_eplb` from model config through `PhiMoEModel` → `PhiMoEDecoderLayer` → `PhiMoE` - Pass `enable_eplb` and `num_redundant_experts` to `FusedMoE` constructor - Add `set_moe_parameters()` to `PhiMoEForCausalLM` Follows the same mixin pattern used by DeepseekV2, DeepseekV4, and Glm4MoE. Fixes #40740", "url": "https://github.com/vllm-project/vllm/pull/41412", "createdAt": "2026-04-30T20:43:23Z", "updatedAt": "2026-08-13T10:01:55Z", "timestamp": "2026-08-13T10:01:55Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "needs-rebase" ], "author": "SandishKumarHN", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:41567", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine", "text": "Wire factory and ec_transfer config; add two-process e2e test, EPD full-pipeline script, and README notes. ## Purpose - Add **`ECMooncakeConnector`**: encoder-cache (EC) transfer over **Mooncake TransferEngine** (HTTP registry + ZMQ coordination + pull path), for disaggregated setups where consumers load EC tensors without relying on shared filesystem. - Register the connector in **`ECConnectorFactory`** and document **`ECTransferConfig.ec_connector`** options (`ECExampleConnector` vs `ECMooncakeConnector`, extra config expectations). - Add **integration coverage / ops glue**: - **Two-process CUDA e2e** (`test_ec_mooncake_transfer_e2e.py`): producer on `cuda:0`, consumer on `cuda:1`, registry + tensor equality check. - **EPD full-pipeline script** (`run_epd_mooncake_ec_full_pipeline.sh`): baseline vs 1E+1PD with Mooncake EC + proxy (optional/heavy path). - **README** section for how to run the Mooncake smoke test and dependencies. ## Test Plan **Lightweight (connector transfer only)** - Requires: **2+ CUDA GPUs**, `mooncake-transfer-engine`, `pyzmq`, `httpx`, `fastapi`, `uvicorn`, and a **built** vLLM (`import vllm` / `vllm._C` available). - From repo root: ```bash PYTHONPATH=. MOONCAKE_EC_PROTOCOL=tcp python tests/v1/ec_connector/integration/test_ec_mooncake_transfer_e2e.py", "url": "https://github.com/vllm-project/vllm/pull/41567", "createdAt": "2026-05-03T18:28:57Z", "updatedAt": "2026-08-13T01:35:19Z", "timestamp": "2026-08-13T01:35:19Z", "metrics": { "reactions": 5, "comments": 4 }, "labels": [ "documentation", "v1", "kv-connector" ], "author": "stmatengss", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:41834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes", "text": "## Summary This PR enables DeepSeek V4 Flash on SM120/SM121 Blackwell client hardware by carrying the SM12x fallback and tuning stack needed for the current vLLM V1 path. It targets RTX PRO 6000 Blackwell Workstation Edition, RTX 5090-class SM120, and GB10 / DGX Spark SM121 users who cannot use SM100-only TMEM / `tcgen05` kernels. The branch is **reconciled on top of the merged #43477** and provides the **stock-deps path**: DeepSeek V4 on SM120/121 that builds and serves on *released* FlashInfer / DeepGEMM wheels, complementing #43477's route that needs the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches. It is kept **synced onto current `upstream/main`**. **Latest validated head: tag `sm120-pr-41834-stable-preview-20260809` (`aa0d513027`), synced onto `upstream/main` as of 2026-08-09 (`f18e10a7e1`)** — see [Update 2026-08-09](#update-2026-08-09) below. **The default model runner is now V2**; `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Model / speculative-decode status.** `deepseek-ai/DeepSeek-V4-Flash-0731` is the checkpoint this branch is validated on. It **removed the MTP heads** and folded the DSpark draft into the main checkpoint, so **DSpark (`method: \"dspark\"`, `num_speculative_tokens: 5`) is the speculative path**; MTP is supported only for older checkpoints that still carry those weights. Running without speculation is also fully supported and validated. ## Change footprint — model kernels vs. core-vLLM touch points 187 files, ~+29.1k / −1.3k against `upstream/main`, of which ~10.7k added lines are tests. The branch splits cleanly into model/kernel code and a small set of core-vLLM integration points: - **DeepSeek-V4 model + SM12x kernels — the enablement itself.** Everything under `vllm/models/deepseek_v4/**` plus the SM12x sparse-MLA decode / indexer / DeepGEMM kernels that live in shared dirs (`v1/attention/backends/mla/sparse_mla_kernels.py`, `model_executor/layers/sparse_attn_indexer.py`, `v1/attention/backends/mla/{indexer,sparse_swa}.py`, `utils/deep_gemm.py`, `kernels/mhc/tilelang.py`), the new DSv4 reasoning parser / tokenizer, and device tuning JSONs. - **C128A metadata device→host sync removed** (`models/deepseek_v4/sparse_mla.py`, perf) — `_c128a_effective_topk_width` takes the max position from the CPU-side `CommonAttentionMetadata.max_seq_len` instead of a per-step `int(positions.max().item())` device sync, dropping a launch-stream stall on every C128A metadata step. Decode is identical (`max_seq_len-1 == positions.max()`); only chunked prefill sees a safe, slightly-wider 128-aligned top-k. - **Core-vLLM integration — the hooks below.** Almost all are gated by model architecture / quant config / an env flag and are inert for other models. | Subsystem | Files | What it does | | --- | --- | --- | | KV-cache core | `single_type_kv_cache_manager.py`, `kv_cache_coordinator.py`, `kv_cache_manager.py`, `sched/scheduler.py` (+1) | prefix-cache correctness for DSv4 sparse-MLA + speculative decode: an MLA cache-manager with prompt-block protection, a hybrid-coordinator `cache_blocks` tail-block-reuse rewrite | | Speculative decode | `v1/spec_decode/{dspark,dspark_sampling,llm_base_proposer,dflash}.py`, `config/speculative.py` | DSpark self-drafting proposer + sampling; DSv4 probabilistic draft sampling and per-step draft-layer routing in the shared proposer base; DSpark/MTP method detection and validation | | MoE quantization | `fused_moe.py`, `oracle/mxfp4.py`, `routed_experts.py`, `experts/flashinfer_cutlass_moe.py`, `quantization/mxfp4.py`, `oracle/nvfp4.py` | MXFP4 / NVFP4 backend selection; the one-line NVFP4 fix (FLASHINFER_CUTLASS into the SwiGLU-clamp allow-list) lets DSv4-Flash-NVFP4 serve | | FP8 / Marlin GEMM | `quantization/utils/fp8_utils.py`, `linear/scaled_mm/{cutlass,marlin}.py`, `csrc/.../marlin_moe_wna16/ops.cu` (the only C++) | SM12x e8m0→fp32 upcast + Marlin MoE SM12.0a cudagraph hardening | | cudagraph / compile / config | `config/vllm.py`, `compilation/breakable_cudagraph.py`, `passes/utility/fix_functionalization.py`, `config/compilation.py` | breakable-cudagraph auto-enable gate (MiniMax-only; DSv4 deliberately excluded), DSv4 custom-op defunctionalization + splitting-op registration | | OpenAI entrypoints / parsers | `chat_completion/protocol.py`, `serve/render/serving.py`, `tool_parsers/structural_tag_registry.py`, `chat_utils.py`, `engine/protocol.py`, `chat_completion/{serving,batch_serving}.py`, `reasoning/__init__.py` | expose DSv4 API semantics — `reasoning_content` / `thinking` param / tool-call streaming (jasl/vllm#19 instruction-following) | | Kernel warmup | `model_executor/warmup/deepseek_v4_sm12x_warmup.py` (new), `kernel_warmup.py` (+11) | DSv4 warmup passes (D512-split prefill precompile, paged-MQA rowwise, draft path) that avoid JIT-during-inference wedges. Kept in a separate module so `kernel_warmup.py` stays a two-line hook on upstream's file | | Weight loading | `weight_utils.py`, `default_loader.py` | fast-safetensors weight filter + EP-skip (lowers DSv4 load overhead on GB10) | | env / utils | `envs.py`, `utils/flashinfer.py`, `utils/import_utils.py`, `v1/worker/{gpu_model_runner,ubatch_utils}.py` | `VLLM_DEEPSEEK_V4_*` flags + `has_cutedsl` / `has_flashinfer_trtllm_sparse_mla` probes | Two notes for review: - The most invasive generic edits were **removed** in the 2026-06-21 audit cleanup: the scheduler carries a single +1-line change (the prefill-fairness heuristics were dropped) and the prefix-cache write-fence is gone. - A few hooks touch code paths shared with non-DSv4 models and are worth a closer look: the `kv_cache_coordinator` `cache_blocks` rewrite (affects hybrid-KV models; validated ≥ prior behavior), the proposer base-class change, and the OpenAI-entrypoint plumbing. Everything else (MoE oracle, fp8_utils, cudagraph gate, warmup, envs) is arch / quant / env-gated and inert for other models. ## Duplicate-work check The nearest open/merged PRs are related but not duplicates: | PR | Difference | | --- | --- | | #43477 | **Merged 2026-06-22.** Enables DeepSeek V4 + GLM-5.1 on SM120 via the FlashInfer-SM120 sparse-MLA route, but on its merged form **requires the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches** — on released/stock wheels its SM12x path raises at model construction. This PR is **reconciled on top of #43477** (merge `42657aca65`) and carries the **stock-deps** DSv4 SM120/121 path that runs on released wheels. | | #40929 | Earlier WIP Triton fallback effort. This PR is the maintained replacement branch with the broader scheduler, prefix-cache, parser, quant, warmup, and harness-validated fixes carried forward. | | #42856 | Focused workspace-bound fix that explicitly depends on / references this PR; a subset-style bugfix, not the full DeepSeek V4 SM12x enablement branch. | | #49335 | mxfp8 activation-scale swizzle after DP/EP dispatch — **carried in this branch** (unclaimed upstream). Inert at DP=1; taken for this branch's multi-node DP users. | | #50686 | Consecutive-assistant-message merging in DSv4 prompt encoding — **carried in this branch** (reproduced here before taking it). | | #50693 | B300-targeted prefill-workspace fix. **Test carried, code not needed**: this branch's `_prefill_workspace_topk_bound` returns early for `compress_ratio <= 1` and never reaches the affected buffer. | Upstream PRs whose fixes this branch previously carried as local deltas and has since **retired** in favour of upstream's own version: #48304, #48911, #48959 (via #49052). ## Fixed preview tags These tags are in `jasl/vllm` and give users stable pins while the PR is still moving: | Tag | Commit | Notes | | --- | --- | --- | | `sm120-pr-41834-stable-preview-20260809` | `aa0d513027` | **latest validated head** — 87 upstream commits incl. FlashInfer 0.6.16.post3; the V2 recall collapse root-caused as a prefix-cache ghost-block race and fixed (port of #42359); **default runner switched to V2**. See **Update 2026-08-09**. | | `sm120-pr-41834-stable-preview-20260804` | `0f59188db1` | 35 further upstream commits, four fixes from community reports (DSpark out-of-vocab draft token, eager scratch pool), two contributor PRs. Validated on **both** SM121 and SM120. See **Update 2026-08-04**. | | `sm120-pr-41834-stable-preview-20260802` | `9a94c54292` | 234 upstream commits, `DeepSeek-V4-Flash-0731` support, two DSpark config fixes, #49335 / #50686 absorbed. See **Update 2026-08-02**. | | `sm120-pr-41834-stable-preview-20260727d` | `d64074e6f0` | 209-commit upstream sync + torch 2.13 (tag `…-20260727`, `70a33886bd`); DSpark VRAM work (jasl/vllm#27) merged; bounded block-table gather in `compute_global_topk_indices_and_lens`. | | `sm120-pr-41834-stable-preview-20260721` | `832775efd1` | 79-commit upstream sync; #48911 dropped in favour of upstream's merged version; compact CPU KV offload (opt-in). | | `sm120-pr-41834-stable-preview-20260717` | `f63bfd3d7b` | 195-commit upstream sync; prefill ctx_pp +4.7% @ d8192. | | `sm120-pr-41834-stable-preview-20260711` | `b5c0d43b96` | 181-commit upstream sync; #48304 MTP unscaled-draft-rope; ~1097-line dead-kernel cleanup. | | `sm120-pr-41834-stable-preview-20260704` | `b43470e871` | @GanyX19 GB10 fixes: per-shape `constexpr`→runtime (stops the Triton recompile → unified-memory leak → hard-freeze) + fp8-einsum `tl.multiple_of(16)` (~24% decode @256K). | | `sm120-pr-41834-stable-preview-20260703` | `444fe3ac8b` | DSpark spec-decode (self-drafting block-5), V2 padded-Q OOM fix (jasl/vllm#26), exact non-cooperative `persistent_topk` for <128 KB-smem parts. | Older tags (`…-20260705` back to `…-20260612…`) remain in `jasl/vllm` for history. ## Update 2026-08-02 — `DeepSeek-V4-Flash-0731`, 234 upstream commits, two DSpark fixes Validated head **`9a94c54292`** (tag `sm120-pr-41834-stable-preview-20260802`), 234 upstream commits absorbed, level with `upstream/main` as of 2026-08-02. **What's in it** - **`DeepSeek-V4-Flash-0731` support.** The new checkpoint ships **no MTP heads** — `enorm`, `hnorm`, `e_proj`, `h_proj` and `shared_head` are absent from the weight index, and `mtp.{0,1,2}.*` now carries the DSpark-style `main_norm` / `main_proj` structure (matching `dspark_target_layer_ids: [40, 41, 42]`). DSpark is the speculative path going forward; the MTP code is retained for older checkpoints. - **`num_speculative_tokens` vs `dspark_block_size`** — *this rule was relaxed on 2026-08-04; see Update 2026-08-04. It now errors only BELOW the block size and warns above it.* The original reasoning and measurements follow. - The validator was tightened to require equality. The validator previously accepted `>=` and its error message recommended exceeding it. The drafter emits exactly one block per pass, so the extra slots are structurally unreachable — measured on a prose workload, the **7th draft position accepted 0.000 in every sample** (the 6th in all but one, 0.004 there), and `nst=7` drafts 40% more tokens per step for strictly worse acceptance: | configuration | mean acceptance length (3 samples) | avg draft acceptance rate | | --- | --- | --- | | nst=5 probabilistic | 2.15 / 2.16 / **2.19** | 22.9 / 23.2 / **23.8%** | | nst=7 probabilistic | 1.61 / 1.75 / 1.95 | 8.7 / 10.7 / 13.6% | | nst=5 greedy | 1.82 / 2.06 / **2.23** | 16.4 / 21.2 / **24.5%** | | nst=7 greedy | 1.57 / 1.66 / 1.75 | 8.2 / 9.5 / 10.8% | All samples are shown rather than a single figure: the probe reads whatever `SpecDecoding metrics` lines vLLM flushed inside its window, so a low sample means \"not much steady traffic in that slice\", not a worse drafter. Both `nst=7` runs also hit connection errors partway through, so their spread is noisier. - **`method: \"mtp\"` is no longer silently rewritten to `\"dspark\"`.** Auto-detection preserved an explicitly requested method only for `eagle`/`eagle3`/`dflash`/`dspark`. Since 0731 puts `dspark_block_size` in every DSv4 config, `method: \"mtp\"` fell through to the dspark branch, was rewritten, and then failed validation with a DSpark message the user never asked for. - **Absorbed** #49335 (mxfp8 activation-scale swizzle after DP/EP dispatch, verified non-vacuous) and #50686 (consecutive assistant messages merged in prompt encoding — it also removes a spurious mid-turn `<|end_of_sentence|>` on the default `drop_thinking=True` path). #50693's regression test is carried; its code fix is not reachable here. - **Three community reports re-tested and not reproduced** on this head: nst=7 draft-position collapse, TP=4 + expert-parallel `Check failed: num_tokens > 64`, and `FLASHMLA_SPARSE_DSV4` missing `tile_sched`. Details in [this comment](https://github.com/vllm-project/vllm/pull/41834#issuecomment-5152562124). **Validation (GB10 SM121, 2-node TP=2, `DeepSeek-V4-Flash-0731`, torch 2.13.0, FlashInfer 0.6.15.post1, nccl 2.30.7)** | | DSpark nst=5 | no speculation | | --- | --- | --- | | GSM8K 8-shot (flexible) | 0.9394 | 0.9500 | | GSM8K 8-shot (strict) | 0.9363 | 0.9484 | | instruction-following (jasl/vllm#19, JSON-only) | PASS | PASS | | long-context recall (arthur needle, c=1) | 2/2 | 2/2 | | illegal-access / assertion in serve log | 0 | 0 | | draft acceptance (prose) | mean 2.08, 21.7% | — | The GSM8K difference (1.06 pp flexible / 1.21 pp strict) is within this gate's measured single-run spread (~1.1 pp). **Resolved:** three runs per cell were collected and the arms interleave, so it was noise. `0731` is the first checkpoint where the strict and flexible extractors disagree at all; on every prior baseline they were identical. Perf — pinned llama-benchy standard (fp8 KV, prefix-cache on, `FULL_AND_PIECEWISE`, mml 49152, util 0.85; C=1, 3 runs), against the **full** recorded range of the prior MTP2 baselines. This crosses a checkpoint boundary, so read it as a sanity band rather than a controlled A/B: | metric | prior MTP2 range (n=10) | 0731 + DSpark | vs band | | --- | --- | --- | --- | | pp2048 @ d8192 | 1339.11 – 1400.81 | **1432.23** ± 11.74 | **above** | | pp2048 @ d16384 | 1308.77 – 1344.68 | **1356.56** ± 11.78 | **above** | | pp2048 @ d32768 | 1089.05 – 1226.63 | **1250.75** ± 2.18 | **above** | | ctx_pp @ d8192 | 1757.16 – 1876.01 | 1816.97 ± 5.89 | inside | | ctx_pp @ d16384 | 1769.85 – 1842.16 | 1817.43 ± 1.43 | inside | | ctx_pp @ d32768 | 1595.87 – 1756.01 | 1740.22 ± 2.87 | inside | | tg128 @ d8192 | 36.27 – 43.08 | 41.72 ± 5.09 | inside | | tg128 @ d16384 | 34.59 – 43.14 | 37.92 ± 9.92 | inside | | tg128 @ d32768 | 32.77 – 42.91 | 34.88 ± 5.78 | inside | | ctx_tg @ d8192 | 38.52 – 43.01 | 39.37 ± 2.34 | inside | | ctx_tg @ d16384 | 39.29 – 43.07 | **35.07** ± 0.67 | **below, −10.7%** | | ctx_tg @ d32768 | 38.02 – 42.73 | 40.70 ± 6.85 | inside | **Batched prefill (pp2048) is above the historical band at all three depths** (+2.2% / +0.9% / +2.0%) — the only consistent directional move here. Clearing the max of ten prior runs at all three depths says more than any single one of those margins would: +0.9% is inside this metric's own resolution, so read the consistency rather than the magnitudes. No sign of DSpark being slower than MTP2 was. One caveat reported rather than buried: **`ctx_tg @ d16384` sits 10.7% below its historical minimum**, the only metric outside its band. It is non-monotonic against our own neighbouring depths (39.37 at d8192, 40.70 at d32768, where history has d16384 ≈ d8192), which points at a single-run artifact rather than a depth-specific regression. **Resolved:** repeated on later heads and it did not recur. A measurement caveat for anyone benchmarking this branch: the `±` in a benchy row is the spread of the three runs *inside one invocation*, and it runs 5–30× smaller than the build-to-build spread. This branch's own history spans 31% on tg128 @ d32768 and ~1.3% on ctx_pp, so anything under ~15% on tg or ~2% on ctx_pp is not resolvable this way. ## Update 2026-08-04 — four fixes from community reports, 35 upstream commits, and first SM120 validation Validated head **`0f59188db1`** (tag `sm120-pr-41834-stable-preview-20260804`). **This is the first head validated on both SM121 and SM120.** Every SM120 discrete-GPU result on this PR up to now was a contributor's measurement we could not reproduce. We have since rebuilt a 2× RTX PRO 6000 Blackwell box as a first-party SM120 target. ### Fixes - **DSpark's fused Markov sampler could emit an out-of-vocab token id** (`e171c51036`). `_dspark_markov_probs_blocks_kernel` stores `vocab_size` as the filler for a block with no active lane. On a fully-masked row — every candidate `-inf`, which structured-output constraints can produce — *no* block has an active lane, so every block stores the filler and the reduce kernel returns it verbatim as the sampled token. Nothing downstream bounded it: the runner clamped `input_ids` with `min=0` only, and the DSv4 hash-MoE router indexes `tid2eid[token_id * 6 + lane]` on a `[vocab_size, 6]` table. Result is an illegal memory access on every TP rank. This is the producer on the **V1** path, which is this branch's default. @alexbi29's report traced the same class of defect to the V2 samplers (#50843) — a real defect, but a different tree. Fixed by folding out-of-range to `0` (matching `torch.argmax` on such a row, so the fused kernel stays bit-identical to the eager reference) and making the runner clamp two-sided. Worth stating plainly for anyone with similar gates: **our own gates could not have caught this**. The fused path is skipped when `all_greedy`, and both our long-context recall gate and GSM8K are greedy, so they are structurally incapable of executing that kernel. The new regression test is explicitly non-greedy. - **Adopted #50844** (`3df857ba50`) — bound `token_id` before the `tid2eid` gather. Defence in depth; `prompt_token_ids` reach that gather directly when `--skip-tokenizer-init` disables the engine's vocab check. **Not** taking #50843 (V2-tree only, inert on our default) or #50845, which has a defect reported on its own thread. - **Eager scratch pool is now OFF by default** (`d42b8d9f55`, `b1ef3033f4`), opt-in via `VLLM_DEEPSEEK_V4_EAGER_SCRATCH_POOL=1`. @tobymao bisected output corruption under concurrent mixed prefill+decode to it: pool active 7/7 rounds corrupt, disabled 0/2. We first removed the cross-template aliasing (`max()` → `sum()` sizing with per-family offsets); **they tested that commit directly and it was still corrupt in round 1**. Their diagnosis is the useful part: the pre-pool code was race-free *for free* because per-call transients go through the caching allocator, whose cross-stream reuse is event-guarded — the pool reuses memory without that machinery, so no static partitioning fixes it. Making the cross-layer reuse safe needs producer-waits-on-consumer events against the real stream graph; until then, off by default. - **Two contributor PRs merged** — jasl/vllm#37 (tuned FP8 W8A8 config for `N=4096,K=12288` on RTX PRO 6000) and jasl/vllm#38 (hoist the E8M0 block-scale upcast out of the FP8 GEMM hot path, 13,561 kernel launches removed per 25 decode steps), both from @alexbi29. - **`num_speculative_tokens` rule relaxed.** Upstream removed its own assertion in #50869 as \"invalid\". They were right that erroring *above* `dspark_block_size` is wrong — two users on this thread run `nst=7` against `block_size=5` and it demonstrably works. The two directions are not symmetric, so this branch now **errors below** the block size (that genuinely garbles output) and **warns above** it, quoting the acceptance cost. Strictly more permissive than what shipped before. ### Validation Full gate battery on **both** architectures, same branch: | gate | SM121 (2× GB10, 2-node TP=2) | SM120 (2× RTX PRO 6000, TP=2) | | --- | --- | --- | | serve, DSpark nst=5, `--block-size 256` | ✅ | ✅ | | instruction-following (jasl/vllm#19) | PASS | PASS | | long-context recall, arthur c=1 | 2/2 | 2/2 | | long-context recall, arthur c=12 | 22, 23, 22 / 24 | 22, 23 / 24 | | GSM8K 8-shot flexible | 0.9484 / 0.9507 / 0.9492 | 0.9371 | | GSM8K 8-shot strict | 0.9462 / 0.9477 / 0.9462 | 0.9303 | | tool-calling, 135 cases | 256/270 (94.8%) | — | | illegal-access / assertion lines | 0 | 0 | The ~1.1 pp GSM8K difference between architectures sits inside this gate's measured single-run spread and spans different silicon, different memory architecture and a 3× smaller KV cache (6.25 GiB vs ~18.5 GiB). We are not claiming a difference from it. **`Check failed: num_tokens > 64` does not reproduce on this branch.** @fuzzifikation reported stock 0.26.0 dying there on SM120 at `--block-size 256`, correctly tracing it to the DSv4 decode dispatch requiring `page_block_size == 64`. On our SM120 box, at the same `--block-size 256`, the serve comes up and the assertion never appears — the DSv4 packed KV cache is laid out in 64-token pages independent of vLLM's logical block size, and FlashInfer derives `page_block_size` from tensor geometry rather than the engine config. The SM120 packed decode path is confirmed engaged in the same run. Note the same assertion has **two** distinct gates (`page_block_size` and `(num_heads, topk)`, the latter being #50720 / flashinfer#3989), so patching one and still seeing it means checking the other. ### Prefill: V1 vs V2 model runner The 2026-08-02 V1-vs-V2 comparison never measured throughput. It has now been measured, blocked and pre-registered — 10 blocks, both arms inside each node pair, exact sign-flip permutation test, Holm-corrected across the six prefill cells, with the decision rule committed before any data was collected: | metric | V2 / V1 | 95% CI | exact p | | --- | --- | --- | --- | | ctx_pp @ d8192 | **+1.18%** | [+0.82, +1.53] | 0.0020 | | ctx_pp @ d16384 | **+1.11%** | [+0.35, +1.87] | 0.0137 | | ctx_pp @ d32768 | **+1.61%** | [+1.17, +2.04] | 0.0020 | | pp2048 @ d8192 | **+4.18%** | [+3.28, +5.09] | 0.0020 | | pp2048 @ d16384 | **+4.18%** | [+3.36, +5.00] | 0.0020 | | pp2048 @ d32768 | **+4.33%** | [+3.56, +5.11] | 0.0020 | All six survive Holm; both node pairs agree in direction on every cell. **Decode is not resolved in either direction** — `tg128` was declared unresolvable before the run (its within-build spread equals its entire historical range) and is reported for the record only. **V1 was the default when this was written; that was reversed on 2026-08-09 — see Update 2026-08-09.** The reasoning below was correct on the evidence available at the time, and the collapse it describes was real; it turned out not to be a property of the runner. Kept unedited because how the conclusion failed is the useful part. V2 is ahead on prefill, KV headroom (+4.70 GiB) and draft acceptance (+6.6%), but its long-context recall under concurrency is unreliable in a way that is worse than a consistent deficit: across 14 independent serves on the same build and configuration, roughly two thirds land in a state that loses most of the needles (arthur c=12 as low as 3/24), while the rest match V1 at 22–24/24. The mode is fixed at startup and stable within a serve, and nothing we have found predicts or detects it. A deployment could run clean for days and restart into the bad mode. The cause is **not identified**. Eliminated so far: the eager-scratch cross-template aliasing, the upstream merges, and the eager scratch pool as a whole (pool on 2 good / 6 bad vs pool off 3 good / 3 bad over 14 serves — no effect). The startup logs of a good and a bad serve are structurally identical, which rules out \"a different code path was taken\". Anyone opting into V2 with `VLLM_USE_V2_MODEL_RUNNER=1` should know this. ### Measurement note Two errors from our own process, since they affect how the numbers above should be read. The n=8 sampling that originally established V2's recall deficit took eight gate runs from **one serve** — it measured within-serve variance while the quantity that actually varies is across-serve. Raising n on the wrong axis. The 14-serve figures above use the inverted design: many serves, few gates each. And the `±` in a benchy row is the spread *within one invocation*; it runs 5–30× smaller than the build-to-build spread. The blocked design above exists because of that: a coarse range screen over the same 10 blocks returns \"no measurable difference\" on all six prefill cells, while the paired test finds all six. Had the screen been the decisive statistic, this section would have concluded the opposite and been wrong. ## Update 2026-08-09 — the V2 recall collapse was a prefix-cache race, not the runner; V2 becomes the default 70 upstream commits (to `643c125fab`), and the long-standing reason this branch pinned V1 is gone: it was **an unfixed upstream bug**, not a property of the V2 model runner. ### The defect `FullAttentionManager.cache_blocks()` commits prefix block hashes to the shared `BlockPool` **at scheduling time, before the forward pass writes their KV**. A request admitted later in the same step can match those hashes and read unwritten values. `MambaManager` has guarded this since #29387; no other manager does. This is **vllm-project/vllm#42359**, open and unmerged. Two more reports look like the same triple on different models — **#50188** (prefix caching + MTP spec decode + fp8 KV, byte-identical repeat requests, RTX 5090 / Qwen3.6-27B-NVFP4) and **#43559** (closed without a merged fix). Anyone on `--enable-prefix-caching` with speculative decoding is exposed; DeepSeek-V4 is not special here. What makes it hard to catch: **the damage persists**. A serve that loses the race keeps serving from the poisoned blocks for its lifetime, so a later *serial* request fails too — which is why it looked like a per-serve \"mode\" rather than a race. It is also stochastic, roughly half of cold serves. ### Evidence Same binary, `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT` the only variable, cache populated by the real gate, **4 fresh serves per arm** (a single clean serve proves nothing at ~50% incidence), 3 arthur c=12 runs each: | | serve 1 | serve 2 | serve 3 | serve 4 | mean | min | | --- | --- | --- | --- | --- | --- | --- | | guard **off** | 22/23/24 | **6/5/3** | **14/9/10** | **7/8/7** | 11.5 | 3 | | guard **on** | 23/22/21 | 20/22/20 | 22/23/23 | 23/22/23 | 22.0 | 20 | Mann-Whitney U, p = 0.0043. Every serve's runner and guard state was read back from the serve log rather than assumed. **It also fixes V1**, which was not expected: V1's arthur c=12 goes 20.7 → 23.0 with the guard on (24.0 with prefix caching disabled entirely). V1's own 2–4 needle shortfall was the same defect, not an inherent concurrency margin. ### Runner arbitration, re-run on the fixed tree Same tree, same guard mode, **runner the only variable**: | | V1 | V2 | | --- | --- | --- | | arthur c=12, 4 serves × 3 | 22.3 / 21.7 / 21.7 / 21.0 → **21.67** | 22.0 / 20.7 / 22.7 / 22.7 → **22.00** | | pp2048 d8192 / 16384 / 32768 | 1427 / 1363 / 1216 | **1472 / 1421 / 1303** (+3.1% / +4.2% / +7.1%) | | tg128 mean | 39.95 / 41.16 / 35.25 | **41.56 / 49.61 / 45.18** | | e2e TTFT | 1437 / 1506 / 1689 ms | **1393 / 1444 / 1576 ms** | | GPU KV cache | 339,194 tok | **423,752 tok** (+24.9%) | | GSM8K strict / flexible | 0.9378 / 0.9401 | 0.9401 / 0.9439 | | issue19 · multi-needle · c=1 | PASS · 48/48, 0 leaks · 2/2 | PASS · 48/48, 0 leaks · 2/2 | Recall: p = 0.697, neither side with a single-digit serve. GSM8K differs by <0.4 pp against ~1.1 pp single-run noise. **V2 is not behind anywhere and leads on throughput, latency and KV headroom, so it becomes the default.** `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Correction:** the *\"V2 +6.6% draft acceptance\"* figure in Update 2026-08-04 does **not** survive re-measurement — 2.772 (V1) vs 2.710 (V2) on the same formula and sample size, i.e. a tie. It was measured while V2 was poisoned. ### If you are running this branch **Nothing to set — the guard is on by default where it matters.** `KVCacheCoordinator` enables it whenever prefix caching and speculative decoding are both active, which is the only configuration in which a block hash can be published before its KV is written *and* a second request admitted in the same step to match it. This correction matters: an earlier revision of this update shipped V2 as the default while leaving the guard off by default, which would have handed a plain serve the exact combination measured at mean 11.5 with a 3/24 floor. Both changes looked like improvements in isolation. If you pulled `c054feedac`, take `aa0d513027` instead, or set the variable yourself. To turn it **off** (it is a real escape hatch, pinned by a test): ```bash VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0 ``` `1` is upstream's semantics, gated on `use_eagle`; on DeepSeek-V4 that covers only **2 of 5** managers and leaves the main MLA path unguarded — measured, not assumed, via a startup log line this branch adds that reports how many managers are actually guarded. `2` covers every group and is what the engine selects. Regression sweep on the merged tree: 261 passed, 1 failed, that one failing identically on the pre-merge tree `4ebd1fb698`. ### The two endpoints disagreed about the same model, twice `ResponsesRequest.reasoning` took its type from the OpenAI SDK, whose `ReasoningEffort` stops at `xhigh`, so DeepSeek's documented top tier `max` was rejected by schema validation on `/v1/responses` while `/v1/chat/completions` accepted it. Worse, and on the default path: with no thinking kwarg, `DeepSeekV4Tokenizer.apply_chat_template` defaults thinking **on** while `DeepSeekV4ReasoningParser` defaults it **off** and selects `IdentityReasoningParser`. The model reasoned and its reasoning, with a bare ``, came back inside `output_text` as though it were the answer — whenever a request omitted `reasoning`, which is exactly what a stock OpenAI SDK sends. Chat was immune only because it normalises thinking state at the protocol boundary, and its own docstring says why: *\"so the tokenizer and reasoning parser see the same effective state\"*. Responses never called that hook. The derivation now lives in `deepseek_v4_chat_kwargs` and both request types call it, so a third endpoint cannot repeat it. Measured on both checkpoints with no workaround flag set, 21/21 each: | | `DeepSeek-V4-Flash-0731` | `DeepSeek-V4-Flash` | | --- | --- | --- | | silence: reasoning in its own field | PASS | PASS | | silence: no `` in the answer | PASS | PASS | | `effort: none` disables thinking | PASS | PASS | | six spellings × two endpoints | PASS | PASS | | `high` reasons deeper than `low` | +85% / +110% | +19% / +35% | 26 unit cases accompany it, 11 of which fail on the unpatched tree. `tests/reasoning` 440 passed, `tests/tokenizers_/test_deepseek_v4.py` 45 passed. ### Acceptance on the exact published SHA Everything above was re-measured on `d44e224ab9` — the commit this tag points at, after a second upstream sync (17 further commits, FlashInfer 0.6.16.post3) — not on an ancestor assumed to be equivalent. 17 of 18 checks pass: | check | result | | --- | --- | | four nodes clean at the SHA, FlashInfer 0.6.16.post3 | PASS | | `tests/v1/core` | 509 passed, 1 pre-existing failure | | default serve with **nothing set**: boots, runner V2, guard 5/5, no NameError | PASS | | arthur c=12 ×3 / c=1 | 22 / 20 / 23 · 2/2 | | GSM8K strict · issue19 · multi-needle | 0.9363 · PASS · 48/48, 0 leaks | | pp2048 d8192 | 1480.77 (arbitration V2 arm 1471.8) | | `VLLM_USE_V2_MODEL_RUNNER=0` → V1, guard still 5/5, c=1 2/2 | PASS | | `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0` → guard 0/5, c=1 2/2 | PASS | **The one non-pass, and what it turned out to be.** `tests/v1/spec_decode` does not complete on this hardware — it wedges under a 30-minute bound on this head and on `4ebd1fb698` alike. Narrowed to `test_max_len.py` and measured both ways: | how it is run | result | | --- | --- | | whole file, one pytest process | wedges after ~7 min, 5 of 11 done | | each case in its own process | **11 of 11 pass**, free memory steady at 117 GiB | No individual case is broken. Each stands up a full engine, and repeated create/tear-down inside one process does not release resources fast enough on a **single-GPU** unified-memory node. That also explains why both trees wedge and why they stop at different points. It remains **unverified coverage rather than a pass**; running one process per case produces a verdict instead of a hang. `test_async_scheduling_pp_allows_rescheduling_with_output_placeholders` is the same class: it builds `pipeline_parallel_size=2`, and a GB10 node has one GPU, so it fails at config construction. It is the only case in `tests/v1/core` that needs more than one GPU; the other 509 pass. ### What this arbitration does and does not cover Everything above was measured on **one configuration**: 2-node TP=2, `DeepSeek-V4-Flash-0731`, DSpark `num_speculative_tokens: 5`, fp8 KV, `max_model_len` 131072, prefix caching on, GB10 (SM121). The default now applies to **every** DSpark config, including shapes not measured here — TP=4, other context lengths, the NVFP4 checkpoint, single-node setups. The reasoning generalises better than the numbers do: the race is in block publication and is not specific to a model shape, and V2's advantage comes from KV headroom and scheduling rather than anything config-specific. But if you run a materially different shape and see something worse, `VLLM_USE_V2_MODEL_RUNNER=0` returns you to V1 and a report would be welcome — that is a gap in our coverage, not a claim we have ruled out. ## Running DSpark DSpark is DeepSeek's self-drafting speculative-decode variant; on `0731` the draft weights are carried in the main checkpoint, so no separate `--speculative-model` is needed. ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --max-model-len 49152 \\ --max-num-seqs 64 \\ --max-num-batched-tokens 8192 \\ --gpu-memory-utilization 0.85 \\ --enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\"}' ``` - `num_speculative_tokens` **must equal** the checkpoint's `dspark_block_size` (5). Larger values are rejected: they are never accepted and only waste draft compute. - `--kv-cache-dtype fp8` is mandatory — DSv4's `fp8_ds_mla` attention asserts an fp8 KV layout, so the default `auto` fails at model construction. Not DSpark-specific. - Runs on the **V1 runner by default** (correct long-context recall). `VLLM_USE_V2_MODEL_RUNNER=1` opts into the V2 DSpark speculator; V2's long-context recall is correct after the jasl/vllm#26 padded-Q fix. - If you measure draft acceptance yourself, **use prose**. On counting or repeated text the Markov head alone reaches 68–100% acceptance even with the neural draft path degraded, which hides real regressions entirely. ## Dependencies (stock-deps path) Pins on the current head: torch 2.13.0 (triton 3.7.1) · `flashinfer-python` / `flashinfer-cubin` 0.6.15.post1 · tilelang 0.1.12 · `nvidia-cutlass-dsl[cu13]` 4.6.0 · `quack-kernels>=0.6.1` · nvidia-nccl-cu13 2.30.7 (multi-node, see below). - **FlashInfer** is pinned in `requirements/cuda.txt` (`flashinfer-python` **and** the GitHub-release `flashinfer-cubin`, which must be the same version); it ships the SM120 packed sparse-MLA kernels, so a stock build picks them up with no manual install dance. - **GB10 / multi-node**: pin `nvidia-nccl-cu13==2.30.7` on every node. A rebuild silently reverts it to torch's bundled version, and a per-node mismatch hangs the NCCL handshake. - The SM120 **decode** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_DECODE`) and **prefill** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_PREFILL`) FlashInfer sparse-MLA paths **default on**; set either `=0` to fall back to the FlashMLA / Triton path. Both are availability-gated, so stock installs without the kernel degrade gracefully rather than raising. ## Running the NVFP4 checkpoint This branch also serves `nvidia/DeepSeek-V4-Flash-NVFP4` on SM12x (RTX PRO 6000 / GB10). The NVFP4 MoE auto-selects the FlashInfer CUTLASS backend (the SwiGLU-clamp model gate accepts it), so no `--moe-backend` flag and no special FlashInfer build are required: ```bash vllm serve nvidia/DeepSeek-V4-Flash-NVFP4 \\ --trust-remote-code --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --tokenizer-mode deepseek_v4 ``` Expert-parallel off (plain TP) is the supported path. Accuracy matches MXFP4 (GSM8K 8-shot ~0.96 on both SM120 and SM121). On SM12x NVFP4 is **not** a memory or throughput win versus MXFP4: NVFP4 weights are ~4 GiB/GPU larger, leaving less KV-cache room; single-stream prefill is marginally faster and aggregate decode marginally slower. Its value here is checkpoint availability / parity with the SM100 datacenter path — MXFP4 remains the better practical choice on consumer Blackwell. ## AI assistance disclosure AI assistants, including OpenAI Codex/GPT models and Anthropic Claude models, were used for code review, refactoring support, regression-script writing, and benchmark analysis. The branch was validated through human review plus the commands and harness artifacts listed above; every performance and accuracy number quoted was measured on real SM120/SM121 hardware.", "url": "https://github.com/vllm-project/vllm/pull/41834", "createdAt": "2026-05-06T15:17:15Z", "updatedAt": "2026-08-13T14:36:24Z", "timestamp": "2026-08-13T14:36:24Z", "metrics": { "reactions": 41, "comments": 441 }, "labels": [ "new-model", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "tool-calling", "deepseek", "kv-connector", "nvidia", "quantization", "mrv2" ], "author": "jasl", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:42436", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fused_moe: add VLLM_TRITON_USE_TD tensor-descriptor path", "text": "Working on perf optimizations for the Triton MoE kernels. Adds an opt-in `VLLM_TRITON_USE_TD` env var that switches the fused MoE kernel onto a tensor-descriptor based load/store path, mirroring `VLLM_TRITON_ATTN_USE_TD` (PR #40327). Auto-on for XPU; off by default on CUDA/ROCm (opt-in on Blackwell). The changes in this PR are scoped to the fused MoE kernel (`fused_moe_kernel`) only. The env var name follows the single-flag design proposed in the TD adoption strategy RFC (#42545): one `VLLM_TRITON_USE_TD` switch gates the tensor-descriptor path across Triton kernels. Note the auto-select policy is per-subsystem: with the env unset, both the attention dispatcher (`triton_attn.py`) and this MoE path auto-enable on XPU only. Blackwell CUDA can run the path but must opt in explicitly (see below). This PR renames the previously-proposed `VLLM_TRITON_MOE_USE_TD` to the general form for consistency with the RFC. ## Quantization The TD gather branch is validated for non-quantized (bf16) weights only. Under any quantization (fp8 w8a8, int8 w8a8, int8 w8a16, int4 w4a16) the launch site falls back to the pointer path — `use_td = resolve_moe_use_td() and not is_quantized` — so quantized MoE is byte-identical to `main`. When `VLLM_TRITON_USE_TD` is set explicitly but ineffective (non-Triton backend, or quantized weights), a one-shot warning fires against the *selected* MoE backend. Extending the TD path to quantized MoE is a known gap and is being addressed as a separate follow-up under the same RFC (#42545); that PR is almost ready for publication. ## Hardware gating (CUDA) The A gather uses `tensor_descriptor.gather`, which lowers to the PTX `tile::gather4` instruction (part of the `tcgen05`/Tensor Memory family introduced with Blackwell). `ptxas` rejects it on Hopper (sm90) and earlier — `Feature '.tile::gather4 ...' requires .target sm_100 or higher`. `moe_use_td_hw_supported()` reports whether a device can *compile* the path: `True` on XPU, `has_device_capability(100)` on CUDA (gather4 spans the whole sm100+ range incl. consumer Blackwell sm120/121, unlike `scatter4` — see triton-lang/triton#8498), `False` otherwise. Auto-select does not follow compile-capability: with the env unset the path is XPU-only. Blackwell CUDA can compile it but is opt-in (`VLLM_TRITON_USE_TD=1`) pending its own accuracy/perf validation — it currently has none, and this PR deliberately does not enable it by default. An explicit `VLLM_TRITON_USE_TD=1` on hardware that cannot compile the path still forces it (useful for A/B), so the parametrized test carries a matching hardware skip-guard, and the launch registers the Triton scratch allocator (`make_tensor_descriptor` needs it on CUDA). A `tl.static_assert` enforces that `USE_TD` and `SWAP_AB` are never both enabled. ## Testing / CI coverage `test_fused_moe` is parametrized over `use_td`. The `use_td=True` cell skips on any device that cannot compile the TD path (Hopper/Ampere, and Triton < 3.6) — i.e. it does not execute on the default upstream kernel-CI fleet. The TD branch is exercised only on XPU / Blackwell CI, so the validation below was run by hand on the target hardware. [pr-42436-multiplatform-validation.zip](https://github.com/user-attachments/files/30166388/pr-42436-multiplatform-validation.zip) ## Results - multi-platform production-readiness validation (2026-07-17/18) Extended the validation above to a full unit → accuracy → E2E-perf → quantized-smoke → kernel-level pass on one card each of Intel Arc Pro B70 (XPU), NVIDIA H200 (Hopper, sm90), NVIDIA B200 (Blackwell, sm100). Pinned commit: `92a58914f9ee6a39bc884c278a2fc6367aea7c3a`. Full methodology, kernel-level benchmark results, dispatch-log verification of the TD auto-select resolver on every platform, hardware-gate verification on H200, quantization no-op verification, an unresolved B70 host-to-host perf disagreement, and a data-correctness correction applied to some `sharegpt` numbers below (documented in the report itself), are in the attached campaign report (`TRITONXPU-181_production_readiness_campaign_2026-07-18.md`). Full raw logs attached per-platform (`b70/`, `h200/`, `b200/`). Summary: correctness (K-alignment guard fix, TD/pointer parity, quantization no-op fallback, hardware-gate enforcement on Hopper) is well-evidenced across all 3 platforms via dispatch-log-level verification, not just output matching. Perf is shape/platform-dependent with no universal TD win — reported honestly below, not as a performance pitch. Variant naming used below: `default` = no `--moe-backend` flag, no `VLLM_TRITON_USE_TD` (platform's true native default — SYCL on B70, FlashInfer TRTLLM on B200, Triton on H200 per vLLM's own Hopper-specific backend priority); `td0`/`td1` = `--moe-backend triton` with `VLLM_TRITON_USE_TD=0`/`=1`. ### Accuracy — GSM8K, full 1319-question test set, 5-shot, greedy decode (temperature=0.0), seed=42 | Platform | td0 (pointer) | td1 (TD) | Δ | |---|---:|---:|---:| | B70 | 46.85% | 46.85% | 0.00pp (bit-identical) | | H200 | 48.60% | N/A (TD uncompilable on sm90) | — | | B200 | 48.07% | 48.07% | 0.00pp (bit-identical) | TD does not change accuracy anywhere it runs — bit-identical to the pointer path token-for-token on both B70 and B200 (confirms the K-alignment correctness fix). Raw per-platform results below.
B70 ```json --- sycl (default) --- { \"accuracy\": 0.47763457164518575, \"invalid_rate\": 0.001516300227445034, \"latency\": 43.434272416867316, \"questions_per_second\": 30.367724071458788, \"total_output_tokens\": 178930, \"tokens_per_second\": 4119.5578984883405, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.46853677028051555, \"invalid_rate\": 0.002274450341167551, \"latency\": 103.7776588909328, \"questions_per_second\": 12.70986466736766, \"total_output_tokens\": 179153, \"tokens_per_second\": 1726.3156821477774, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } --- td1 --- { \"accuracy\": 0.46853677028051555, \"invalid_rate\": 0.002274450341167551, \"latency\": 74.65302416798659, \"questions_per_second\": 17.66840680200637, \"total_output_tokens\": 179153, \"tokens_per_second\": 2399.8090097042054, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"1\", \"seed\": 42 } ```
H200 ```json --- default (= td0, dispatch-confirmed identical Triton code path) --- { \"accuracy\": 0.48597422289613346, \"invalid_rate\": 0.0, \"latency\": 9.003313882742077, \"questions_per_second\": 146.5016123150292, \"total_output_tokens\": 179112, \"tokens_per_second\": 19894.00817662586, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.48597422289613346, \"invalid_rate\": 0.0, \"latency\": 10.623563874978572, \"questions_per_second\": 124.15795824474773, \"total_output_tokens\": 179112, \"tokens_per_second\": 16859.878860601406, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } ``` (No `td1` row: TD cannot compile on Hopper/sm90 — `ptxas` rejects `tile::gather4` as requiring sm_100+. Attempted and captured as an expected-failure artifact, see the attached logs.)
B200 ```json --- default (FlashInfer TRTLLM, real distinct backend) --- { \"accuracy\": 0.4836997725549659, \"invalid_rate\": 0.000758150113722517, \"latency\": 10.077878876123577, \"questions_per_second\": 130.8807156955382, \"total_output_tokens\": 178460, \"tokens_per_second\": 17708.09137454568, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.4806671721000758, \"invalid_rate\": 0.002274450341167551, \"latency\": 8.92053499398753, \"questions_per_second\": 147.8610869066721, \"total_output_tokens\": 179372, \"tokens_per_second\": 20107.762608509165, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } --- td1 --- { \"accuracy\": 0.4806671721000758, \"invalid_rate\": 0.002274450341167551, \"latency\": 9.130568650085479, \"questions_per_second\": 144.45978673931242, \"total_output_tokens\": 179372, \"tokens_per_second\": 19645.21673010155, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"1\", \"seed\": 42 } ```
`td0`/`td1` are bit-identical on both B70 and B200 (confirms the K-alignment fix preserves output correctness under TD). Full accuracy methodology caveat (N=1 has a ~1-2pp run-to-run noise floor on this class of backend) is in the attached report §5. ### Perf — `vllm bench sweep serve`, N=3 per (variant, dataset), median reported, seed=42, `--no-enable-prefix-caching` Datasets: `sharegpt` (500 prompts, concurrency 64), `prefill_heavy` (random, in=2048/out=512, concurrency 32), `decode_heavy` (random, in=512/out=2048, concurrency 8). Model: `ibm-granite/granite-3.0-3b-a800m-instruct` (bf16). **td0 vs td1, mean output tok/s over N=3 runs** (mean used here instead of median — with only 3 samples per cell, one dataset's median flips sign depending on which single run is treated as the outlier; mean is more stable for this specific comparison. Raw per-run values are below, unchanged, so this is fully reproducible from the same numbers): | Platform | Dataset | td0 (pointer) | td1 (TD) | Δ | |---|---|---:|---:|---:| | B70 | sharegpt | 719.5 | 762.6 | **+6.0%** | | B70 | prefill_heavy | 420.1 | 417.6 | -0.6% | | B70 | decode_heavy | 134.0 | 133.3 | -0.5% | | B200 | sharegpt | 8181.1 | 8057.0 | -1.5% | | B200 | prefill_heavy | 6264.2 | 6094.0 | -2.7% | | B200 | decode_heavy | 2228.7 | 2144.7 | -3.8% | B70: a real throughput gain on `sharegpt` (mixed realistic prompt/response lengths, BMG's most representative workload), flat on the two synthetic datasets. B200: small negative deltas on prefill/decode-heavy, consistent with this hardware's already-documented TD-neutral-to-slightly- regressive pattern — this PR does not enable TD by default on CUDA, and this data supports that choice.
B70 ```json { \"sycl (default)\": { \"sharegpt\": [982.13, 994.13, 1065.42], \"prefill_heavy\": [640.0, 638.36, 633.31], \"decode_heavy\": [158.47, 158.87, 159.72] }, \"td0\": { \"sharegpt\": [599.93, 774.98, 783.72], \"prefill_heavy\": [418.45, 421.03, 420.96], \"decode_heavy\": [133.91, 134.13, 133.88] }, \"td1\": { \"sharegpt\": [685.9, 768.48, 833.28], \"prefill_heavy\": [418.19, 419.06, 415.58], \"decode_heavy\": [133.4, 133.4, 133.14] } } ``` Medians: sycl sharegpt=994.1, prefill_heavy=638.4, decode_heavy=158.9 | td0 sharegpt=775.0, prefill_heavy=421.0, decode_heavy=133.9 | td1 sharegpt=768.5, prefill_heavy=418.2, decode_heavy=133.4 tok/s.
H200 ```json { \"default\": { \"sharegpt\": [8003.47, 7249.66, 7635.22], \"prefill_heavy\": [5049.66, 4989.72, 4961.28], \"decode_heavy\": [2007.62, 1979.14, 1977.29] }, \"td0\": { \"sharegpt\": [7266.54, 7919.86, 7713.62], \"prefill_heavy\": [5140.22, 5040.31, 5046.53], \"decode_heavy\": [2009.16, 1969.85, 2007.99] } } ``` Medians: default sharegpt=7635.2, prefill_heavy=4989.7, decode_heavy=1979.1 | td0 sharegpt=7713.6, prefill_heavy=5046.5, decode_heavy=2008.0 tok/s. No `td1` (uncompilable on sm90).
B200 ```json { \"default (env-unset, still --moe-backend triton, kept for reference)\": { \"sharegpt\": [7351.53, 8372.63, 8720.87], \"prefill_heavy\": [6207.44, 6315.79, 6205.99], \"decode_heavy\": [2187.75, 2230.64, 2213.36] }, \"default_native (true no-flag default, FlashInfer TRTLLM)\": { \"sharegpt\": [9115.8, 9278.24, 9218.8], \"prefill_heavy\": [7542.21, 7465.44, 7468.28], \"decode_heavy\": [2981.51, 2979.89, 2979.25] }, \"td0\": { \"sharegpt\": [7803.92, 7989.08, 8750.16], \"prefill_heavy\": [6224.09, 6285.98, 6282.4], \"decode_heavy\": [2242.47, 2216.71, 2226.98] }, \"td1\": { \"sharegpt\": [7640.3, 7953.87, 8576.79], \"prefill_heavy\": [6092.19, 6106.03, 6083.66], \"decode_heavy\": [2139.16, 2157.93, 2137.13] } } ``` Medians: `default_native` (FlashInfer, true platform default) sharegpt=9218.8, prefill_heavy=7468.3, decode_heavy=2979.9 | td0 sharegpt=7989.1, prefill_heavy=6282.4, decode_heavy=2227.0 | td1 sharegpt=7953.9, prefill_heavy=6092.2, decode_heavy=2139.2 tok/s. Collected with `--no-enable-flashinfer-autotune` (see attached report §4.5 for an unrelated FlashInfer autotuner crash found and root-caused during this measurement — not a PR#42436 issue).
### Attachments [pr-42436-multiplatform-validation.zip](https://github.com/user-attachments/files/30166403/pr-42436-multiplatform-validation.zip) Attached as a single zip (`pr-42436-multiplatform-validation.zip`) containing: - `TRITONXPU-181_production_readiness_campaign_2026-07-18.md` — full campaign report. - Per-platform raw logs (`b70/`, `h200/`, `b200/`), each split into `accuracy/`, `perf/`, `quant_smoke/`, `kernel_bench/`, `result_json/` — every command's full stdout/stderr/meta.json, no empty directories.", "url": "https://github.com/vllm-project/vllm/pull/42436", "createdAt": "2026-05-12T16:15:09Z", "updatedAt": "2026-08-13T14:49:09Z", "timestamp": "2026-08-13T14:49:09Z", "metrics": { "reactions": 4, "comments": 10 }, "labels": [ "ready" ], "author": "afierka-intel", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:42662", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[LoRA][Gemma4] Support vision tower LoRA", "text": "This PR adds the remaining LoRA plumbing needed for Gemma4 multimodal LoRA support. After #43798, Gemma4-MM vision linear layers are already converted through the Transformers backend path, so this PR no longer reimplements the Gemma4 vision tower. Instead, it focuses on the runtime LoRA mapping and token-counting pieces needed by Gemma4 image/video/audio inputs. Main changes: - Add a multimodal LoRA token-count interface so models can report separate tower and connector token counts. - Update Gemma4-MM to report modality-specific LoRA token counts for image, video, and audio inputs. - Size multimodal LoRA wrappers using the largest tower/connector token budget across modalities. FIX https://github.com/vllm-project/vllm/issues/40693 ## Test Plan Additional end-to-end tests for real Gemma4 vision LoRA adapters should also be added in a follow-up. ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42662", "createdAt": "2026-05-14T16:37:41Z", "updatedAt": "2026-08-13T14:07:44Z", "timestamp": "2026-08-13T14:07:44Z", "metrics": { "reactions": 4, "comments": 17 }, "labels": [ "documentation", "ready", "ci/build", "v1", "mrv2" ], "author": "linitra24", "state": "closed", "assignees": [ "jeejeelee" ] }, { "id": "github:vllm-project/vllm:pull_request:42773", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] laguna xs gsm8k test", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42773", "createdAt": "2026-05-15T19:41:52Z", "updatedAt": "2026-08-13T17:26:20Z", "timestamp": "2026-08-13T17:26:20Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "ready", "needs-rebase" ], "author": "mgoin", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:42785", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MM][CG] Enable encoder CUDA Graph for MiniCPM-V", "text": "## Purpose Add encoder CUDA Graph support for MiniCPM-V 2.5, 2.6, 4.0 as part of tracker #38175. This implementation follows the existing workflow introduced in #38061. The captured graph covers both the ViT encoder (VPM) and the resampler. MiniCPM-V 2.0 is not included, as it predates the slice-based vision architecture required by this implementation. MiniCPM‑V 4.5 is not included, as its dynamic frame fusion introduces input-dependent encoder shapes that are not compatible with the current static-shape CUDA Graph capture mechanism. It continues to use the eager path and does not benefit from encoder CUDA Graph replay, which may reduce performance. ## Key Updates This PR extends the multimodal encoder CUDA Graph cache to key captured graphs by` (token_budget, secondary_capture_axis_key)` instead of `token_budget` alone. `token_budget` continues to determine greedy packing and output buffer sizing for resampler tokens. However, for vision encoders such as those in OpenBMB's MiniCPM-V models, the encoder capture shape also depends on discrete patch-grid / pixel-layout tiers. As a result, different input shapes can share the same token budget while requiring distinct CUDA Graph captures. Without this secondary capture axis, inputs with the same token budget are forced into a single capture shape, which can introduce unnecessary padding and reduce the efficiency benefits of CUDA Graphs. By including `secondary_capture_axis_key `in the cache key, the encoder can maintain separate captures for shape-dependent tiers while preserving the existing token-budget-based scheduling and memory allocation logic. ## Test Plan **Unit test** ```bash pytest tests/v1/cudagraph/test_encoder_cudagraph.py -v ``` **Functional test** GPU: RTX 5090 Model: MiniCPM-V-4_0 No CUDA Graph ```bash vllm serve /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --served-model-name MiniCPM-V-4 \\ --gpu-memory-utilization 0.75 \\ --max-model-len 4096 \\ --max-num-batched-tokens 4096 \\ --limit-mm-per-prompt '{\"video\": 1, \"image\": 1}' ``` With CUDA Graph ```bash vllm serve /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --served-model-name MiniCPM-V-4 \\ --gpu-memory-utilization 0.75 \\ --max-model-len 4096 \\ --max-num-batched-tokens 4096 \\ --limit-mm-per-prompt '{\"video\": 1, \"image\": 1}' \\ --compilation-config '{ \"cudagraph_mm_encoder\": true, \"encoder_cudagraph_token_budgets\": [1024], \"encoder_cudagraph_max_vision_items_per_batch\": 32 }' ``` **benchmark** GPU: RTX 5090 Model: MiniCPM-V-2_6 / MiniCPM-V-4_0 ```bash vllm bench mm-processor \\ --model /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --tokenizer-mode slow \\ --dataset-name random-mm \\ --num-prompts 500 \\ --num-warmups 100 \\ --max-model-len 4096 \\ --seed 42 \\ --gpu-memory-utilization 0.8 \\ --random-mm-base-items-per-request 8 \\ --random-mm-num-mm-items-range-ratio 0.0 \\ --random-mm-bucket-config '{\"(448, 448, 1)\": 1.0}' \\ --compilation-config '{ \"cudagraph_mm_encoder\": true, \"encoder_cudagraph_token_budgets\": [256, 512], \"encoder_cudagraph_max_vision_items_per_batch\": 8 }' ``` ## Test Result ✅ **Unit test** ```bash 36 passed, 16 warnings in 7.11s ``` ✅**Functional test** Image ```bash # No CUDA Graph -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying an image of a ship and some text in Chinese. -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying what appears to be a news broadcast about an aircraft carrier. -------------------------------------------------- # With CUDA Graph -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying an image of a ship and some text in Chinese. -------------------------------------------------- A woman in a green t-shirt and denim shorts standing in a room with a television screen in the background displaying what appears to be a news broadcast about an aircraft carrier. -------------------------------------------------- ``` Video ```bash # No CUDA Graph -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with a mix of dark and light clouds, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, creating a bright line against the dimly lit sky. The reflection of the train's lights shimmers on the water's surface below, adding to the tranquil atmosphere. As the train progresses, the background reveals faint outlines of industrial structures, possibly factories or power plants, with their silhouettes barely visible against the horizon. The overall ambiance is calm and quiet, with the train's movement being the primary action in the scene. -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with varying shades of gray, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, casting a reflection on the water below. As the train progresses, the lights on the train create a shimmering effect on the water's surface. In the background, faint outlines of industrial structures can be seen, adding an urban element to the otherwise natural landscape. The overall ambiance of the video is calm and tranquil, with the train's movement providing a subtle dynamic element to the scene. -------------------------------------------------- # With CUDA Graph -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with a mix of dark and light clouds, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, creating a bright line against the dimly lit sky. The reflection of the train's lights shimmers on the water's surface below, adding to the tranquil atmosphere. As the train progresses, the background reveals faint outlines of industrial structures, possibly factories or power plants, with their silhouettes barely visible against the horizon. The overall ambiance is calm and quiet, with the train's movement being the primary action in the scene. -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with varying shades of gray, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, casting a reflection on the water below. As the train progresses, the lights on the train create a shimmering effect on the water's surface. In the background, faint outlines of industrial structures can be seen, adding an urban element to the otherwise natural landscape. The overall ambiance of the video is calm and tranquil, with the train's movement providing a subtle dynamic element to the scene. -------------------------------------------------- ``` ✅ **Benchmark:** > Single RTX 5090 · random-mm · 500 prompts / 100 warmups Benchmark results for MiniCPM-V 2.6 and 4.0 show no measurable regression. Latency remains within normal run-to-run variance, with small improvements observed in some runs. | Version | Backend | Mean | P99 | |------|------|------|------| | MiniCPM-V-2_6 | FLASH_ATTN2 | +0.60% (44555.14→44286.37ms) | +0.10% (67712.77ms→67644.05ms) | | MiniCPM-V-4 | FLASH_ATTN2 | +2.01% (44723.41→43822.66ms) | +2.69% (84580.66ms→82305.93ms) | **MiniCPM-V-2_6** No CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.03 0.39 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.16 29.88 1.43 35.51 merge_mm_kwargs_ms 0.44 0.41 0.08 0.63 apply_prompt_updates_ms 6.15 6.39 0.74 7.54 preprocessor_total_ms 37.13 36.90 1.77 42.96 encoder_forward_ms 64.14 63.93 14.47 111.22 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 544 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 44723.41 Median 43812.82 Std 13619.35 P99.0 84580.66 ``` With CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.02 0.38 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.12 29.93 1.31 34.74 merge_mm_kwargs_ms 0.44 0.41 0.06 0.55 apply_prompt_updates_ms 6.16 6.40 0.73 7.38 preprocessor_total_ms 37.09 36.92 1.55 43.03 encoder_forward_ms 62.67 61.63 14.23 113.54 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 543 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 43822.66 Median 42867.49 Std 13261.40 P99.0 82305.93 ``` **MiniCPM-V-4_0** No CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.03 0.39 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.16 29.88 1.43 35.51 merge_mm_kwargs_ms 0.44 0.41 0.08 0.63 apply_prompt_updates_ms 6.15 6.39 0.74 7.54 preprocessor_total_ms 37.13 36.90 1.77 42.96 encoder_forward_ms 64.14 63.93 14.47 111.22 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 544 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 44723.41 Median 43812.82 Std 13619.35 P99.0 84580.66 ``` With CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.02 0.38 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.12 29.93 1.31 34.74 merge_mm_kwargs_ms 0.44 0.41 0.06 0.55 apply_prompt_updates_ms 6.16 6.40 0.73 7.38 preprocessor_total_ms 37.09 36.92 1.55 43.03 encoder_forward_ms 62.67 61.63 14.23 113.54 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 543 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 43822.66 Median 42867.49 Std 13261.40 P99.0 82305.93 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42785", "createdAt": "2026-05-15T21:19:04Z", "updatedAt": "2026-08-13T07:37:22Z", "timestamp": "2026-08-13T07:37:22Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "documentation", "v1", "multi-modality", "llama", "qwen", "deepseek", "nvidia", "kimi" ], "author": "YunzhuLu", "state": "open", "assignees": [ "shen-shanshan" ] }, { "id": "github:vllm-project/vllm:pull_request:42792", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[WIP][Model Runner V2] support spec decode + mamba align prefix caching", "text": "## Accuracy Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' \\ --default-chat-template-kwargs '{\"enable_thinking\": false}' ``` **Results** ``` | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |---------|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k_cot| 3|flexible-extract| 8|exact_match|↑ |0.7998|± |0.0110| | | |strict-match | 8|exact_match|↑ |0.7968|± |0.0111| ``` ## Performance Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' ``` **Results** | Metric | #42406 | This PR | Delta | |--------|-----------|-----------|-------| | **General** | | | | | Successful requests | 512 | 512 | — | | Failed requests | 0 | 0 | — | | Max request concurrency | 16 | 16 | — | | Benchmark duration (s) | 119.40 | 113.97 | -4.5% | | Total input tokens | 2,167,806 | 2,271,087 | +4.8% | | Total generated tokens | 262,144 | 262,144 | — | | Request throughput (req/s) | 4.29 | 4.49 | +4.7% | | Output token throughput (tok/s) | 2,195.48 | 2,300.20 | +4.8% | | Peak output token throughput (tok/s) | 1,553.00 | 1,040.00 | -33.0% | | Peak concurrent requests | 28.00 | 28.00 | — | | Total token throughput (tok/s) | 20,351.05 | 22,227.97 | +9.2% | | **Time to First Token** | | | | | Mean TTFT (ms) | 282.49 | 105.66 | -62.6% | | Median TTFT (ms) | 145.19 | 84.13 | -42.1% | | P99 TTFT (ms) | 3,800.32 | 995.73 | -73.8% | | **Time per Output Token** | | | | | Mean TPOT (ms) | 6.64 | 6.68 | +0.6% | | Median TPOT (ms) | 6.59 | 6.61 | +0.3% | | P99 TPOT (ms) | 10.03 | 7.95 | -20.7% | | **Inter-token Latency** | | | | | Mean ITL (ms) | 16.35 | 17.04 | +4.2% | | Median ITL (ms) | 10.58 | 15.83 | +49.6% | | P99 ITL (ms) | 105.52 | 38.90 | -63.1% | | **End-to-end Latency** | | | | | Mean E2EL (ms) | 3,677.74 | 3,521.25 | -4.3% | | Median E2EL (ms) | 3,528.08 | 3,468.17 | -1.7% | | P99 E2EL (ms) | 8,904.19 | 4,462.88 | -49.9% | | **Speculative Decoding** | | | | | Acceptance rate (%) | 73.88 | 78.34 | +4.46pp | | Acceptance length | 2.48 | 2.57 | +3.6% | | Drafts | 105,785 | 102,119 | -3.5% | | Draft tokens | 211,570 | 204,238 | -3.5% | | Accepted tokens | 156,310 | 160,003 | +2.4% | | Position 0 acceptance (%) | 83.86 | 85.68 | +1.82pp | | Position 1 acceptance (%) | 63.90 | 71.00 | +7.10pp |", "url": "https://github.com/vllm-project/vllm/pull/42792", "createdAt": "2026-05-15T22:55:32Z", "updatedAt": "2026-08-13T17:08:05Z", "timestamp": "2026-08-13T17:08:05Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "TheEpicDolphin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:42963", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunnerV2] Support prompt embeds", "text": "## Purpose Support prompt embeds for ModelRunnerV2. ## Test Plan ```bash VLLM_USE_V2_MODEL_RUNNER=1 pytest -sv tests/basic_correctness/test_basic_correctness.py::test_models -k \"True-uni or True-mp\" ``` Before ```bash E pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig E Value error, VLLM_USE_V2_MODEL_RUNNER does not yet support: prompt embeds [type=value_error, input_value=ArgsKwargs((), {'model_co... 'shutdown_timeout': 0}), input_type=ArgsKwargs] E For further information visit https://errors.pydantic.dev/2.13/v/value_error vllm/engine/arg_utils.py:2171: ValidationError ============================================================ warnings summary ============================================================ :488 :488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute :488 :488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute ../.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings /root/vllm-workspace/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================================================== short test summary info ========================================================= FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig ======================================== 8 failed, 8 deselected, 16 warnings in 237.83s (0:03:57) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` After ```bash ======================================== 8 passed, 8 deselected, 16 warnings in 443.73s (0:07:23) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ``` vllm serve --enable-prompt-embeds ``` ``` \"\"\"Smoke test for prompt_embeds over the OpenAI-compatible HTTP server. Usage: # Terminal 1 (server): see vllm serve command in the chat. # Terminal 2: .venv/bin/python test_embeds_serve.py \"\"\" import io import openai import pybase64 as base64 import torch from transformers import AutoModelForCausalLM, AutoTokenizer MODEL_PATH = \"/root/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B\" SERVED_NAME = \"Qwen/Qwen3-0.6B\" # must match --served-model-name BASE_URL = \"http://localhost:8000/v1\" PROMPT = \"The capital of France is\" def to_b64_embed(tensor: torch.Tensor) -> str: buf = io.BytesIO() # torch.save is what the server expects (it calls torch.load on the bytes). torch.save(tensor, buf) return base64.b64encode(buf.getvalue()).decode(\"utf-8\") def main() -> None: tok = AutoTokenizer.from_pretrained(MODEL_PATH) hf = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16) with torch.no_grad(): ids = tok(PROMPT, return_tensors=\"pt\").input_ids embeds = ( hf.get_input_embeddings()(ids) .squeeze(0) .to(torch.bfloat16) .cpu() .contiguous() ) del hf encoded = to_b64_embed(embeds) client = openai.OpenAI(base_url=BASE_URL, api_key=\"EMPTY\") # Case 1: prompt_embeds only. out = client.completions.create( model=SERVED_NAME, prompt=None, # leave empty so the server falls through to prompt_embeds max_tokens=16, temperature=0.0, extra_body={\"prompt_embeds\": encoded}, ) print(f\"[prompt_embeds] {out.choices[0].text!r}\") # Case 2: same prompt via text path, for sanity comparison. out_text = client.completions.create( model=SERVED_NAME, prompt=PROMPT, max_tokens=16, temperature=0.0, ) print(f\"[text ] {out_text.choices[0].text!r}\") if out.choices[0].text == out_text.choices[0].text: print(\"MATCH: prompt_embeds output equals text output\") else: print(\"DIVERGE: outputs differ (expected only if tokenizer/embedding \" \"scaling differs from raw lookup)\") if __name__ == \"__main__\": main() ``` ## Test Result ``` python test_embeds_serve.py [transformers] `torch_dtype` is deprecated! Use `dtype` instead! Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 6652.97it/s] [prompt_embeds] ' Paris. The capital of France is also the capital of the Republic of France.' [text ] ' Paris. The capital of France is also the capital of the Republic of France.' MATCH: prompt_embeds output equals text output ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42963", "createdAt": "2026-05-18T10:04:55Z", "updatedAt": "2026-08-13T16:25:01Z", "timestamp": "2026-08-13T16:25:01Z", "metrics": { "reactions": 1, "comments": 13 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "gcanlin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Support spec decode with draft model", "text": "## Purpose Support spec decode with draft models in model runner v2 ## Test Plan ``` python examples/features/speculative_decoding/spec_decode_offline.py \\ --method draft_model \\ --model-dir /shared/models/modelscope/models/Qwen/Qwen3-32B \\ --draft-model /shared/models/modelscope/models/Qwen/Qwen3-1.7B \\ --num-spec-tokens 3 \\ --dataset-name hf \\ --dataset-path philschmid/mt-bench \\ --num-prompts 400 \\ --temp 0 \\ --output-len 256 \\ --max-model-len 10000 ``` ## Test Result **Model Runner V1:** ``` -------------------------------------------------- total_num_output_tokens: 102355 num_drafts: 36593 num_draft_tokens: 109779 num_accepted_tokens: 65181 mean acceptance length: 2.78 -------------------------------------------------- acceptance at token 0: 0.75 acceptance at token 1: 0.57 acceptance at token 2: 0.45 ``` **Model Runner V2:** ``` -------------------------------------------------- total_num_output_tokens: 102312 num_drafts: 36394 num_draft_tokens: 109182 num_accepted_tokens: 65379 mean acceptance length: 2.80 -------------------------------------------------- acceptance at token 0: 0.76 acceptance at token 1: 0.58 acceptance at token 2: 0.46 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/43091", "createdAt": "2026-05-19T11:35:21Z", "updatedAt": "2026-08-13T16:25:24Z", "timestamp": "2026-08-13T16:25:24Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "ready", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "wxsIcey", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43107", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][WIP] Check for GPU<->CPU sync during CI", "text": "vLLM now uses asynchronous scheduling by default and in the majority of cases. Performance relies on the absence of any gpu<->cpu synchronizations on the main cuda stream, but such syncs can be opaque and it is easy for them to creep in accidentally. This change adds a `VLLM_GPU_SYNC_CHECK` env var which enables `torch.cuda.set_sync_debug_mode` for the model forward pass and sampler, so that we can easily check for such syncs. I'm trying first to enable it globally in the CI to flush out syncs that need to be fixed or where they are unavoidable and the check needs to be suppressed. A new `gpu_sync_allowed()` context manager can be used to wrap known syncs (i.e. temporary or unavoidable). This PR replaces https://github.com/vllm-project/vllm/pull/40561 - rebased now that man of the exposed \"low hanging\" sync fixes have been merged to main.", "url": "https://github.com/vllm-project/vllm/pull/43107", "createdAt": "2026-05-19T15:45:29Z", "updatedAt": "2026-08-13T17:34:59Z", "timestamp": "2026-08-13T17:34:59Z", "metrics": { "reactions": 3, "comments": 23 }, "labels": [ "rocm", "speculative-decoding", "ready", "ci/build", "v1", "multi-modality", "qwen", "kv-connector", "nvidia", "ready-run-all-tests", "quantization", "mistral", "cohere" ], "author": "njhill", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43249", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix MRV2 Gumbel sampling for non-finite logits", "text": "## Summary Fix MRV2 Gumbel sampling when the candidate logits contain `-inf` entries. The current Gumbel path only clamps the uniform draw away from zero. If the draw reaches the upper endpoint after floating-point rounding, the Gumbel noise becomes `inf`. Adding that noise to a masked/non-candidate `-inf` logit produces `NaN`, and a subsequent Triton reduction can return an invalid token index. This change: - clamps fp32/fp64 uniform draws away from both `0` and `1`; - only adds Gumbel noise to finite logits; - adds a CUDA/Triton regression test that deterministically reproduces the `-inf + inf -> NaN` edge case. ## Why this matters We hit this downstream in a Kimi K2.6 MRV2 speculative decoding deployment with DCP8 and Eagle/MTP. The failure surfaced during spec-decode/logprobs warmup as an invalid sampled token id and then a CUDA illegal memory access: ```text Invalid sampled token ids for spec logprobs: count=1 vocab=163840 idx=[508] vals=[9223372034707292159] ``` The DCP8/Kimi setup makes the bug easy to trigger, but the root cause is not DCP-specific. It is the generic Gumbel sampler being asked to sample from residual logits where rejected tokens are represented as `-inf`. A minimal deterministic reproduction of the arithmetic issue is: ```python logits = -inf u = 1.0 gumbel_noise = -log(-log(u)) # inf logits + gumbel_noise # NaN ``` After this patch, the same case stays `-inf` and does not poison the reduction. ## Tests ```text python3 - <<'PY' import importlib.util path = \"tests/v1/sample/test_gumbel.py\" spec = importlib.util.spec_from_file_location(\"test_gumbel_direct\", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(False) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(True) print(\"direct test passed\") PY python3 -m py_compile vllm/v1/worker/gpu/sample/gumbel.py tests/v1/sample/test_gumbel.py git diff --check ``` I could not run the repository pytest entrypoint in this local container because `tests/conftest.py` imports `tblib`, which is not installed in the environment: ```text ModuleNotFoundError: No module named 'tblib' ```", "url": "https://github.com/vllm-project/vllm/pull/43249", "createdAt": "2026-05-20T20:58:39Z", "updatedAt": "2026-08-13T16:25:59Z", "timestamp": "2026-08-13T16:25:59Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "voipmonitor", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43327", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Add per-call decode budget to sparse-MLA indexer", "text": "## Purpose The ROCm AITER sparse-MLA indexer decode path allocates a fresh fp32 logits tensor on every call to `rocm_fp8_paged_mqa_logits`: `(heads, B*next_n, max_model_len)` on the stage1 3D path, or `(B*next_n, max_model_len)` on the gfx942/gfx950 fused 2D path. With long context and a large running batch the per-call working set is multi-GiB; across ~60 layers per decode forward the caching allocator fragments and OOMs. Prefill had the same problem and got `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` plus chunking in #36178. This is the decode mirror. Two pieces, decode-only, ROCm-only: - the fp32 logits buffer is now drawn from a per-process workspace keyed by (device, shape) and refilled with `-inf` between calls, so no per-layer alloc/free of multi-GiB tensors; - `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` (default 4096) bounds the per-call working set. When a single call would exceed the budget the batch is split into sub-batches that fit and copied into the full output. The aiter kernel does not consume `schedule_metadata` on ROCm, so the metadata doesn't need to be re-derived per chunk. Set to 0 to restore the legacy unbounded behaviour. Prefill is not touched. The CUDA path is not touched (this file only runs on ROCm via the AITER sparse-MLA backend). Both `_ON_GFX942` and `_ON_GFX950` are covered, plus the stage1 3D else-branch that older arches still hit. Related: #36178 is the prefill-side mirror (same shape of fix, opposite forward-pass phase). #41002 is in flight in the same area, happy to rebase on top of it after it merges. ## Test Plan 1. Numeric equivalence test added at `tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py`. Skipped on non-ROCm and when the AITER `paged_mqa_logits` module is not importable, so it sits with the rest of the ROCm-only kernel tests. On a ROCm gate it asserts the chunked path returns bit-identical logits to the full-batch call. ```bash pytest -s -v tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py ``` 2. Synthetic micro-bench on a single MI355X (gfx950), 60 back-to-back calls into `rocm_fp8_paged_mqa_logits` with a pre-allocated paged kv at `B=1024, next_n=1, heads=64, head_dim=128, block_size=64, max_model_len=131072`. Stress shape, picked so the 3D per-call working set is well above any reasonable budget. Measured peak `torch.cuda.max_memory_allocated()` minus the paged-kv baseline, sweeping `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` against the legacy code. 3. End-to-end re-run of the GLM-5.1-FP8 customer repro that motivated this: ```bash vllm bench serve --random-input-len 1 --random-output-len 2 --num-prompts 1024 \\ --ignore-eos --temperature 0 --seed 0 \\ --gpu-memory-utilization 0.9 -tp 8 --enforce-eager ``` ## Test Result Indexer working set at the high-water mark, 60-layer loop: | setting | indexer working set @ peak | | -------------------------------------------------- | -------------------------: | | legacy (no patch) | 33792 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` unset (= 4096) | ~4160 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=2048` | ~2080 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=1024` | ~1040 MiB | The 33.8 GiB legacy delta is exactly `heads * B * next_n * max_model_len * 4` bytes for the 3D stage1 buffer; chunking brings it down to `chunk_b * heads * next_n * max_model_len * 4` plus a one-shot 2D accumulator. Wall time across all four settings stays between 1.09 and 1.24 s for the 60-call loop, the only overhead is one `copy_` per chunk into the persistent output buffer. End-to-end on GLM-5.1-FP8, the decode-time HIP OOM that originally fired at ~24 GiB per layer no longer occurs; 1017/1024 prompts succeed, the 7 remaining failures are bench-client side (\"never received a valid chunk\") and clear with `--max-concurrency 256`.", "url": "https://github.com/vllm-project/vllm/pull/43327", "createdAt": "2026-05-21T15:29:18Z", "updatedAt": "2026-08-13T17:31:29Z", "timestamp": "2026-08-13T17:31:29Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "rocm", "v1" ], "author": "reger-men", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43449", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Formatting] Collapse multi-line arg lists where possible", "text": "This is a very large PR because it adjusts a ruff formatting rule which affects a lot of files. This might be partially personal preference but I find it much cleaner when lists / method defs are collapsed onto a single line when they fit. It reduces the lines of code and how much vertical real estate you have to scan to understand things. And just looks cleaner imo. I can understand the case for vertically formatted lists in that it's easier to see quickly the length and delineated items, but feel that the cons outweigh the pros. This formatting change reduces vLLM LoC by 32,000. The actual change is to `pyproject.toml`, adding `skip-magic-trailing-comma = true`: ```toml [tool.ruff.format] docstring-code-format = true skip-magic-trailing-comma = true [tool.ruff.lint.isort] split-on-trailing-comma = false ```` Anyhow please feel free to vote on this! P.S. I guess merging this all in one go may introduce a lot of conflicts to existing PRs, so perhaps we could apply incrementally across the codebase.", "url": "https://github.com/vllm-project/vllm/pull/43449", "createdAt": "2026-05-22T20:57:47Z", "updatedAt": "2026-08-13T17:32:13Z", "timestamp": "2026-08-13T17:32:13Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "needs-rebase", "ci/build", "v1", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "rust", "kimi", "cohere" ], "author": "njhill", "state": "open", "assignees": [ "sangstar" ] }, { "id": "github:vllm-project/vllm:pull_request:43642", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Warm up hybrid GDN/Mamba/MRoPE kernels", "text": "## Purpose Hybrid Qwen-style GDN/Mamba/MRoPE models can trigger Triton compilation during the first inference request when startup warmup does not cover the exact runtime compile keys. This revision migrates the remaining warmups to the shared kernel-owned warmup contract and removes coverage that is already provided by current upstream `main`. ## Changes - Merge the latest upstream `main` and remove duplicate warmups for zero-KV blocks, slot mapping, causal convolution, fused post-convolution, fused sigmoid update, and chunked GDN prefill. - Add kernel-owned `VllmJitKernel` implementations for packed recurrent GDN decode and MRoPE. - Route runtime launches and warmup compilation through the same owners and exact `CompileKey` definitions. - Use compile-only warmup calls under `enable_jit_warmup`; no dummy model execution or startup synchronization is added. - Cover the integer specialization buckets used by MRoPE and the static meta-parameters used by packed GDN decode. ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/model_executor/test_hybrid_gdn_mamba_mrope_warmup.py -q CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest \\ tests/kernels/core/test_mrope.py -q pre-commit run --files \\ vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py \\ vllm/model_executor/layers/rotary_embedding/mrope.py \\ vllm/model_executor/warmup/hybrid_gdn_mamba_mrope_warmup.py \\ vllm/model_executor/warmup/kernel_warmup.py \\ vllm/third_party/flash_linear_attention/ops/fused_recurrent.py \\ tests/model_executor/test_hybrid_gdn_mamba_mrope_warmup.py ``` GPU compile-only and direct numerical smoke tests were run on GPU 1 (RTX 3090). ## Test Result - Shared-contract unit tests: 3 passed. - MRoPE native-versus-CUDA tests: 12 passed. - Compile-only smoke compiled both MRoPE keys and the packed recurrent GDN key. - Direct packed recurrent GDN runtime comparison had maximum absolute error `0.007812`, within the existing test tolerance. - All applicable pre-commit hooks, including mypy, passed. The existing packed recurrent numerical pytest still fails for fp16/bf16 on both this branch and an unmodified latest-upstream worktree; fp32 passes. This is therefore not introduced by this revision. AI assistance: Codex.", "url": "https://github.com/vllm-project/vllm/pull/43642", "createdAt": "2026-05-26T04:51:32Z", "updatedAt": "2026-08-13T17:42:29Z", "timestamp": "2026-08-13T17:42:29Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "needs-rebase", "v1" ], "author": "lesj0610", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:43671", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][MiniMax-M2] Fix EAGLE-3 aux hidden-state layer off-by-one", "text": "## Summary When EAGLE-3 speculative decoding is enabled for `MiniMaxAI/MiniMax-M2.5`, the auxiliary hidden states fed to the draft head are taken from the wrong decoder layer, causing position-0 acceptance to collapse to ~10% (vs ~45% expected) and making MTP a throughput *regression* vs no speculative decoding. The root cause is an off-by-one between the two halves of the existing EAGLE-3 plumbing: - `MiniMaxM2Model.forward` (already in vLLM) captures aux state AFTER each decoder layer: ```python for idx, layer in enumerate(islice(self.layers, ...)): hidden_states, residual = layer(positions, hidden_states, residual) self._maybe_add_hidden_state(aux_hidden_states, idx + 1, hidden_states, residual) ``` So a stored index of `k` captures the activation AFTER layer `k-1` ran. - The default `SupportsEagle3.set_aux_hidden_state_layers` passes the draft-config indices straight through: ```python parent_ref.model._set_aux_hidden_state_layers(layers) ``` Net effect: a config that asks for \"layer k\" gets \"output of layer k-1\". The SGLang reference implementation -- which is what the upstream `thoughtworks/MiniMax-M2.5-Eagle3` draft head was trained against -- compensates for the very same offset in [`set_eagle3_layers_to_capture`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/minimax_m2.py): ```python self.model.layers_to_capture = [val + 1 for val in layer_ids] ``` This PR overrides `set_aux_hidden_state_layers` on `MiniMaxM2ForCausalLM` to apply the same +1 shift, so vLLM and SGLang capture identical auxiliary tensors for the same draft config. ## Test plan Bench: `MiniMaxAI/MiniMax-M2.5` + `thoughtworks/MiniMax-M2.5-Eagle3`, TP=4 on AMD MI350, `quantization=fp8`, `attention-backend=ROCM_AITER_UNIFIED_ATTN`, random in=1000 / out=100, `num_speculative_tokens=3`, `method=eagle3`, `temperature=0`, `ignore_eos=true`, `max_concurrency=16`, num_warmups=8, num_prompts=64. | Metric | Before fix | After fix | |------------------------------|-----------:|----------:| | Pos-0 acceptance | ~10 % | 46.5 % | | Pos-1 acceptance | ~5 % | 27.8 % | | Pos-2 acceptance | ~2 % | 16.3 % | | Mean acceptance length | ~1.1 | 1.90 | | Total tok/s (MTP / no-MTP) | ~0.9× | 1.0×* at conc=16; up to 1.73× at conc=1 | ## Notes - The fix is intentionally local to `MiniMaxM2ForCausalLM` rather than changing the `EagleModelMixin._maybe_add_hidden_state` indexing scheme, because other EAGLE-3-supporting models in the tree may already rely on the existing \"after-layer\" indexing convention. - An alternative would be to align `MiniMaxM2Model.forward` with SGLang's \"before-layer\" capture point instead of overriding the setter; both are equivalent in semantics but the override is the minimum-surface change.", "url": "https://github.com/vllm-project/vllm/pull/43671", "createdAt": "2026-05-26T13:15:58Z", "updatedAt": "2026-08-13T06:26:22Z", "timestamp": "2026-08-13T06:26:22Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "ready", "needs-rebase", "v1", "mrv2" ], "author": "pbkowalski", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44298", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for write_zeros_to_output MoE kernel", "text": "Adds a unit test for `write_zeros_to_output` (`vllm/model_executor/layers/fused_moe/fused_moe.py`), part of RFC #48480. The function zeroes the output tile when a block's expert is not on the current EP rank (`off_experts == -1`). It is a `@triton.jit` device function, so a launcher kernel wraps it and results are compared against a PyTorch reference. 4 tests / 11 cases cover the token mask, padding sentinel ids, a partial column tail and a strided output view. Test-only change. ## Test Plan ``` pytest tests/kernels/moe/test_write_zeros_to_output.py ``` ## Test Result All 11 tests passed", "url": "https://github.com/vllm-project/vllm/pull/44298", "createdAt": "2026-06-02T07:57:22Z", "updatedAt": "2026-08-13T07:09:18Z", "timestamp": "2026-08-13T07:09:18Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44343", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DO NOT MERGE][Perf][Feat] Add generic cuteDSL LL FP32 router (GEMM)", "text": "Depends on: https://github.com/vllm-project/vllm/pull/50174 ## Motivation This kernel is intended to be a generic low-latency fp32-router kernel. We should not expect it to outperform highly specialized PTX/CUDA kernels tuned for one exact router shape. The goal is different: make the CuTeDSL implementation competitive with those specialized kernels while keeping the code flexible enough to support new router shapes without writing a new kernel each time. That flexibility matters because router shapes keep changing across new models. Without a dedicated shape-specific CUDA path, the fallback is typically cuBLAS/`torch.mm`, and the kernel-wise results show why a generic low-latency router kernel is still useful: even when it only ties or slightly trails the specialized dispatch kernel, it is dramatically faster than cuBLAS across these low-M router workloads. ## Kernel-level results Speedup is computed as `baseline_us / ll_fp32w_us`; values above `1.0x` mean `ll_fp32w` is faster. `dispatch` == `fp32_router_gemm_dispatch` ### MiniMax-M3 `(K=6144, N=128)` | M | Config | ll_fp32w us | dispatch us | cuBLAS us | Speedup vs dispatch | Speedup vs cuBLAS | |---:|---|---:|---:|---:|---:|---:| | 1 | `(384,1,1)` | 1.729 | 1.722 | 6.924 | 0.996x | 4.005x | | 2 | `(384,1,1)` | 1.825 | 1.856 | 13.618 | 1.017x | 7.462x | | 3 | `(384,1,1)` | 1.958 | 2.033 | 13.709 | 1.038x | 7.002x | | 4 | `(384,1,1)` | 2.172 | 2.212 | 13.920 | 1.018x | 6.409x | | 5 | `(384,1,1)` | 2.301 | 2.349 | 13.885 | 1.021x | 6.034x | | 6 | `(384,2,1)` | 2.422 | 2.565 | 15.746 | 1.059x | 6.501x | | 7 | `(256,1,1)` | 2.580 | 2.655 | 15.743 | 1.029x | 6.102x | | 8 | `(384,2,1)` | 2.614 | 2.813 | 14.011 | 1.076x | 5.360x | | 9 | `(384,1,1)` | 2.956 | 3.021 | 14.117 | 1.022x | 4.776x | | 10 | `(384,2,1)` | 2.983 | 3.085 | 14.107 | 1.034x | 4.729x | | 11 | `(384,1,1)` | 3.212 | 3.195 | 14.229 | 0.995x | 4.430x | | 12 | `(192,2,1)` | 2.964 | 3.158 | 14.238 | 1.065x | 4.804x | | 13 | `(384,1,1)` | 3.374 | 3.477 | 14.255 | 1.031x | 4.225x | | 14 | `(192,2,1)` | 3.256 | 3.290 | 14.300 | 1.010x | 4.392x | | 15 | `(256,1,1)` | 3.683 | 3.662 | 14.311 | 0.994x | 3.886x | | 16 | `(192,2,1)` | 3.529 | 3.493 | 14.393 | 0.990x | 4.078x | | 17 | `(384,1,1)` | 3.891 | 3.919 | 15.516 | 1.007x | 3.988x | | 18 | `(192,2,1)` | 3.635 | 3.669 | 15.678 | 1.009x | 4.313x | | 19 | `(256,1,1)` | 4.018 | 4.074 | 15.735 | 1.014x | 3.916x | | 20 | `(192,2,1)` | 3.900 | 3.937 | 15.851 | 1.009x | 4.064x | | 21 | `(256,1,1)` | 4.273 | 4.317 | 15.826 | 1.010x | 3.704x | | 22 | `(192,2,1)` | 4.145 | 4.163 | 15.730 | 1.004x | 3.795x | | 23 | `(384,1,1)` | 4.560 | 4.689 | 15.888 | 1.028x | 3.484x | | 24 | `(192,2,1)` | 4.317 | 4.329 | 15.731 | 1.003x | 3.644x | | 25 | `(384,1,1)` | 4.836 | 4.867 | 15.777 | 1.006x | 3.262x | | 26 | `(128,2,1)` | 4.314 | 4.467 | 15.798 | 1.035x | 3.662x | | 27 | `(384,1,1)` | 5.052 | 5.121 | 15.899 | 1.014x | 3.147x | | 28 | `(128,2,1)` | 4.683 | 4.682 | 15.959 | 1.000x | 3.408x | | 29 | `(384,1,1)` | 5.318 | 5.442 | 16.048 | 1.023x | 3.018x | | 30 | `(128,2,1)` | 4.761 | 4.860 | 15.927 | 1.021x | 3.345x | | 31 | `(384,1,1)` | 5.531 | 5.559 | 16.046 | 1.005x | 2.901x | | 32 | `(192,2,1)` | 5.044 | 5.044 | 15.955 | 1.000x | 3.163x | ### GLM5.2 `(K=6144, N=256)` | M | Config | ll_fp32w us | dispatch us | cuBLAS us | Speedup vs dispatch | Speedup vs cuBLAS | |---:|---|---:|---:|---:|---:|---:| | 1 | `(384,1,1)` | 1.899 | 2.078 | 6.763 | 1.094x | 3.561x | | 2 | `(384,1,1)` | 2.102 | 2.300 | 21.317 | 1.094x | 10.141x | | 3 | `(384,1,1)` | 2.302 | 2.607 | 21.377 | 1.132x | 9.286x | | 4 | `(128,1,1)` | 2.615 | 2.938 | 21.326 | 1.124x | 8.155x | | 5 | `(128,1,2)` | 2.790 | 2.877 | 21.633 | 1.031x | 7.754x | | 6 | `(128,1,1)` | 2.964 | 3.103 | 15.648 | 1.047x | 5.279x | | 7 | `(128,1,2)` | 3.100 | 3.248 | 15.757 | 1.048x | 5.083x | | 8 | `(128,2,2)` | 3.197 | 3.476 | 20.018 | 1.087x | 6.261x | | 9 | `(128,1,2)` | 3.458 | 3.702 | 20.244 | 1.071x | 5.854x | | 10 | `(128,2,2)` | 3.359 | 3.716 | 20.326 | 1.106x | 6.051x | | 11 | `(128,1,1)` | 3.850 | 3.880 | 20.396 | 1.008x | 5.298x | | 12 | `(128,2,2)` | 3.758 | 4.033 | 20.414 | 1.073x | 5.432x | | 13 | `(256,1,2)` | 4.203 | 4.190 | 20.516 | 0.997x | 4.881x | | 14 | `(128,2,2)` | 3.857 | 4.433 | 20.524 | 1.149x | 5.321x | | 15 | `(256,1,2)` | 4.556 | 4.526 | 20.718 | 0.993x | 4.547x | | 16 | `(128,2,2)` | 4.138 | 4.434 | 20.798 | 1.072x | 5.026x | | 17 | `(256,1,2)` | 4.844 | 5.002 | 17.310 | 1.033x | 3.573x | | 18 | `(128,2,2)` | 4.433 | 4.751 | 17.400 | 1.072x | 3.925x | | 19 | `(256,1,2)` | 5.173 | 5.135 | 17.421 | 0.993x | 3.368x | | 20 | `(128,2,2)` | 4.653 | 4.848 | 17.405 | 1.042x | 3.741x | | 21 | `(256,1,2)` | 5.427 | 5.561 | 17.370 | 1.025x | 3.201x | | 22 | `(128,2,2)` | 4.909 | 5.031 | 17.493 | 1.025x | 3.563x | | 23 | `(256,1,2)` | 5.793 | 5.808 | 17.641 | 1.003x | 3.045x | | 24 | `(128,2,2)` | 5.360 | 5.252 | 17.626 | 0.980x | 3.288x | | 25 | `(256,1,2)` | 6.141 | 6.199 | 17.617 | 1.009x | 2.869x | | 26 | `(128,2,2)` | 5.631 | 5.542 | 17.686 | 0.984x | 3.141x | | 27 | `(256,1,2)` | 6.458 | 6.563 | 17.537 | 1.016x | 2.716x | | 28 | `(128,2,2)` | 5.850 | 5.914 | 17.588 | 1.011x | 3.006x | | 29 | `(256,1,2)` | 6.763 | 6.891 | 17.694 | 1.019x | 2.616x | | 30 | `(128,2,2)` | 6.139 | 6.223 | 17.623 | 1.014x | 2.871x | | 31 | `(256,1,2)` | 7.099 | 7.162 | 17.849 | 1.009x | 2.514x | | 32 | `(128,2,2)` | 6.399 | 6.431 | 17.834 | 1.005x | 2.787x | ## E2E serving results (MiniMax-M3) | Batch | main run | pr run | main median ITL | pr median ITL | median ITL diff | main tok/s | pr tok/s | tok/s diff | |---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1 | 2 | 3 | 5.73 | 5.73 | +0.0% | 173.79 | 173.79 | +0.0% | | 2 | 3 | 3 | 6.44 | 6.44 | +0.0% | 308.71 | 308.72 | +0.0% | | 3 | 3 | 2 | 7.60 | 7.64 | +0.5% | 392.90 | 391.41 | -0.4% | | 4 | 3 | 3 | 7.55 | 7.53 | -0.3% | 527.77 | 529.02 | +0.2% | | 5 | 2 | 3 | 8.62 | 8.53 | -1.0% | 574.68 | 579.33 | +0.8% | | 6 | 3 | 2 | 8.97 | 8.92 | -0.6% | 662.75 | 668.50 | +0.9% | | 7 | 2 | 2 | 9.29 | 9.31 | +0.2% | 745.72 | 747.07 | +0.2% | | 8 | 2 | 2 | 9.26 | 9.23 | -0.3% | 857.08 | 859.11 | +0.2% | | 9 | 2 | 3 | 10.18 | 10.19 | +0.1% | 878.64 | 873.63 | -0.6% | | 10 | 3 | 3 | 10.42 | 10.40 | -0.2% | 955.08 | 954.98 | -0.0% | | 11 | 3 | 2 | 10.72 | 10.72 | +0.0% | 1019.76 | 1021.32 | +0.2% | | 12 | 3 | 2 | 11.07 | 10.99 | -0.7% | 1077.26 | 1081.18 | +0.4% | | 13 | 3 | 3 | 11.25 | 11.32 | +0.6% | 1147.66 | 1138.11 | -0.8% | | 14 | 2 | 3 | 11.44 | 11.42 | -0.2% | 1209.73 | 1206.35 | -0.3% | | 15 | 2 | 2 | 11.79 | 11.74 | -0.4% | 1260.54 | 1264.29 | +0.3% | | 16 | 2 | 3 | 11.70 | 11.66 | -0.3% | 1348.45 | 1356.88 | +0.6% |", "url": "https://github.com/vllm-project/vllm/pull/44343", "createdAt": "2026-06-02T16:54:43Z", "updatedAt": "2026-08-13T16:00:58Z", "timestamp": "2026-08-13T16:00:58Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ci/build" ], "author": "LopezCastroRoberto", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44385", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for chunk_local_cumsum kernels", "text": "## Purpose Adds unit tests for the `chunk_local_cumsum` GDN/FLA kernels, which currently have none. Tests call the public entry point, so both the scalar (3D) and vector (4D) paths are covered, along with varlen (`cu_seqlens` plus caller-supplied chunk indices, as GDN/KDA pass them), partial tail chunks, reverse mode, and the bf16 downcasting store. ## Test Plan `pytest tests/kernels/test_chunk_local_cumsum.py` ## Test Result 88 passed on Arc Pro B70; same cases passed on CUDA.", "url": "https://github.com/vllm-project/vllm/pull/44385", "createdAt": "2026-06-03T06:30:39Z", "updatedAt": "2026-08-13T08:54:27Z", "timestamp": "2026-08-13T08:54:27Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44488", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix sparse BlockStored event token/hash mapping", "text": "## Purpose Fixes #44451. `BlockStored` events can become ambiguous when a KV cache group skips logical blocks, such as Mamba groups with `--mamba-cache-mode align`. Before this change, `block_hashes` and `extra_keys` only included emitted non-null blocks, while `token_ids` still covered the full logical token range. This could produce sparse events like one block hash with multiple block-sized token chunks, without enough metadata for external KV event consumers to determine which token chunk belongs to the emitted hash. This PR adds optional `BlockStored.block_offsets`. When present, `block_offsets[i]` identifies the block-sized chunk in `token_ids` corresponding to `block_hashes[i]`. Dense events keep `block_offsets=None`. ## Test Plan Unit tests: ```bash python -m pytest \\ tests/v1/core/test_prefix_caching.py::test_block_stored_event_offsets_for_null_blocks \\ tests/v1/core/test_prefix_caching.py::test_block_stored_event_offsets_for_masked_blocks \\ -q ``` End-to-end validation used a Qwen3.5 Mamba align server with KV events enabled: ```bash vllm serve /mnt/models/Qwen/Qwen3.5-4B \\ --trust-remote-code \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 1 \\ --gpu-memory-utilization 0.90 \\ --enforce-eager \\ --skip-mm-profiling \\ --limit-mm-per-prompt '{\"image\": 0, \"video\": 0}' \\ --kv-events-config '{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:8100\"}' ``` The e2e verifier subscribes to the ZMQ KV event stream, sends OpenAI-compatible completion requests, and checks that every sparse BlockStored event has reconstructable hash-to-token mapping: ```bash import threading import time from collections import Counter from typing import Any import msgspec import requests import zmq from msgspec.msgpack import Decoder class EventBatch(msgspec.Struct, array_like=True, omit_defaults=True, gc=False): ts: float events: list[Any] data_parallel_rank: int | None = None class KVCacheEvent( msgspec.Struct, array_like=True, omit_defaults=True, gc=False, tag=True ): pass class BlockStored(KVCacheEvent): block_hashes: list[Any] parent_block_hash: Any | None token_ids: list[int] block_size: int lora_id: int | None medium: str | None lora_name: str | None extra_keys: list[tuple[Any, ...] | None] | None = None group_idx: int | None = None kv_cache_spec_kind: str | None = None kv_cache_spec_sliding_window: int | None = None block_offsets: list[int] | None = None class BlockRemoved(KVCacheEvent): block_hashes: list[Any] medium: str | None group_idx: int | None = None class AllBlocksCleared(KVCacheEvent): pass class KVEventBatch(EventBatch): events: list[BlockStored | BlockRemoved | AllBlocksCleared] stats = Counter() bad_events = [] sparse_events = [] def listen(stop_event): decoder = Decoder(type=KVEventBatch) ctx = zmq.Context() sub = ctx.socket(zmq.SUB) sub.connect(\"tcp://:8100\") sub.setsockopt(zmq.SUBSCRIBE, b\"\") poller = zmq.Poller() poller.register(sub, zmq.POLLIN) while not stop_event.is_set(): if not poller.poll(500): continue frames = sub.recv_multipart() stats[\"raw_zmq\"] += 1 try: _, _, payload = frames batch = decoder.decode(payload) except Exception: stats[\"decode_error\"] += 1 continue for event in batch.events: if not isinstance(event, BlockStored): continue stats[\"block_stored\"] += 1 token_blocks = len(event.token_ids) // event.block_size hash_blocks = len(event.block_hashes) if event.kv_cache_spec_kind: stats[f\"kind:{event.kv_cache_spec_kind}\"] += 1 if token_blocks > hash_blocks: stats[\"sparse\"] += 1 record = { \"group_idx\": event.group_idx, \"kind\": event.kv_cache_spec_kind, \"block_size\": event.block_size, \"token_blocks\": token_blocks, \"hash_blocks\": hash_blocks, \"len_token_ids\": len(event.token_ids), \"block_offsets\": event.block_offsets, } sparse_events.append(record) ok = ( event.block_offsets is not None and len(event.block_offsets) == hash_blocks and all(0 <= off < token_blocks for off in event.block_offsets) ) if not ok: bad_events.append(record) def send_requests(): base_url = \"http://:8000\" model = \"/mnt/models/Qwen/Qwen3.5-4B\" url = f\"{base_url}/v1/completions\" lengths = [500, 527, 529, 800, 1055, 1057, 1584, 2112, 2500, 3168] salts = [\"same-salt\", \"tenant-A\", \"tenant-B\"] for salt in salts: for n in lengths: prompt = f\"{salt}\\n\" + \" \".join(f\"tok{i}\" for i in range(n)) r = requests.post( url, json={ \"model\": model, \"prompt\": prompt, \"temperature\": 0, \"max_tokens\": 16, }, timeout=120, ) r.raise_for_status() def main(): stop_event = threading.Event() t = threading.Thread(target=listen, args=(stop_event,), daemon=True) t.start() time.sleep(1) send_requests() time.sleep(3) stop_event.set() t.join(timeout=2) print(\"Summary:\", dict(stats)) print(\"Sparse events:\") for event in sparse_events: print(event) if bad_events: print(\"BAD sparse events:\") for event in bad_events: print(event) raise SystemExit(1) if not sparse_events: print(\"No sparse events observed.\") raise SystemExit(2) print(\"PASS: sparse BlockStored events are reconstructable.\") if __name__ == \"__main__\": main() ``` ## Test Result ``` Unit tests pass. ``` Before the fix, reverting the code reproduces ambiguous sparse Mamba events: BAD sparse events: ``` {'group_idx': 0, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': None} {'group_idx': 1, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': None} {'group_idx': 2, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 2, 'hash_blocks': 1, 'len_token_ids': 1056, 'block_offsets': None} ``` After the fix, sparse Mamba events include offsets and pass validation: ``` {'group_idx': 0, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} {'group_idx': 1, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} {'group_idx': 2, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} PASS: sparse BlockStored events are reconstructable. ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/44488", "createdAt": "2026-06-04T04:28:25Z", "updatedAt": "2026-08-13T17:40:02Z", "timestamp": "2026-08-13T17:40:02Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "v1" ], "author": "Li-brua", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] AITER FP8 ASM prefill backend", "text": " ## Purpose Add AITER persistent ASM kernels as a MLA prefill backend. Only for FP8 on gfx950, where it is now the default. #42509 added the (causal) AITER ASM prefill kernel as part of the AITER MLA backend, but it only works for _non-chunked prefills_. For chunked batches _we still currently fall back to FA_. This PR integrates the non-causal ASM prefill kernel so it can be used together with chunked prefill. Dependent on: - [[Bugfix][MLA] Correct final_lse in PS MLA prefill kernel for chunked prefill](https://github.com/ROCm/aiter/pull/3606) (aiter#3606). That PR adds `max_kvlen` to the formal parameters of `get_ps_metadata_info_v1` so we can gate on that. **Hence safe to merge this PR without the AITER PR being merged and released and baked into vllm docker.** ### (TLDR) Perf improvement - 1k/1k ISL/OSL: basically unchanged perf (tested at 32-256 concurrency) - 8k/1k ISL/OSL: **7-9% lower TTFT**, **2-6% higher QPS** (tested at 32-128 concurrency) _Note: I did not yet delete the PS ASM code path that currently sits in `vllm/v1/attention/backends/mla/rocm_aiter_mla.py` which is activated only for pure causal prefills. We need to keep this as we otherwise would have a performance regression until the gate on aiter#3606 goes through. Also makes it fully backwards compatible, even though it might be architecturally advisable to delete the PS ASM code path from rocm_aiter_mla.py once the AITER_ASM backend is fully available._ ## Test Plan ### 1. Patch nightly image with this branch, and re-build aiter from aiter#3606
```Dockerfile # docker/Dockerfile.rocm_aiter_asm_src # Build: # docker build --no-cache -f docker/Dockerfile.rocm_aiter_asm_src \\ # --build-arg AITER_REPO=https://github.com/simondanielsson/aiter.git \\ # --build-arg AITER_BRANCH=fix/mla-prefill-ps-final-lse \\ # -t vllm-rocm-aiter-asm-prefill:src . # # Override the base if you want a pinned nightly: # --build-arg BASE_IMAGE=rocm/vllm-dev:nightly_main_20260604 # Override the PR if you rebased / opened a new one: # --build-arg PR_NUMBER=44544 ARG BASE_IMAGE=vllm/vllm-openai-rocm:nightly-a16dbd5b8572d4128be9f10b9dcff4999b594b25 ### ### Stage 1: build AITER from source against THIS image's torch. ### FROM ${BASE_IMAGE} AS build_aiter ARG AITER_REPO=https://github.com/simondanielsson/aiter.git ARG AITER_BRANCH=fix/mla-prefill-ps-final-lse # Restrict to the arch you actually run on; add gfx942 back if you need MI300. ARG AITER_ROCM_ARCH=gfx950 RUN set -eux; \\ apt-get update && apt-get install -y --no-install-recommends git curl; \\ rm -rf /var/lib/apt/lists/* # Clone with submodules (CK etc.) exactly like Dockerfile.rocm_base. RUN set -eux; \\ git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO} /aiter; \\ cd /aiter && git submodule update --init --recursive; \\ git rev-parse HEAD > /aiter_sha.txt # Build deps. We deliberately do NOT touch torch here: the nightly base already # ships the torch we must ABI-match against, so requirements.txt must not pull a # different torch. If it tries to, pin/strip it rather than letting it upgrade. RUN set -eux; \\ cd /aiter; \\ pip install pyyaml; \\ pip install -r requirements.txt # AOT-compile kernels into the wheel (PREBUILD_KERNELS=1) for gfx950, matching # the stock build. This is the path the runtime actually exercises; a JIT build # (PREBUILD_KERNELS unset) is a different code path and not what ships. RUN set -eux; \\ cd /aiter; \\ PREBUILD_KERNELS=1 GPU_ARCHS=${AITER_ROCM_ARCH} \\ python3 setup.py bdist_wheel --dist-dir=/aiter/dist; \\ ls -l /aiter/dist/*.whl ### ### Stage 2: patched nightly vLLM + the freshly built aiter wheel. ### FROM ${BASE_IMAGE} AS final ARG PR_NUMBER=44544 ARG PATCH_URL=https://github.com/vllm-project/vllm/pull/${PR_NUMBER}.diff # Apply the vLLM PR diff onto the installed vllm package in site-packages. RUN set -eux; \\ apt-get update && apt-get install -y --no-install-recommends curl patch; \\ rm -rf /var/lib/apt/lists/*; \\ VLLM_DIR=\"$(python3 -c 'import vllm, os; print(os.path.dirname(vllm.__file__))')\"; \\ REPO_ROOT=\"$(dirname \"${VLLM_DIR}\")\"; \\ echo \"Applying ${PATCH_URL} into ${REPO_ROOT}\"; \\ curl -fsSL \"${PATCH_URL}\" -o /tmp/pr.patch; \\ cd \"${REPO_ROOT}\"; \\ git apply -p1 --include=\"vllm/**\" /tmp/pr.patch; \\ rm /tmp/pr.patch; \\ find \"${VLLM_DIR}\" -name '__pycache__' -type d -exec rm -rf {} + # Install the aiter we just built. --no-deps so pip cannot drag in a different # torch; --force-reinstall so it replaces whatever aiter the nightly shipped. RUN --mount=type=bind,from=build_aiter,src=/aiter/dist,target=/aiter_dist \\ set -eux; \\ pip install --no-deps --force-reinstall /aiter_dist/*.whl # Record which aiter SHA we built so the running container is self-describing. COPY --from=build_aiter /aiter_sha.txt /aiter_sha.txt ``` Build: ``` docker build --build-arg AITER_REV=1781510439 --build-arg PR_REV=60e1866dbd1d6deb1dc9b7f0141590388a1b45f4 -f docker/Dockerfile.rocm_aiter_asm_src-cached -t vllm-rocm-aiter-asm-prefill:local . ```
### 2. `vllm bench serve` on DSv3, vs main Run model ```shell docker run \\ --rm \\ --name p-replay-dsv3 \\ --init --network host --ipc host --privileged \\ --cap-add SYS_PTRACE --security-opt seccomp=unconfined \\ --ulimit memlock=-1 --ulimit stack=67108864 \\ --shm-size 256G \\ --group-add video --group-add render \\ --device /dev/kfd --device /dev/dri --device /dev/infiniband \\ -v /sys:/sys \\ -v \"/mnt/nfs_share/huggingface/:/root/.cache/huggingface\" \\ -e HF_HOME=/root/.cache/huggingface \\ -e HF_TOKEN=$HF_TOKEN \\ -e HF_HUB_OFFLINE=1 \\ -e HF_HUB_ENABLE_HF_TRANSFER=0 \\ -e NCCL_MIN_NCHANNELS=112 \\ -e SAFETENSORS_FAST_GPU=1 \\ -e VLLM_ROCM_USE_AITER=1 \\ vllm-rocm-aiter-asm-prefill:local \\ deepseek-ai/DeepSeek-V3 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.8 \\ --tensor-parallel-size 8 \\ --max-model-len 16K \\ |& tee asm_aiter_backend.log ``` - For 1k/1k ISL/OSL, sweep concurrencies 32-256 - For 8k/1k, sweep 32-128 conc. Always warmup with 2x concurrency # prompts, and run for 10x. ```bash docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c 'vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280' ``` ### 3. Acc with DSv3 FP8: this branch vs main ```shell docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c \"pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k\" ``` ### 4. Profile sanity check We need to profile at high enough concurrency to see that the expected kernels are being called during mixed batches with chunked prefills, e.g. using the following commands. Specifically, we need to check 1. main falls back to FA for chunked prefill (even for the causal part!) 2. this branch invokes the ASM kernels for both causal and non-causal prefill during chunked prefill Run profile with 128 concurrent requests, warmup for 32 requests, and run 256 prompts. ### 5. Unit tests ``` python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_backends.py -k \"AITER_ASM and fp8-\" python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_prefill_selector.py -k \"aiter\" ``` ## Test Result ### 2. vllm bench serve #### 128 conc, 8k/1k ISL/OSL Showing the full bench output for this workload. The remaining workload results I just show in a single table. **This branch:** ``` ============ Serving Benchmark Result ============ Successful requests: 1280 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 815.86 Total input tokens: 10484480 Total generated tokens: 1310720 Request throughput (req/s): 1.57 Output token throughput (tok/s): 1606.55 Peak output token throughput (tok/s): 3968.00 Peak concurrent requests: 133.00 Total token throughput (tok/s): 14457.39 ---------------Time to First Token---------------- Mean TTFT (ms): 4032.52 Median TTFT (ms): 1423.93 P99 TTFT (ms): 46993.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.66 Median TPOT (ms): 76.42 P99 TPOT (ms): 82.60 ---------------Inter-token Latency---------------- Mean ITL (ms): 75.66 Median ITL (ms): 37.64 P99 ITL (ms): 406.58 ================================================== ``` **Nightly (vllm/vllm-openai-rocm:nightly-43914dd743ab0500abcd69fe072e02465c944dcf)** ``` ============ Serving Benchmark Result ============ Successful requests: 1280 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 863.50 Total input tokens: 10484480 Total generated tokens: 1310720 Request throughput (req/s): 1.48 Output token throughput (tok/s): 1517.91 Peak output token throughput (tok/s): 3968.00 Peak concurrent requests: 133.00 Total token throughput (tok/s): 13659.75 ---------------Time to First Token---------------- Mean TTFT (ms): 4394.12 Median TTFT (ms): 1569.67 P99 TTFT (ms): 50166.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 79.97 Median TPOT (ms): 81.64 P99 TPOT (ms): 85.45 ---------------Inter-token Latency---------------- Mean ITL (ms): 79.97 Median ITL (ms): 37.22 P99 ITL (ms): 440.01 ================================================== ``` #### Full bench serve results #### 1k/1k Mostly unchanged (within noise factor), because decode heavy workload. | Branch | Concurrency | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | TTFT change (%) | |---|---|---|---|---|---|---| | Main | 32 | 643.5 | 19.55 | 1.51 | | | | This branch | 32 | 656.5 | 19.48 | 1.52 | +0.5% | +2% | | Main | 64 | 837.5 | 26.43 | 2.27 | | | | This branch | 64 | 839.6 | 25.7 | 2.35 | +3.5% | +0.2% | | Main | 128 | 996.54 | 36.86 | 3.29 | | | | This branch | 128 | 1003.5 | 36.5 | 3.27 | -0.6% | +0.7% | | Main | 256| 947.5 | 47.54 | 5.1 | | | | This branch | 256 | 979.5 | 48.5 | 5.00 | -1.9% | +3% | #### 8k/1k 7-9% improved TTFT, 2-6% improved QPS. | Branch | Concurrency | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | TTFT change (%) | |---|---|---|---|---|---|---| | Main | 32 | 1936.2 | 29.9 | 0.96 | | | | This branch | 32 | 1787.95 | 29.1 | 0.99 | +2.8% | -7.6% | | Main | 64 | 1896.8 | 47.5 | 1.24 ||| | This branch | 64 | 1763.7 | 45.2 | 1.30 | +4.8% | -7% | | Main | 128 | 1569.67 | 81.63 | 1.48 | || | This branch | 128 | 1423.93 | 76.4 | 1.57 | +6% | -9.3% | ### 3. Accuracy #### This branch, with AITER changes ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9431|± |0.0064| | | |strict-match | 5|exact_match|↑ |0.9424|± |0.0064| ``` #### vllm/vllm-openai-rocm:nightly-43914dd743ab0500abcd69fe072e02465c944dcf ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ``` ### 4. Profile sanity check ### On `vllm/vllm-openai-rocm:nightly-3f0a91bb96f8d72e0498b95c166e817deae14d62` **First prefill batch**: causal ASM kernel already gets called (`aiter::mla_pfl_qh192_vh128_m32x8_n128x1_causal1`, in purple) \"image\" **Mixed batched but with pure (new) prefills:** ASM kernel also already called \"image\" **Mixed batch with cached prefill (extends)**: FA fallback (this is the `void ck_tile::kentry<256, 1, ck_tile::FmhaFwdKernel` etc), plus lots of small eager quant ops right before it. **This is what we want to get rid of with this PR.** \"image\" ### This branch **First pure prefill batch:** (causal) ASM kernel called (in purple) \"image\" **Mixed batched but with pure (new) prefills:** (causal) ASM called as expected \"image\" **Mixed batch with cached prefill (extends)**: (**NEW**) Two prefill ASM kernels & 1 decode ASM called \"image\" The first one is the causal (see the trailing `causal1`), and the second one is the non-causal _extend_ kernel: \"image\" **Worth noting**: The get_ps_metadata function is blocking (it's doing 5 sync memcpys unfortunately) so host side looks like this. Doesn't seem like a big issue though since we are only doing this once per forward. \"image\" ### 5. Unit tests Skips coming from non-unit scale tests (e.g. `test_backend_correctness[MLAPrefillBackendEnum.AITER_ASM-2.0-3.0-fp8-16-deepseek-ai/DeepSeek-R1-spec_decode_medium`) because AITER ASM only supports unit scales. ``` $ python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_backends.py -k \"AITER_ASM and fp8-\" ... -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =================================================== 48 passed, 48 skipped, 1344 deselected, 23 warnings in 154.41s (0:02:34) ==================================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ``` $ python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_prefill_selector.py -k \"aiter\" ============================================================================== test session starts ============================================================================== platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /app/repos/vllm configfile: pyproject.toml plugins: asyncio-1.4.0, anyio-4.13.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 19 items / 13 deselected / 6 selected tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability0-fp8-True-True-None] PASSED [ 16%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability1-auto-True-False-fp8] PASSED [ 33%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability2-fp8-True-False-compute capability] PASSED [ 50%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability3-fp8-False-False-R1] PASSED [ 66%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmSelectorPriority::test_aiter_asm_wins_on_gfx950_fp8 PASSED [ 83%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmSelectorPriority::test_falls_through_to_flash_attn_when_not_fp8 PASSED [100%] ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/44544", "createdAt": "2026-06-04T14:51:11Z", "updatedAt": "2026-08-13T08:59:42Z", "timestamp": "2026-08-13T08:59:42Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "v1" ], "author": "simondanielsson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU][Zen] Route Int8 MoE inference through zentorch on AMD", "text": "## Purpose Routes CPU MoE on AMD Zen through a zentorch-backed Int8 W8A8 fused-MoE backend: the per-expert int8 GEMMs are dispatched through torch.ops.zentorch.*, mirroring the existing zentorch W8A8 dense-linear integration. On non-Zen CPUs, or when zentorch is not installed, the int8 MoE oracle falls through to the existing backends, so behaviour elsewhere is unchanged. - CPUInt8Experts: per-expert int8 experts kernel dispatching through torch.ops.zentorch (zentorch_dynamic_qlinear / zentorch_fused_moe). - Int8 MoE oracle: add the CPU_ZEN backend, gated on is_zen_cpu(); - LLM-Compressor per-expert W8A8 checkpoint loading (GPT-OSS), including the SwiGLU-OAI interleaved weight layout for the fused W8A8 path. - Centralize the topk_softmax / sigmoid CPU fallback in _custom_ops. ## Test Plan lm_eval --model vllm --model_args pretrained=amd/gpt-oss-20b-BF16-w8a8-llmcompressor-v0.10.0.2,dtype=bfloat16,max_model_len=4096 --tasks gsm8k --batch_size auto --trust_remote_code --num_fewshot 5 --apply_chat_template --log_samples --gen_kwargs \"max_gen_toks=2048\" ## Test Result Verified GPT-OSS / Qwen3 MoE W8A8 inference on AMD Zen CPU. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/44834", "createdAt": "2026-06-08T05:51:47Z", "updatedAt": "2026-08-13T08:26:06Z", "timestamp": "2026-08-13T08:26:06Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "rocm", "ci/build", "v1", "cpu", "gpt-oss", "quantization", "mistral" ], "author": "ganeshr10", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44927", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(structured_output): pass new_token_ids to should_advance() to fix MTP spec-decode off-by-one", "text": "Fixes #34650. ## Problem When Multi-Token Prediction (MTP) speculative decoding accepts a draft token, `_update_after_schedule` pre-increments `num_computed_tokens` by `(1 main + N spec)` tokens **before** model execution. After `update_from_output` appends the accepted tokens, the index-based delta in `should_advance()` starts one position past the main token. This means a `` token emitted as the main token is silently missed: `reasoning_ended` is never set to `True`, and grammar constraints (JSON mode, structured output) are **permanently disabled** for the rest of the request. Users running Qwen3 models with MTP enabled see unstructured output with no error or warning. ## Root Cause `should_advance()` infers which new tokens to scan by computing a delta from `num_computed_tokens`: ```python delta_from = request.num_computed_tokens - request.num_output_placeholders start = delta_from if delta_from >= 0 else max(len(all_token_ids) + delta_from, 0) ``` After an MTP step, `num_computed_tokens` has already been pre-incremented to include the spec tokens, so `start` lands past the `` token — it is never examined by `is_reasoning_end_streaming`. ## Fix Add an optional `new_token_ids: list[int] | None = None` parameter to `should_advance()`. The one call site inside `update_from_output()` already holds the accepted token list, so passing it directly bypasses the broken index arithmetic: ```python # scheduler.py — caller passes accepted tokens directly if new_token_ids and self.structured_output_manager.should_advance( request, new_token_ids ): # structured_output/__init__.py — use provided delta when available if new_token_ids is not None: delta: Iterable[int] = new_token_ids else: # existing path for non-MTP callers: async-scheduling-aware delta delta_from = request.num_computed_tokens - request.num_output_placeholders ... delta = itertools.islice(all_token_ids, start, None) if reasoner.is_reasoning_end_streaming(all_token_ids, delta): ``` All other call sites (`update_draft_token_ids`, `update_draft_token_ids_in_output`) omit the new parameter and continue using the existing async-scheduling-aware delta, which is correct for those contexts. ## Tests `tests/v1/structured_output/test_reasoning_structured_output.py` — added `test_mtp_reasoning_grammar_not_disabled`: - Simulates an MTP step by pre-incrementing `num_computed_tokens` as `_update_after_schedule` would - Calls `update_from_output` with a `` token appended - Asserts `reasoning_ended` is `True` (was `False` before fix) - Asserts `should_advance()` returns `True`, confirming grammar is still enabled", "url": "https://github.com/vllm-project/vllm/pull/44927", "createdAt": "2026-06-08T20:48:57Z", "updatedAt": "2026-08-13T09:20:20Z", "timestamp": "2026-08-13T09:20:20Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "structured-output", "needs-rebase", "v1" ], "author": "nac7", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44969", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Gating more ROCm tests", "text": "Add 22 AMD mirrors for existing CI test groups as the Stage D gating expansion. ### `mi250_1` - Cudagraph - Docker Build Metadata - Kernels Mamba Test - Kernels Helion Test - Language Models Test (PPL) - Language Models Test (MTEB) ### `mi250_2` - Plugin Tests (2 GPUs) ### `mi300_1` - Model Runner V2 Spec Decode - PyTorch Compilation Unit Tests - Speculators Correctness - Language Models Test (Extended Generation) - Entrypoints Integration (Responses API) - Entrypoints Integration (Multimodal) - Async Engine, Inputs, Utils, Worker ### `mi300_2` - Model Runner V2 Distributed (2 GPUs) - Distributed Model Tests (2 GPUs) ### `mi300_4` - V1 e2e (4xH100-4xMI300) - LoRA TP (Distributed) ### `mi355_1` - Entrypoints Unit Tests - Multi-Modal Processor - Multi-Modal Models (Extended Generation 2) - Multi-Modal Models (Extended Generation 3)", "url": "https://github.com/vllm-project/vllm/pull/44969", "createdAt": "2026-06-09T06:56:07Z", "updatedAt": "2026-08-13T17:38:30Z", "timestamp": "2026-08-13T17:38:30Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "rocm", "ready", "needs-rebase", "ci/build", "v1", "nvidia" ], "author": "AndreasKaratzas", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:44987", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] eplb", "text": "Enable XPU eplb and add TorchDistXCCLStagedEplbCommunicator for this. Torch_xccl and torch_gloo can both be used on XPU. ``` model=Qwen/Qwen3-30B-A3B VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model $model --enforce-eager --port 30088 --trust-remote-code --gpu-memory-util=0.9 --no-enable-prefix-caching --max-num-batched-tokens=8192 --no-enable-log-requests --max-model-len=8192 --block-size 64 -tp=4 \\ --enable_expert_parallel --enable_eplb --eplb-config '{\"num_redundant_experts\":32, \"step_interval\":200, \"log_balancedness\":true, \"communicator\":\"torch_xccl\", \"use_async\":false}' lm_eval --model local-completions --tasks gsm8k --model_args \"model=$model,base_url=http://localhost:30088/v1/completions,max_gen_toks=1024,num_concurrent=64,timeout=9999999\" --limit 100 ``` log: \"image\" lm_eval limits 100 results: \"image\"", "url": "https://github.com/vllm-project/vllm/pull/44987", "createdAt": "2026-06-09T09:03:45Z", "updatedAt": "2026-08-13T02:44:02Z", "timestamp": "2026-08-13T02:44:02Z", "metrics": { "reactions": 4, "comments": 5 }, "labels": [ "intel-gpu" ], "author": "mayuyuace", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45111", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Re-enable cross-layer KV cache layout for MLA via stride-aware kernels", "text": "## Purpose #37090 disabled the cross-layer (block-major) KV cache layout for all MLA backends after #37032 (GLM-4.7-Flash garbage output with KV offloading), attributing the bug to \"MLA kernels requiring contiguous per-layer KV cache views\". The actual cause is narrower: a few kernels computed page addresses from `block_size * entry_size` instead of reading the cache tensor's block-dim stride. The MLA write path (`concat_and_cache_mla`), the prefill gather kernels (`cp_gather_cache`, `gather_and_maybe_dequant_cache`), DeepGEMM `fp8_paged_mqa_logits`, and several decode kernels are already stride-aware. This PR fixes the three kernels that genuinely assumed packed pages and re-enables the cross-layer layout per backend (opt-in), keeping the safe identity default on `MLACommonBackend` for backends not yet verified (ROCm AITER, tokenspeed, XPU). ### Kernel fixes - `vllm/v1/attention/ops/triton_decode_attention.py`: both stage-1 decode kernels addressed the cache as `(page_number * PAGE_SIZE + offset) * stride(-3)`, baking in `stride(block) == PAGE_SIZE * stride(token)`. They now take the page-dim stride separately. This is the kernel behind the original #37032 report (A100 falls back to TRITON_MLA). - `csrc/libtorch_stable/attention/mla/sm100_cutlass_mla_kernel.cu`: `stride_C` hardcoded `page_size * (D_latent + D_rope)`; now built from `kv_c_and_k_pe_cache.stride(0)/stride(1)` (identical values for contiguous caches). - `csrc/libtorch_stable/cache_kernels.cu` `indexer_k_quant_and_cache`: block base was `block_idx * cache_block_size * kv_cache.size(2)`; now `block_idx * kv_cache.stride(0)`. Writing through a strided view previously corrupted the target layer and bled into neighbouring layers' segments. This also unblocks the DeepSeek V3.2/V4 indexer KV cache group under packed/cross-layer layouts. ### Per-backend opt-in `get_kv_cache_stride_order(include_num_layers_dimension=True)` returns `(1, 0, 2, 3)` on backends whose decode kernels verifiably honor the cache's block-dim stride: TritonMLA and CutlassMLA (fixed above), FlashAttnMLA (FA3 reads `k_batch_stride = kcache.stride(0)`), FlashMLA (dense decode reads `kcache.stride(0)`), and FlashInferMLA (verified bit-exact). `MLACommonBackend` keeps the identity permutation as the safe default, so unverified backends remain opted out and can opt in individually once verified. ### Tests `tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py` (new) runs each kernel on a contiguous cache vs a per-layer view carved from a cross-layer buffer (inflated `stride(0)`, non-zero storage offset, neighbour layers filled with garbage) and asserts bit-exact equality: `concat_and_cache_mla` write (incl. zero bleed), FlashMLA dense decode (Hopper-gated), FlashMLA dense fp8 decode (Hopper-gated), FA3 decode (Hopper-gated), FlashInfer MLA dense decode (bf16 + fp8), FlashMLA fp8 sparse decode, and `indexer_k_quant_and_cache` (incl. zero bleed). Similar strided-view tests are added for the triton decode kernels (all three address paths: MLA grouped, GQA grouped, MHA normal) and CUTLASS sm100 MLA decode. The MLA stride-order unit tests are updated for the opt-in design. ## Why this is not duplicating an existing PR Searched open PRs for cross-layer/stride-order work: #44577 packs DSv4 KV caches into contiguous per-block allocations but only touches allocation/connector/runner plumbing, no kernels — this PR is complementary (the kernel stride fixes here are what make such packed per-block layouts safe for MLA decode/write kernels). #41093 adds cross-layer support on the Mooncake connector side only. The KV-layout refactor series (#44458 draft, #44455, #42374) standardizes layout plumbing and overlaps some files but does not address the packed-page stride bugs or the MLA cross-layer opt-ins. #34742 is the stride-order default refactor referenced in #37090 review and is orthogonal. ## Test commands and results ``` pytest tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py -v # GB200 (sm100), extensions built from source: 5 passed, 3 skipped (Hopper-gated: FA3, FlashMLA dense bf16/fp8) pytest tests/kernels/attention/test_triton_decode_attention.py -v # 118 passed (incl. new cross-layer strided-view tests; no regression in existing paged/fp8 paths) pytest tests/kernels/attention/test_cutlass_mla_decode.py::test_cutlass_mla_decode_cross_layer_view -v # 1 passed on GB200 (fails against the pre-fix kernel with max diff 0.7, confirming the bug) pytest tests/v1/kv_connector/unit/test_kv_cache_layout.py -v # 7 passed ``` The three Hopper-gated tests were verified to skip cleanly here; they exercise on sm90 CI. Before the fixes, the strided-view tests reproduce the #37032 failure mode: triton/CUTLASS decode read wrong blocks for block_id > 0, and the indexer write corrupts neighbouring layers. ## AI assistance disclosure This PR was developed with AI assistance (Claude Code). The root-cause analysis, kernel fixes, and tests were reviewed line-by-line and the test suite was run on GB200 hardware by the submitter; an independent automated review (Codex) of the final diff reported no findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/45111", "createdAt": "2026-06-10T06:54:30Z", "updatedAt": "2026-08-12T21:11:24Z", "timestamp": "2026-08-12T21:11:24Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "ready", "v1", "kv-connector", "nvidia" ], "author": "ivanium", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45423", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Correct prompt lengths for timed_traces benchmark", "text": "## Purpose #39795 introduced `timed_traces` support for `vllm bench serve`. The traces look as follows: ``` {\"timestamp\": 0, \"input_length\": 6758, \"output_length\": 500, \"hash_ids\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]} {\"timestamp\": 0, \"input_length\": 7322, \"output_length\": 490, \"hash_ids\": [0, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]} {\"timestamp\": 0, \"input_length\": 7236, \"output_length\": 794, \"hash_ids\": [0, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]} {\"timestamp\": 0, \"input_length\": 2290, \"output_length\": 316, \"hash_ids\": [0, 42, 43, 44, 45]} ``` Current implementation creates prompts with these input_lengths, and then runs: ``` prompt = tokenizer.decode(prompt_ids) [...] samples.append( SampleRequest( prompt=prompt, prompt_len=prompt_len, ``` vLLM engine on the server side runs: ``` prompt_idx = tokenizer.encode(prompt) ``` Unfortunately, tokenizer logic is not idempotent, so whereas the client generates requests of length 6758, 7322, 7236, 2290, etc., the server receives requests of different length 7253, 7844, 7679, 2445, etc. **Solution:** Avoid calling `.decode` and `.encode` The OpenAI API and vLLM engine support direct tokens id passing, see: `vllm/entrypoints/openai/completion/protocol.py` ``` class CompletionRequest(OpenAIBaseModel): # Ordered by official OpenAI API documentation # https://platform.openai.com/docs/api-reference/completions/create model: str | None = None prompt: ( list[Annotated[int, Field(ge=0)]] | list[list[Annotated[int, Field(ge=0)]]] | str | list[str] | None ) = None ``` This PR avoids the `.encode()` to directly pass token ids, maintaining correct prompt lengths on the server side. ## Test Plan Just run any of the traces **sequentially** (**--max-concurrency 1**) : ``` vllm bench serve \\ --model Qwen/Qwen3.5-0.8B \\ --dataset-name timed_trace \\ --num-prompts 50 \\ --dataset-path conversation_trace_optimal_store_and_prefix_match.jsonl \\ --base-url http://localhost:12305 \\ --ignore-eos \\ --self-timed \\ --timed-trace-label-timestamp timestamp \\ --timed-trace-chunk-hash-size 512 \\ --timed-trace-label-hash-ids hash_ids \\ --timed-trace-label-input-length input_length \\ --timed-trace-label-output-length output_length \\ --timed-trace-sec-multiplier 0.001 --max-concurrency 1 ``` + vLLM serve in debug mode: ``` vllm serve --port 12305 --enable-prefix-caching --model Qwen/Qwen3.5-0.8B ``` and observe `request.num_tokens` in `vllm/v1/core/sched/scheduler.py::Scheduler.schedule` ## Test Result Json file input_lengths: 6758, 7322, 7236, 2290, ... **Main:** For consecutive scheduling rounds, `request.num_tokens=7253, 7844, 7679, 2445,...` **This PR:** For consecutive scheduling rounds, `request.num_tokens=6758, 7322, 7236, 2290, ...` -- as expected @animeshtrivedi @tdoublep", "url": "https://github.com/vllm-project/vllm/pull/45423", "createdAt": "2026-06-12T15:44:03Z", "updatedAt": "2026-08-13T11:06:38Z", "timestamp": "2026-08-13T11:06:38Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "bug", "performance", "ready" ], "author": "s3woz", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45559", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Kernel] Extend skinny gemm N=5 to N=8 cases on GFX12 (RDNA4) using SWMMAC optimization", "text": "## Purpose This PR adds an optimized ROCm GFX12 (RDNA4) SWMMAC-based skinny GEMM path. The new path targets serving workloads with low max concurrency, matching the benchmarked decode GEMM range of N=5-8. The implementation uses sparse wave matrix multiply-accumulate instructions on GFX12 (RDNA4) to improve throughput for these skinny GEMM cases, while keeping the existing fallback path for unsupported cases. Skinny GEMMs can be important for ROCm/GFX12 (RDNA4) serving performance, which is often used in local or small-scale serving setups. In these deployments, the server is expected to handle single users, or a small number of users with a relatively lower request rate, so the active decode batch can remain small instead of being filled to a large batch size. This optimization follows the same general direction described in the Hugging Face technical blog, [*Creating custom kernels for the AMD MI300*](https://huggingface.co/blog/mi300kernels). The blog discusses custom kernels for AMD MI300X vLLM serving and describes a skinny GEMM optimization for inputs with 8 or fewer rows. The key idea is to map each logical skinny A row into two adjacent SWMMAC sparse rows. The first sparse row selects one set of 16-bit pairs from `A[n, :]`, and the second sparse row selects the complementary pairs from the same logical row. Both sparse rows multiply with the same normal `B[32 x 16]` fragment, producing two partial accumulator rows. The epilogue then adds the adjacent partial rows back together to recover the final logical output row. This interleaved sparse-row layout lets the kernel use the SWMMAC instruction shape more effectively for N=5-8 instead of leaving the skinny rows underfilled. A related optimization direction is also being explored in IREE: [*[AMDGPU][Codegen] Optimizing skinny GEMM workloads with sparse matrix instructions*](https://github.com/iree-org/iree/issues/22863). The IREE issue describes using sparse matrix instructions to optimize skinny GEMM workloads on AMD CDNA GPUs. Although the IREE discussion focuses on IREE codegen and CDNA-style sparse MFMA examples, the motivation is closely related: skinny GEMM workloads can waste matrix instruction capacity due to padding or under-filled dimensions, and sparse matrix instructions can recover more effective throughput. Note: this PR is not a direct port of either implementation. It adapts the same idea to the vLLM ROCm/GFX12 (RDNA4) path using SWMMAC with tuned optimization strategies for the measured small-N decode GEMM cases. **This PR:** * Adds a GFX12 (RDNA4) SWMMAC skinny GEMM kernel path * Targets decode GEMM shapes with N=5-8 * Preserves the existing fallback path for unsupported cases * Adds correctness coverage for the new SWMMAC path across: * N = 5, 6, 7, 8 * FP16 and BF16, matching the existing skinny GEMM dtype coverage * Multiple M/K sizes * Aligned (divisible by 16) and non-aligned M dimensions * Large-shape cases Note: N=4 was also evaluated, but it did not show a benefit in this implementation, so this PR keeps the optimized path scoped to N=5-8. ## Test Plan ### Unit tests ```bash pytest -v tests/kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_swmmac_gemm_kernel ``` ### GSM8K end-to-end accuracy The GSM8K run was performed with the vLLM server launched using --max-num-seqs 8, and lm_eval used num_concurrent=8. This keeps the active decode batch within the targeted skinny GEMM range and exercises the SWMMAC N=5-8 paths. **Model:** `Qwen/Qwen3-4B` Server command: ```bash vllm serve Qwen/Qwen3-4B \\ --host 0.0.0.0 \\ --port 8000 \\ --dtype bfloat16 \\ --max-num-seqs 8 \\ --gpu-memory-utilization 0.90 ``` Benchmark command: ```bash lm_eval \\ --model local-completions \\ --model_args \"model=${MODEL},base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=8,max_retries=10,max_gen_toks=1024,max_length=8192,timeout=60000\" \\ --batch_size auto \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path ./results_qwen3_4b_gsm8k \\ --log_samples ``` ### Performance Benchmark - **Hardware:** GPU: AMD Radeon AI PRO R9700, gfx1201, GFX12 (RDNA4), **x1** | CPU: AMD Ryzen 9 9950X 16-Core Processor, **x32** | RAM: 123Gi - **Model:** Qwen/Qwen3-4B - **Benchmark setup:** - **Concurrency:** 5, 6, 7, 8 - **Input/output token shapes:** 512/128, 1024/256, 2048/128, 2048/512, 4096/512, 8192/512, 1024/2048, 512/4096, 512/8192 Baseline commit (https://github.com/vllm-project/vllm/commit/6bdabbad5bce747865fd3a249658518a4269cc22): ```text commit 6bdabbad5bce747865fd3a249658518a4269cc22 Author: Jee Jee Li <[pandaleefree@gmail.com](mailto:pandaleefree@gmail.com)> Date: Sun May 31 13:16:12 2026 +0800 [CI/Build] Enable Step3p7ForConditionalGeneration testing (#43956) Signed-off-by: Jee Jee Li <[jeejeelee@inferact.ai](mailto:jeejeelee@inferact.ai)> ``` Server script: ```bash #!/bin/bash set -euo pipefail rm -rf ~/.cache/vllm vllm serve Qwen/Qwen3-4B \\ --dtype auto \\ --max-model-len 16384 \\ --gpu-memory-utilization 0.9 ``` Benchmark script: ```bash #!/bin/bash set -euo pipefail BASE_URL=${BASE_URL:-http://localhost:8000} RESULT_DIR=${RESULT_DIR:-./bench-results} RESULT_PREFIX=${RESULT_PREFIX:-qwen3-4b} CONCURRENCIES=${CONCURRENCIES:-\"5 6 7 8\"} NUM_WARMUPS=${NUM_WARMUPS:-16} for workload in \\ \"512 128\" \\ \"1024 256\" \\ \"2048 512\" \\ \"4096 512\" \\ \"8192 512\" \\ \"2048 128\" \\ \"1024 2048\" \\ \"512 4096\" \\ \"512 8192\"; do set -- $workload INPUT_LEN=$1 OUTPUT_LEN=$2 for C in ${CONCURRENCIES}; do NUM_PROMPTS=$((C * 10)) vllm bench serve \\ --backend openai-chat \\ --base-url \"${BASE_URL}\" \\ --endpoint /v1/chat/completions \\ --model Qwen/Qwen3-4B \\ --dataset-name random \\ --input-len \"${INPUT_LEN}\" \\ --output-len \"${OUTPUT_LEN}\" \\ --num-prompts \"${NUM_PROMPTS}\" \\ --request-rate inf \\ --max-concurrency \"${C}\" \\ --num-warmups \"${NUM_WARMUPS}\" \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,99 \\ --save-result \\ --result-dir \"${RESULT_DIR}\" \\ --result-filename \"${RESULT_PREFIX}-${INPUT_LEN}-${OUTPUT_LEN}-C${C}.json\" \\ --metadata model=qwen3-4b concurrency=\"${C}\" workload=random_${INPUT_LEN}_${OUTPUT_LEN} num_prompts=\"${NUM_PROMPTS}\" done done ``` ## Test Results ### Unit tests 216 passed, 16 warnings in 52.55s ### GSM8K end-to-end accuracy | Version | flexible-extract exact_match | strict-match exact_match | Request progress | |---|---|---|---| | **Baseline** | 0.8613 ± 0.0095 | 0.8650 ± 0.0094 | 1319 requests in 11:13, 1.96 it/s | | **SWMMAC** | 0.8605 ± 0.0095 | 0.8666 ± 0.0094 | 1319 requests in 09:48, 2.24 it/s | *The GSM8K accuracy is comparable between the baseline and SWMMAC runs. The SWMMAC run also completed faster in this setup.* ### Performance Benchmark Results The SWMMAC path improves serving throughput across all 36 measured cases. Averaged by workload across concurrency 5-8, request throughput improves by about 10-23%, output token throughput improves by about 7-23%, and mean E2EL decreases by about 6-19%. TTFT stays neutral. #### Performance summary Average percentage change across concurrency 5-8 for each workload: | Metric | 512/128 | 1024/256 | 2048/128 | 2048/512 | 4096/512 | 8192/512 | 1024/2048 | 512/4096 | 512/8192 | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Request throughput | +23.0% | +22.2% | +18.5% | +18.8% | +12.2% | +9.8% | +20.8% | +16.0% | +20.8% | | Output token throughput | +23.2% | +22.5% | +18.5% | +18.5% | +12.0% | +7.2% | +21.2% | +20.2% | +17.0% | | Mean E2EL | -19.0% | -18.2% | -15.5% | -15.8% | -10.8% | -6.5% | -17.5% | -17.0% | -14.5% | | Mean TTFT | -6.0% | -5.8% | -7.5% | -1.8% | +2.8% | +0.0% | -0.2% | -0.5% | -0.2% | #### Detailed performance results | Concurrency | Input | Output | Baseline Req/s | Baseline TTFT ms | Baseline Out tok/s | Baseline E2EL ms | SWMMAC Req/s | SWMMAC TTFT ms | SWMMAC Out tok/s | SWMMAC E2EL ms | Req/s Δ | Req/s Δ% | TTFT Δ ms | TTFT Δ% | Out tok/s Δ | Out tok/s Δ% | E2EL Δ ms | E2EL Δ% | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 5 | 512 | 128 | 1.83 | 91.64 | 233.69 | 2737.24 | 2.24 | 90.12 | 286.76 | 2229.83 | 0.41 | 22% | -1.52 | -2% | 53.07 | 23% | -507.41 | -19% | | 6 | 512 | 128 | 2.15 | 101.55 | 274.57 | 2795.15 | 2.65 | 91.26 | 338.93 | 2264.20 | 0.50 | 23% | -10.29 | -10% | 64.36 | 23% | -530.95 | -19% | | 7 | 512 | 128 | 2.52 | 106.90 | 322.28 | 2778.32 | 3.11 | 100.77 | 397.88 | 2249.62 | 0.59 | 23% | -6.13 | -6% | 75.60 | 23% | -528.70 | -19% | | 8 | 512 | 128 | 2.91 | 104.41 | 372.54 | 2746.35 | 3.60 | 98.61 | 460.84 | 2219.05 | 0.69 | 24% | -5.80 | -6% | 88.30 | 24% | -527.30 | -19% | | 5 | 1024 | 256 | 0.89 | 107.05 | 228.34 | 5604.23 | 1.09 | 110.63 | 278.86 | 4588.01 | 0.20 | 22% | 3.58 | 3% | 50.52 | 22% | -1016.22 | -18% | | 6 | 1024 | 256 | 1.04 | 122.90 | 267.14 | 5747.64 | 1.28 | 115.96 | 327.74 | 4684.68 | 0.24 | 23% | -6.94 | -6% | 60.60 | 23% | -1062.96 | -18% | | 7 | 1024 | 256 | 1.21 | 155.45 | 309.23 | 5791.30 | 1.48 | 146.18 | 379.74 | 4715.45 | 0.27 | 22% | -9.27 | -6% | 70.51 | 23% | -1075.85 | -19% | | 8 | 1024 | 256 | 1.38 | 168.67 | 354.14 | 5779.56 | 1.69 | 145.70 | 433.32 | 4723.14 | 0.31 | 22% | -22.97 | -14% | 79.18 | 22% | -1056.42 | -18% | | 5 | 2048 | 128 | 1.54 | 256.74 | 197.00 | 3244.90 | 1.86 | 221.03 | 238.20 | 2683.05 | 0.32 | 21% | -35.71 | -14% | 41.20 | 21% | -561.85 | -17% | | 6 | 2048 | 128 | 1.78 | 262.92 | 227.79 | 3366.40 | 2.11 | 235.44 | 270.09 | 2838.57 | 0.33 | 19% | -27.48 | -10% | 42.30 | 19% | -527.83 | -16% | | 7 | 2048 | 128 | 2.04 | 287.25 | 260.95 | 3427.79 | 2.41 | 286.41 | 308.32 | 2900.50 | 0.37 | 18% | -0.84 | 0% | 47.37 | 18% | -527.29 | -15% | | 8 | 2048 | 128 | 1.53 | 1062.40 | 195.54 | 5230.33 | 1.77 | 993.84 | 226.15 | 4521.79 | 0.24 | 16% | -68.56 | -6% | 30.61 | 16% | -708.54 | -14% | | 5 | 2048 | 512 | 0.41 | 256.67 | 211.48 | 12101.57 | 0.50 | 253.09 | 253.98 | 10075.56 | 0.09 | 22% | -3.58 | -1% | 42.50 | 20% | -2026.01 | -17% | | 6 | 2048 | 512 | 0.48 | 255.56 | 244.88 | 12540.07 | 0.57 | 235.94 | 294.06 | 10442.02 | 0.09 | 19% | -19.62 | -8% | 49.18 | 20% | -2098.05 | -17% | | 7 | 2048 | 512 | 0.49 | 1018.78 | 252.18 | 14206.53 | 0.58 | 1019.99 | 295.39 | 12127.83 | 0.09 | 18% | 1.21 | 0% | 43.21 | 17% | -2078.70 | -15% | | 8 | 2048 | 512 | 0.55 | 1040.09 | 279.59 | 14643.40 | 0.64 | 1058.69 | 325.90 | 12562.18 | 0.09 | 16% | 18.60 | 2% | 46.31 | 17% | -2081.22 | -14% | | 5 | 4096 | 512 | 0.30 | 1698.20 | 152.78 | 16748.79 | 0.34 | 1751.38 | 173.28 | 14766.71 | 0.04 | 13% | 53.18 | 3% | 20.50 | 13% | -1982.08 | -12% | | 6 | 4096 | 512 | 0.33 | 1769.82 | 171.31 | 17923.55 | 0.38 | 1687.34 | 193.27 | 15886.16 | 0.05 | 15% | -82.48 | -5% | 21.96 | 13% | -2037.39 | -11% | | 7 | 4096 | 512 | 0.37 | 1648.96 | 189.37 | 18914.07 | 0.41 | 1868.01 | 209.70 | 17080.13 | 0.04 | 11% | 219.05 | 13% | 20.33 | 11% | -1833.94 | -10% | | 8 | 4096 | 512 | 0.40 | 1828.76 | 203.68 | 20095.59 | 0.44 | 1821.36 | 226.98 | 18032.25 | 0.04 | 10% | -7.40 | 0% | 23.30 | 11% | -2063.34 | -10% | | 5 | 8192 | 512 | 0.19 | 3829.88 | 99.69 | 25664.15 | 0.21 | 3663.23 | 107.66 | 23763.46 | 0.02 | 11% | -166.65 | -4% | 7.97 | 8% | -1900.69 | -7% | | 6 | 8192 | 512 | 0.21 | 3859.11 | 107.91 | 28448.65 | 0.23 | 3844.42 | 116.08 | 26443.94 | 0.02 | 10% | -14.69 | 0% | 8.17 | 8% | -2004.71 | -7% | | 7 | 8192 | 512 | 0.22 | 4007.26 | 114.38 | 31307.53 | 0.24 | 3968.81 | 122.24 | 29293.29 | 0.02 | 9% | -38.45 | -1% | 7.86 | 7% | -2014.24 | -6% | | 8 | 8192 | 512 | 0.23 | 3929.31 | 119.34 | 34289.29 | 0.25 | 4131.67 | 126.76 | 32281.71 | 0.02 | 9% | 202.36 | 5% | 7.42 | 6% | -2007.58 | -6% | | 5 | 1024 | 2048 | 0.11 | 272.15 | 217.61 | 47052.99 | 0.13 | 257.10 | 264.88 | 38655.63 | 0.02 | 18% | -15.05 | -6% | 47.27 | 22% | -8397.36 | -18% | | 6 | 1024 | 2048 | 0.12 | 319.32 | 252.43 | 48674.07 | 0.15 | 336.41 | 306.06 | 40145.27 | 0.03 | 25% | 17.09 | 5% | 53.63 | 21% | -8528.80 | -18% | | 7 | 1024 | 2048 | 0.14 | 323.50 | 291.21 | 49222.81 | 0.17 | 345.86 | 352.14 | 40706.31 | 0.03 | 21% | 22.36 | 7% | 60.93 | 21% | -8516.50 | -17% | | 8 | 1024 | 2048 | 0.16 | 371.08 | 329.52 | 49714.14 | 0.19 | 345.11 | 398.09 | 41150.64 | 0.03 | 19% | -25.97 | -7% | 68.57 | 21% | -8563.50 | -17% | | 5 | 512 | 4096 | 0.05 | 182.87 | 212.34 | 96449.01 | 0.06 | 183.89 | 256.32 | 79898.13 | 0.01 | 20% | 1.02 | 1% | 43.98 | 21% | -16550.88 | -17% | | 6 | 512 | 4096 | 0.06 | 211.20 | 245.59 | 100068.20 | 0.07 | 208.61 | 295.82 | 83077.61 | 0.01 | 17% | -2.59 | -1% | 50.23 | 20% | -16990.61 | -17% | | 7 | 512 | 4096 | 0.07 | 235.87 | 282.21 | 101596.10 | 0.08 | 234.92 | 338.94 | 84590.28 | 0.01 | 14% | -0.95 | 0% | 56.73 | 20% | -17005.85 | -17% | | 8 | 512 | 4096 | 0.08 | 253.05 | 318.55 | 102863.60 | 0.09 | 248.26 | 381.88 | 85803.16 | 0.01 | 13% | -4.79 | -2% | 63.33 | 20% | -17060.44 | -17% | | 5 | 512 | 8192 | 0.02 | 185.16 | 190.11 | 215451.10 | 0.03 | 183.53 | 223.90 | 182940.48 | 0.01 | 50% | -1.63 | -1% | 33.79 | 18% | -32510.62 | -15% | | 6 | 512 | 8192 | 0.03 | 212.97 | 216.41 | 227118.40 | 0.03 | 212.17 | 253.98 | 193528.35 | 0.00 | 0% | -0.80 | 0% | 37.57 | 17% | -33590.02 | -15% | | 7 | 512 | 8192 | 0.03 | 236.14 | 244.39 | 234637.00 | 0.03 | 235.02 | 285.46 | 200879.94 | 0.00 | 0% | -1.12 | 0% | 41.07 | 17% | -33757.10 | -14% | | 8 | 512 | 8192 | 0.03 | 256.54 | 271.29 | 241564.50 | 0.04 | 255.74 | 315.74 | 207557.14 | 0.01 | 33% | -0.80 | 0% | 44.45 | 16% | -34007.34 | -14% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/45559", "createdAt": "2026-06-14T03:23:39Z", "updatedAt": "2026-08-12T23:01:35Z", "timestamp": "2026-08-12T23:01:35Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "rocm" ], "author": "keneoneth", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45581", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: drop trailing MTP entry from text_config.layer_types", "text": "## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM v0.23.0. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stack. On `transformers` v5, the inner text-config is instantiated as `Exaone4Config` — the legacy `exaone4_5_text` model type is remapped to `exaone4` by the upstream `Exaone4_5_Config.__post_init__` — and `Exaone4Config` has no notion of the MTP layer. Its `@strict` dataclass init runs `validate_layer_type` and fails before vLLM ever gets to the model loader: ``` ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` Full traceback from `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":3}'` on `vllm==0.23.0` / `transformers==5.12.0`: ``` File \".../transformers/models/exaone4_5/configuration_exaone4_5.py\", line 84, in __post_init__ self.text_config = CONFIG_MAPPING[self.text_config[\\\"model_type\\\"]](**self.text_config) File \".../huggingface_hub/dataclasses.py\", line 276, in init_with_validate cls.validate(self) File \".../huggingface_hub/dataclasses.py\", line 253, in validate raise StrictDataclassClassValidationError(...) from e huggingface_hub.errors.StrictDataclassClassValidationError: Class validation error for validator 'validate_layer_type': ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` vLLM v0.23.0's pin `transformers >= 4.56.0, != 5.3.*, ...` rules out the LG-AI `nuxlear` transformers fork pinned at `5.3.0.dev0` (which predates the strict validator and is what the LG-AI repo's deployment recipe still suggests). So the only path forward on mainline is to fix the config on the vLLM side. ## Fix Register a vLLM-side `Exaone4_5Config` (`vllm.transformers_utils.configs.exaone4_5`) that subclasses the upstream config and trims the trailing `layer_types` entry before the inner `Exaone4Config` is built. The trim is gated on `len(layer_types) == num_hidden_layers + 1` exactly; any other shape is left untouched so unrelated config bugs are not silently masked. The MTP layer is consumed separately by the speculative-decode path via `num_nextn_predict_layers` / `_num_mtp_layers` (and the MTP weights themselves come from the `mtp.*` checkpoint keys), so dropping the metadata entry in `layer_types` does not change MTP behavior — it only unblocks the inner `Exaone4Config` validation. 3 files, 57 lines added: - `vllm/transformers_utils/configs/exaone4_5.py` (new, 54 lines) - `vllm/transformers_utils/configs/__init__.py` (+2 — register in `_CLASS_TO_MODULE` and `__all__`) - `vllm/transformers_utils/config.py` (+1 — map `exaone4_5` → `Exaone4_5Config` in `_CONFIG_REGISTRY`) ## Test Manually verified end-to-end on the following stack: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before the patch: `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails at engine init with the validator error above. After the patch the same command starts cleanly and the speculative-decode path is active: ``` INFO [model.py:611] Resolved architecture: Exaone4_5_MTP INFO [llm_base_proposer.py:1347] Detected MTP model. Sharing target model embedding weights with the draft model. INFO [llm_base_proposer.py:1403] Detected MTP model. Sharing target model lm_head weights with the draft model. ``` `vllm bench serve --dataset-name sonnet --dataset-path benchmarks/sonnet.txt --sonnet-input-len 550 --sonnet-output-len 256 --sonnet-prefix-len 200 --num-prompts 64 --max-concurrency 16 --request-rate inf` against `LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":3}'`: ``` ============ Serving Benchmark Result ============ Successful requests: 64 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 24.06 Total input tokens: 34915 Total generated tokens: 16384 Request throughput (req/s): 2.66 Output token throughput (tok/s): 680.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.51 Median TPOT (ms): 20.66 P99 TPOT (ms): 24.84 ---------------Speculative Decoding--------------- Acceptance rate (%): 35.91 Acceptance length: 2.08 Drafts: 7877 Draft tokens: 23631 Accepted tokens: 8486 Per-position acceptance (%): Position 0: 59.64 Position 1: 31.64 Position 2: 16.45 ================================================== ``` The per-position monotonic decay (≈60% / 32% / 16%) matches what `[speculative.py:722]` warns about for `num_speculative_tokens > 1` and is consistent with the published MTP behavior for this checkpoint family on real text. (A control run with `--dataset-name random` reports near-zero acceptance, as expected — random tokens are out-of-distribution for a MTP head trained on text.) ## Notes - This PR only touches the configuration shim path. The vLLM `exaone4_5` model module and the `Exaone4_5_MTP` speculator are not modified. - Separately, on `vllm/model_executor/models/exaone4_5.py:243`, `Exaone4_5_VisionBlock.forward()` still uses `seqlens=` while the Qwen2.5-VL caller it inherits from passes `sequence_lengths=` since the upstream rename in `qwen2_5_vl.py`. That is an unrelated bug and out of scope for this PR; happy to send it as a follow-up if helpful.", "url": "https://github.com/vllm-project/vllm/pull/45581", "createdAt": "2026-06-14T12:48:26Z", "updatedAt": "2026-08-13T07:57:36Z", "timestamp": "2026-08-13T07:57:36Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build", "v1", "cpu" ], "author": "seongyun1104", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45583", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: rename Exaone4_5_VisionBlock.forward kwarg seqlens -> sequence_lengths", "text": "## Purpose `EXAONE4_5_VisionTransformer` inherits from `Qwen2_5_VisionTransformer`, and the parent class's vision-block loop (`vllm/model_executor/models/qwen2_5_vl.py:1100`) now invokes each block with `sequence_lengths=` rather than the older `seqlens=`: ```python hidden_states = blk( hidden_states, cu_seqlens=cu_seqlens_now, rotary_pos_emb_cos=rotary_pos_emb_cos, rotary_pos_emb_sin=rotary_pos_emb_sin, max_seqlen=max_seqlen_now, sequence_lengths=sequence_lengths_now, ) ``` `Exaone4_5_VisionBlock.forward()` (`vllm/model_executor/models/exaone4_5.py:243`) still declares the kwarg as `seqlens`, so loading any `Gemma4ForConditionalGeneration`-style EXAONE 4.5 checkpoint through the multimodal path raises during multi-modal warmup before the engine is ready to serve: ``` TypeError: Exaone4_5_VisionBlock.forward() got an unexpected keyword argument 'sequence_lengths' File \".../vllm/model_executor/models/qwen2_5_vl.py\", line 1100, in forward hidden_states = blk( ^^^^ ``` ## Fix Rename the EXAONE block's xFormers-only kwarg from `seqlens` to `sequence_lengths` so it matches the Qwen2.5-VL caller's signature. The argument is unused inside the block body — only `cu_seqlens`, `max_seqlen`, and the rotary pos embeddings are forwarded into `self.attn` — so this is purely a keyword-argument name fix with no behavior change. 1 file, 1 line: \\`\\`\\` vllm/model_executor/models/exaone4_5.py | 2 +- \\`\\`\\` ## Test Verified end-to-end on the same stack used in #45581: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before this patch: `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails during multi-modal warmup with the TypeError above. After this patch the multimodal warmup completes and the server reaches \\`Application startup complete\\`: \\`\\`\\` INFO ... [base.py:227] Multi-modal warmup completed in 3.886s INFO ... [base.py:227] Readonly multi-modal warmup completed in 0.125s INFO ... [api_server.py:583] Starting vLLM server on http://0.0.0.0:8005 \\`\\`\\` The reproducer / test stack and the bench-serve numbers are identical to the verification block in #45581 — once both patches land, EXAONE 4.5 + MTP loads and serves on mainline vLLM + mainline transformers with no fork dependency. ## Related - #45581: drops the trailing MTP entry from \\`text_config.layer_types\\` so the base config validation passes. This PR addresses the next failure that surfaces once #45581 unblocks engine init.", "url": "https://github.com/vllm-project/vllm/pull/45583", "createdAt": "2026-06-14T12:56:23Z", "updatedAt": "2026-08-13T07:57:33Z", "timestamp": "2026-08-13T07:57:33Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build", "v1", "cpu" ], "author": "seongyun1104", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45584", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: trim trailing MTP entry from text_config.layer_types", "text": "## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stack. (Still present in the current `LGAI-EXAONE/EXAONE-4.5-33B-FP8` config: `num_hidden_layers=64`, `len(layer_types)=65`.) On `transformers` v5 the inner text-config is instantiated as `Exaone4Config` — the `exaone4_5_text` model type is remapped to `exaone4` by the upstream `Exaone4_5_Config.__post_init__` — and `Exaone4Config` has no notion of the MTP layer. Its `@strict` dataclass init runs `validate_layer_type` and fails before vLLM ever reaches the model loader: ``` ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` Full traceback from `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'`: ``` File \".../transformers/models/exaone4_5/configuration_exaone4_5.py\", line 84, in __post_init__ self.text_config = CONFIG_MAPPING[self.text_config[\"model_type\"]](**self.text_config) File \".../huggingface_hub/dataclasses.py\", line 276, in init_with_validate cls.validate(self) huggingface_hub.errors.StrictDataclassClassValidationError: Class validation error for validator 'validate_layer_type': ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` ## Fix Register a vLLM-side `Exaone4_5Config` (`vllm.transformers_utils.configs.exaone4_5`) that subclasses the upstream config and trims the trailing `layer_types` entry before the inner `Exaone4Config` is built. The trim is gated on `len(layer_types) == num_hidden_layers + 1` exactly; any other shape is left untouched so unrelated config bugs are not silently masked. The MTP layer is consumed separately by the speculative-decode path via `num_nextn_predict_layers` (and the MTP weights come from the `mtp.*` checkpoint keys), so dropping the metadata entry in `layer_types` does not change MTP behavior — it only unblocks the inner `Exaone4Config` validation. 3 files, 60 lines added: - `vllm/transformers_utils/configs/exaone4_5.py` (new) - `vllm/transformers_utils/configs/__init__.py` (+2 — register in `_CLASS_TO_MODULE` and `__all__`) - `vllm/transformers_utils/config.py` (+1 — map `exaone4_5` → `Exaone4_5Config` in `_CONFIG_REGISTRY`) ## Not a duplicate No open PR or issue currently addresses the EXAONE 4.5 `layer_types` / MTP config validation failure. The recently merged #45073 fixes a separate surface (the vision encoder `sequence_lengths` kwarg) and does not touch the text-config validation path. ## Test Manually verified end-to-end on: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before the patch, `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails at engine init with the `validate_layer_type` ValueError above. After the patch the same command starts cleanly and the speculative-decode path is active: ``` INFO [model.py:611] Resolved architecture: Exaone4_5_MTP INFO [llm_base_proposer.py:1347] Detected MTP model. Sharing target model embedding weights with the draft model. INFO [llm_base_proposer.py:1403] Detected MTP model. Sharing target model lm_head weights with the draft model. ``` `vllm bench serve --dataset-name sonnet --dataset-path benchmarks/sonnet.txt --sonnet-input-len 550 --sonnet-output-len 256 --sonnet-prefix-len 200 --num-prompts 64 --max-concurrency 16 --request-rate inf` against `LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'`: ``` ============ Serving Benchmark Result ============ Successful requests: 64 Maximum request concurrency: 16 Benchmark duration (s): 24.06 Total input tokens: 34915 Total generated tokens: 16384 Request throughput (req/s): 2.66 Output token throughput (tok/s): 680.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.51 Median TPOT (ms): 20.66 P99 TPOT (ms): 24.84 ---------------Speculative Decoding--------------- Acceptance rate (%): 35.91 Acceptance length: 2.08 Per-position acceptance (%): Position 0: 59.64 Position 1: 31.64 Position 2: 16.45 ================================================== ``` The per-position monotonic decay (≈60% / 32% / 16%) is consistent with the published MTP behavior for this checkpoint family on real text. (A control run with `--dataset-name random` reports near-zero acceptance, as expected.) ## AI assistance AI assistance (Claude) was used while developing this change. I reviewed every changed line, understand the fix end-to-end, and ran the verification above myself on my own RTX PRO 6000 instance. The AI contribution is recorded with a `Co-authored-by:` trailer on the commit per `AGENTS.md`.", "url": "https://github.com/vllm-project/vllm/pull/45584", "createdAt": "2026-06-14T13:32:28Z", "updatedAt": "2026-08-13T13:40:29Z", "timestamp": "2026-08-13T13:40:29Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "bug" ], "author": "seongyun1104", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45802", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine", "text": "## Purpose Add token-aware reasoning token counting for the Streaming Parser Engine and surface the count through OpenAI-compatible usage fields. - Adds `completion_tokens_details.reasoning_tokens` to usage responses. - Propagates token counts through the parser engine pipeline: `TokenIDScanner -> IncrementalLexer -> StreamingParserEngine -> SemanticEvent`. - Counts only `REASONING_CHUNK` tokens, excluding reasoning boundary terminals such as `` / ``. - Reuses the engine-based streaming parser path for non-streaming parsing when output token IDs are available, so streaming and non-streaming reasoning token accounting use the same token-aware boundaries. - Ensures parser instances are scoped per output choice in chat completion handling. - Adds coverage for parser-engine token counting, chat completion usage, and Responses API reasoning token accounting. ## Test Plan ## Test Result ``` curl http://localhost:8001/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -H \"Authorization: Bearer EMPTY\" \\ -d '{ \"model\": \"\", \"messages\": [ { \"role\": \"user\", \"content\": \"北京什么天气？\" } ],\"chat_template_kwargs\":{\"enable_thinking\": true}, \"tools\": [ { \"type\": \"function\",\"strict\":true, \"function\": { \"name\": \"get_weather\", \"description\": \"Get current weather for a city\", \"parameters\": { \"type\": \"object\", \"properties\": { \"city\": { \"type\": \"string\" } }, \"required\": [\"city\"] } } } ], \"tool_choice\": \"auto\", \"stream\": false }' {\"id\":\"chatcmpl-80b2a0ffa82be55a\",\"object\":\"chat.completion\",\"created\":1782098609,\"model\":\"/mnt/data4/models/Qwen/Qwen3.5-35B-A3B-FP8\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":null,\"refusal\":null,\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":[{\"id\":\"chatcmpl-tool-ac88f083f78ccffb\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\": \\\"北京\\\"}\"}}],\"reasoning\":\"用户询问北京的天气情况。我需要使用get_weather工具来获取北京的当前天气信息。\\n\"},\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"stop_reason\":null,\"token_ids\":null,\"routed_experts\":null}],\"service_tier\":null,\"system_fingerprint\":\"vllm-0.23.1rc1.dev243+gb529bfd6c-tp2-4b4f7693\",\"usage\":{\"prompt_tokens\":277,\"total_tokens\":323,\"completion_tokens\":46,\"prompt_tokens_details\":null,\"completion_tokens_details\":{\"reasoning_tokens\":18}},\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null} ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/45802", "createdAt": "2026-06-16T09:31:21Z", "updatedAt": "2026-08-13T17:29:02Z", "timestamp": "2026-08-13T17:29:02Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "frontend", "ready", "tool-calling" ], "author": "chaunceyjiang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:45899", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit tests for the chunked scaled-dot KKT kernels", "text": "## Purpose Adds unit tests for both chunked scaled-dot K@K^T kernels in `tests/kernels/test_chunk_scaled_dot_kkt.py`, neither of which had coverage: - `chunk_scaled_dot_kkt_fwd` (GDN/FLA) — scalar gate, also serves grouped K heads - `chunk_kda_scaled_dot_kkt_fwd` (KDA) — per-channel gate, also returns the non-strictly causal `Aqk` ## Test Plan 16 cases (8 per kernel, bfloat16 and float32) against a chunk-wise PyTorch reference, covering partial chunks, gated and ungated paths, GQA head expansion and ragged varlen input. Sequences are laid out either as a padded batch or as a packed varlen run, so both encodings meet the same reference. Inputs follow the production layers: l2-normalised q/k, fp32 sigmoid `beta`, and a `chunk_local_cumsum` log-gate damped so the causal block does not underflow to zero. ## Test Result All 16 pass on IntelB70 and H200", "url": "https://github.com/vllm-project/vllm/pull/45899", "createdAt": "2026-06-17T06:38:34Z", "updatedAt": "2026-08-13T12:22:21Z", "timestamp": "2026-08-13T12:22:21Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MoE] Preserve unquantized weight storage on ROCm", "text": "## Summary - Preserve unquantized MoE parameter storage during ROCm post-load padding by copying the padded view into existing storage instead of rebinding `.data`. - Return directly after AITER unquantized MoE `shuffle_weights()` so the AITER runtime layout is not forced through an extra `.contiguous()` conversion. - Add a ROCm MoE weight replay test covering both AITER and TRITON backends. ## Motivation In RL-style serving and training loops, external weight updates can trigger `process_weights_after_loading()` more than once after model initialization. Reassigning `param.data` in that path can replace the underlying parameter storage. That is unsafe once CUDA graphs or runtime kernels have captured parameter storage addresses, and can leave later rollouts reading stale or invalid storage. We observed this as a training-time rollout corruption issue after weight updates: generated responses became invalid, and reward/score collapsed to 0. Preserving the existing parameter storage with `.copy_()` keeps captured addresses stable while still applying the padded layout. For AITER unquantized MoE, `shuffle_weights()` already returns the runtime layout consumed by AITER. In local Qwen3.5-35B GRPO measurements, avoiding the final contiguous conversion for AITER was slightly faster after normalizing rollout generation time by response length. ## Replay test The new `tests/rocm/test_moe_weight_replay.py` test constructs a Qwen3.5 MoE LLM with sleep mode and CUDA graphs, generates a deterministic greedy response, runs level-2 sleep/wake, then replays checkpoint-format weights through a framework-agnostic load-then-post-load helper: 1. stream checkpoint-format weights into `model.load_weights(...)` in buckets; 2. call `process_weights_after_loading(...)` once after all buckets are loaded; 3. verify the ROCm MoE padding branch was actually exercised; 4. verify MoE `w13_weight` / `w2_weight` parameter storage addresses do not change; 5. verify greedy token IDs are deterministic after replay. This covers both AITER and TRITON MoE backends as first-class correctness cases. ## Local validation Static/lightweight checks in the PR checkout: - `python3 -m py_compile tests/rocm/test_moe_weight_replay.py vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method.py vllm/model_executor/layers/fused_moe/oracle/unquantized.py` - `python3 -m ruff check tests/rocm/test_moe_weight_replay.py vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method.py vllm/model_executor/layers/fused_moe/oracle/unquantized.py` - `python3 -m pytest tests/kernels/moe/test_unquantized_backend_selection.py tests/kernels/moe/test_moe_weight_loading_padded.py` - 38 passed, 2 skipped Heavy replay validation with Qwen3.5-35B-A3B and tensor parallel size 2: - pre-fix AITER: failed as expected; the test entered the ROCm padding branch and detected MoE parameter `data_ptr` changes. - pre-fix TRITON: failed as expected; the test entered the ROCm padding branch and detected MoE parameter `data_ptr` changes. - fixed AITER: passed. - fixed TRITON: passed. ## Benchmark note Local 10-step Qwen3.5-35B GRPO runs kept score/reward normal with zero aborted responses. Normalized rollout timing for AITER was slightly better without the final contiguous conversion: about 22.95 ms/token vs 23.11 ms/token.", "url": "https://github.com/vllm-project/vllm/pull/46009", "createdAt": "2026-06-18T07:15:27Z", "updatedAt": "2026-08-13T06:40:29Z", "timestamp": "2026-08-13T06:40:29Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "rocm", "ready", "ci/build", "quantization" ], "author": "aaab8b", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46064", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for _fwd_kernel_ep_scatter_1 and _fwd_kernel_ep_…", "text": "## Purpose Adds a unit test for the `_fwd_kernel_ep_scatter_1` and `_fwd_kernel_ep_scatter_2` Triton kernels (DeepGemm group) in `tests/kernels/moe/test_ep_scatter.py`. Both kernels run inside the `ep_scatter` wrapper and had no upstream coverage: - `_fwd_kernel_ep_scatter_1` computes per-expert region offsets and fills `m_indices` (expert label per output row, -1 for padding), counts aligned to BLOCK_E=128. - `_fwd_kernel_ep_scatter_2` copies each token's activation and per-group scale into the row of its (optionally remapped) expert, recording the destination in `output_index`. Phase 1 is checked by comparing `m_indices` against a reference region layout. Phase 2 is checked by reading the destination back out of the kernel: each token's `output_index` must point to a row labelled with its expert in `m_indices`, and that row must hold both the token's activation and its per-group scale. A non-trivial `expert_map` case is included. Device-agnostic (`current_platform.device_type`); runs on CUDA and XPU. ## Test Plan `python -m pytest tests/kernels/moe/test_ep_scatter.py -v` ## Test Result All 15 cases pass.", "url": "https://github.com/vllm-project/vllm/pull/46064", "createdAt": "2026-06-18T15:40:22Z", "updatedAt": "2026-08-13T07:57:27Z", "timestamp": "2026-08-13T07:57:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46067", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][TurboQuant] Fix dangling decode scratch when workspace grows after cudagraph capture", "text": "## Summary Fixes a CUDA \"illegal memory access\" that kills the engine on the **first decode step** when serving with `--kv-cache-dtype turboquant_*` (see issue: #45670) and CUDA graphs enabled (the default `FULL_AND_PIECEWISE`). ## Root cause TurboQuant decode (`_decode_attention`) sourced its scratch buffers (`mid_o`/`output`/`lse`) from the **growable** `WorkspaceManager`. Decode runs inside the **FULL** cudagraph, so those buffer addresses are baked into the captured graphs at capture time. The `WorkspaceManager` frees and reallocates its buffer when it grows — and calls `empty_cache()`, unmapping the old address. CUDA graphs are captured for ascending batch sizes (B=1..max); each larger capture grows the workspace, freeing the address an earlier-captured (smaller-batch) graph still points at. Replaying that graph — e.g. the very first decode (B=1) — dereferences freed memory. The error surfaces asynchronously at the next synchronizing CUDA call (the H2D copy in `prepare_inputs`), which is why the reported traceback looked unrelated to attention. Symptoms that localized it: - `--enforce-eager` works (no graphs). - `--compilation-config '{\"cudagraph_mode\": \"PIECEWISE\"}'` works (decode attention runs eagerly, never captured). - Reproduces on every `turboquant_*` preset (scratch size is preset-independent); no non-turboquant dtype is affected. ## Fix Give TurboQuant decode a **dedicated scratch buffer**, allocated **once at the max cudagraph capture batch** and reused (sliced `[:B]`) by every TQ layer and every captured decode graph. Because it is never resized and is separate from the `WorkspaceManager`, its address never moves — so the addresses baked into the FULL graphs stay valid, including against a later long continuation-prefill that grows the workspace. The eager path (no cudagraphs, or a batch beyond the captured sizes) still uses the workspace, where there is no graph to dangle. Files: - `vllm/v1/attention/backends/turboquant_attn.py` — module-level `_DECODE_SCRATCH` + `_get_decode_scratch` + `reset_tq_decode_scratch`; `_decode_attention` routes captured batches to the fixed buffer, eager batches to the workspace. - `vllm/v1/worker/gpu/shutdown.py` — release the scratch on model-runner teardown. ## Not a duplicate Fix to the in-tree TurboQuant backend; no existing open PR addresses the FULL-cudagraph workspace-pointer staleness in TurboQuant decode. ## Test plan > Run on a GPU box (validated config: Qwen3-8B, GB200). Original failure (crashes before this PR, serves after): ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 ``` Eager / piecewise unaffected: ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 --enforce-eager vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 \\ --compilation-config '{\"cudagraph_mode\": \"PIECEWISE\"}' ``` Other presets (same root cause): ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_4bit_nc ``` Unit: ``` pytest tests/quantization/test_turboquant.py ``` **Results:** _to be filled in by the submitter after running on GPU_ — this branch was authored on a machine without a GPU, so the commands above have not yet been executed here. ## AI assistance Authored with the assistance of an AI coding assistant (Claude Code). All changed lines have been reviewed by the submitter, who is responsible for defending the change and running the tests above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/46067", "createdAt": "2026-06-18T15:59:03Z", "updatedAt": "2026-08-12T19:44:14Z", "timestamp": "2026-08-12T19:44:14Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "v1", "nvidia", "quantization", "mrv2" ], "author": "ldkhang1201", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:46134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for merge_attn_states kernel", "text": "## Purpose Adds a device-agnostic unit test for the `merge_attn_states` Triton kernel (Attention group) in `tests/kernels/attention/test_merge_attn_states_torch_ref.py`. The existing `test_merge_attn_states.py` compares the Triton kernel against a custom CUDA op (`merge_attn_states_cuda`) and skips on non-CUDA platforms, so it cannot run on XPU. This test instead uses a naive PyTorch reference as the oracle (Section 2.2 of arxiv 2501.01005 — the softmax-weighted combination of two split-KV partial attention results), making it runnable on both CUDA and XPU (`current_platform.device_type`, no skip guard). Kept as a separate file so the existing CUDA-parity test is preserved. Covers output and output_lse across token counts, head counts, head sizes, and fp16/bf16, including ~10% +inf lse entries (a split that attended to no keys). ## Test Plan `python -m pytest tests/kernels/attention/test_merge_attn_states_torch_ref.py -v` ## Test Result All 36 cases pass.", "url": "https://github.com/vllm-project/vllm/pull/46134", "createdAt": "2026-06-19T06:54:28Z", "updatedAt": "2026-08-13T07:54:02Z", "timestamp": "2026-08-13T07:54:02Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46336", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Profiler] Add execution trace capture to torch profiler config", "text": "## Summary Adds a `torch_profiler_execution_trace` option to `ProfilerConfig` that captures a PyTorch **execution trace (ET)** alongside the existing Kineto trace, wired through the existing `--profiler-config` flow. Previously vLLM's profiler config could only collect the Kineto/Chrome trace via `tensorboard_trace_handler` — `TorchProfilerWrapper` built `torch.profiler.profile(...)` without an `ExecutionTraceObserver`, and there was no flag to enable one. Collecting an ET (needed for tools like Chakra/PARAM replay and graph-level analysis) required hand-editing `gpu_model_runner._model_forward()`. This makes ET a first-class config option, with no per-model code changes. ## Changes - **`vllm/config/profiler.py`**: new `torch_profiler_execution_trace: bool = False` field, validated to require `profiler=\"torch\"`. - **`vllm/profiler/wrapper.py`**: when enabled, `TorchProfilerWrapper` registers a `torch.profiler.ExecutionTraceObserver` writing `execution_trace_.json` under `torch_profiler_dir`, passes it to `torch.profiler.profile(execution_trace_observer=...)`, and calls `cleanup()` on stop (vLLM stops the profiler directly rather than via a context manager, so the observer is not auto-cleaned). Skipped with a warning for URI trace dirs (gs://, s3://, …) since ET writes only to local files. - **`docs/contributing/profiling.md`**: documents the new option. ## Usage ```bash vllm serve meta-llama/Llama-3.1-8B-Instruct \\ --profiler-config '{\"profiler\":\"torch\",\"torch_profiler_dir\":\"/tmp/prof\",\"torch_profiler_execution_trace\":true}' ``` Produces `execution_trace_.json` per worker, alongside the Kineto trace, and respects the existing `delay_iterations`/`max_iterations` controls. ## Testing Added to `tests/v1/worker/test_gpu_profiler.py`: - `torch_profiler_execution_trace` requires `profiler=\"torch\"` (validation). - Disabled by default / enabled when set. - No `ExecutionTraceObserver` registered when disabled, and skipped for URI dirs. - **CPU-only end-to-end** test: runs the wrapper, asserts a valid `execution_trace_*.json` containing `aten::` ops is produced (no GPU required, CI-friendly). ``` $ python -m pytest tests/v1/worker/test_gpu_profiler.py -q 31 passed ``` Also verified end-to-end on Intel XPU (`CPU+XPU` activities): the wrapper produced `execution_trace_rank0.json` (real `aten::matmul`/`aten::mm`/… nodes) together with the `.pt.trace.json.gz` Kineto trace, observer enabling/disabling cleanly. Ruff lint + format clean; lines ≤88 chars. ## Not a duplicate The only other open PR touching `ProfilerConfig`/`test_gpu_profiler.py` is #37524, which adds profiler capture *during CUDA graph capture* plus roofline/KV trace annotations — a different feature that does not add execution-trace capture. No open PR adds an `ExecutionTraceObserver` / `torch_profiler_execution_trace` option. ## Known limitation When a profiler schedule is active (`warmup_iterations`/`wait_iterations`), the single ET observer captures the whole start→stop window rather than following the per-cycle schedule. The common (unscheduled) path is unaffected. ## Notes - AI assistance (Claude) was used in preparing this change.", "url": "https://github.com/vllm-project/vllm/pull/46336", "createdAt": "2026-06-22T05:21:24Z", "updatedAt": "2026-08-13T10:16:34Z", "timestamp": "2026-08-13T10:16:34Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "documentation", "v1" ], "author": "sachinkademane", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46340", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] TD operand loads for batched MoE GEMM (moe_mmk) on XPU", "text": "### Purpose Add a Tensor-Descriptor (TD) operand-load path to `moe_mmk` / `expert_triton_kernel` (the batched MoE expert GEMM) and enable it on XPU. On XPU, masked `tl.load` feeding `tl.dot` bypasses the Xe XMX 2D-block-read path; loading both operands via `tl.make_tensor_descriptor` restores it. Output is bit-identical to the non-TD path. The batched kernel is the one used by **low-latency Expert-Parallel** dispatch (the batched activation format `[E_local, max_tokens, K]`), so its e2e effect must be measured **in that EP context**, not against the single-GPU fused MoE (a different path, rightly faster when a model fits on one GPU). This addresses the earlier review feedback: the correct comparison is **TD-on vs TD-off on the batched EP path**, which is what this PR touches. (Earlier single-GPU GraniteMoE numbers were misleading and are superseded below.) Gated by `VLLM_TRITON_USE_TD` (auto-on for XPU, opt-in on CUDA sm90+). Stacks under the portable low-latency all-to-all EP backend (#46871). ### Test Plan - Kernel unit tests: `tests/kernels/moe/test_batched_moe.py` (TD vs plain bit-exact, zero-expert handling, device enablement, backend selection). - E2E via `vllm serve` + `vllm bench serve` on **2× Intel Arc B70**, DeepSeek-V2-Lite, batched Triton EP path (`--all2all-backend naive_low_latency`), bf16, `--enforce-eager`, seed 42. Isolates TD by toggling only `VLLM_TRITON_USE_TD`. (`VLLM_MLA_DISABLE=1` — DeepSeek MLA on XPU has a separate pre-existing numerics bug unrelated to this kernel.) - Accuracy: gsm8k on GraniteMoE (standard attention → clean XPU read) confirms TD is neutral. ### Test Result ### Per-kernel microbench - Arc B70 (E=8, K=N=2048, bf16): TD −89…−91% latency across tokens/expert. - RTX 5060 / H200: TD −68…−70% (CUDA), gains scale with expert GEMM size; bit-exact. ### E2E on the batched low-latency EP path — 2× Arc B70, DeepSeek-V2-Lite (TD off vs on) **Saturation (ShareGPT, 500 prompts):** | metric | TD0 | TD1 | Δ TD1 vs TD0 | |---|---|---|---| | output tok/s | 134.2 | 290.6 | **+116%** | | median TTFT | 27.1 s | 12.0 s | **−56%** | | median TPOT | 797 ms | 334 ms | **−58%** | **Single-stream (`--max-concurrency 1`, realistic low-latency regime):** | metric | TD0 | TD1 | Δ | no-EP default | |---|---|---|---|---| | median TTFT | 1033 ms | 600 ms | −42% | 427 ms | | median TPOT | 353 ms | 270 ms | −24% | 124 ms | ### CUDA cross-check (DeepEP low-latency, Qwen3-30B-A3B, 4× H200) +72% req/s, −53% TPOT with TD on; per-kernel 1.3–6.1× (scales with K/N); bit-identical. (Logs: `cuda_td_deepep_ll_logs.zip`, attached below in a comment) ### Accuracy (TD is a bit-exact operand-load change) GraniteMoE gsm8k (8-shot, 200): default 0.500 / TD0 0.480 / TD1 0.470 — flat within noise, matching the model's published ~0.49. Takeaway: TD is a large, consistent win on the batched EP path it actually touches (+116% throughput, −58% TPOT on 2× Arc B70; +72% req/s on H200), bit-exact, accuracy-neutral, never a regression. Full XPU logs: [pr46340_bench_logs.zip](https://github.com/user-attachments/files/30139331/pr46340_bench_logs.zip)", "url": "https://github.com/vllm-project/vllm/pull/46340", "createdAt": "2026-06-22T07:08:49Z", "updatedAt": "2026-08-13T14:54:11Z", "timestamp": "2026-08-13T14:54:11Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "intel-gpu", "ready", "v1", "verified" ], "author": "oonyshch", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46370", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Initial support for fault tolerant ep using scale-down", "text": "## Purpose This PR adds **scale-down** and **auto-recovery** support to the vLLM fault tolerance (FT) framework for DP+EP MoE deployments (external LB mode). The existing FT framework (`--enable-fault-tolerance`) supports `retry` — reinitializing the DP process group after a transient fault where all ranks survive. When a DP rank permanently dies (hardware failure, OOM kill, etc.), `retry` alone is insufficient: the dead rank's expert slots become unreachable and EPLB placement tables reference non-existent slots. This PR introduces two complementary capabilities: 1. **Scale-down** (`POST /fault_tolerance/apply` with `instruction: \"scale_down\"`): Removes dead DP ranks, masks the corresponding EP ranks at the all2all kernel level, redistributes orphaned logical experts onto surviving ranks using EPLB redundant slots, selectively reloads only reassigned expert weights from disk, rebuilds the DP Gloo group over the surviving slots, and resumes inference — all without a full restart. 2. **Auto-recovery** (`--fault-tolerance-config='{\"auto_recovery\": true}'`): Lets the engine self-heal without external orchestrator commands. On fault, the engine queries the all2all mask from its workers and unions it cluster-wide via the DP store: if no new dead ranks are detected (transient fault), it auto-retries; otherwise it derives the dead DP ranks from the EP mask and auto-scales-down. ### Design - **Slots model: EP topology unchanged** From vLLM's point of view, a removed rank remains as an empty slot: `parallel_config.data_parallel_size` / `data_parallel_rank` stay frozen at their initial values and DP rank ids never change, so there is exactly one coordinate system (original) and no densification anywhere. `removed_dp_ranks` is always in original coordinates and is cumulative-safe — it may include ranks removed by earlier rounds; the engine applies only the increment. Likewise the all2all kernel keeps the original EP world size with dead EP ranks masked via `update_mask()` (both `deepep_low_latency` and `nixl_ep`), which avoids model re-init, graph recompile, and CUDA graph re-capture. Only the rebuilt Gloo `cpu_group` uses dense internal ranks over `sorted(alive)`, computed at reinit time and never written back to config. - **EPLB is disabled after scale-down and re-enabled by a future scale-up.** Post-scale-down the EP/EPLB groups still span the original EP world size, so dynamic rebalancing would attempt collectives with dead ranks; EPLB async rebalancing is therefore suppressed (`eep_eplb_suppressed`) until a scale-up restores the slots. - **Failed recovery is retryable.** The engine commits the topology change (`data_parallel_master_ip`, cumulative dead set) only after the Gloo group reinit succeeds, so a failed attempt leaves a consistent state that can be re-dispatched. Invalid commands (own-rank removal, fully stale removal, <1 survivor) are rejected up front with an observable `ft_error` — the orchestrator re-reads status and re-issues. - **The orchestrator can pass an explicit `request_id` to identify the recovery round.** All engines in one round share the same `request_id`, which namespaces that round's coordination keys in the DP store so concurrent or stale rounds cannot cross-talk. When empty, it falls back to the engine's local reinit epoch. - **Auto-recovery rejects external commands while it is in progress.** A `_recovery_lock` serializes auto-recovery (busy-loop thread) against external commands (input-sockets thread); a command arriving during an ongoing recovery is rejected with a recorded `ft_error`, keeping a single recovery driver per engine at any time. ### Implementation Notes - **TP sibling lockstep barrier (TP > 1).** `_synchronize_dp_ranks` (v1) and `sync_cudagraph_and_dp_padding` (v2) run a per-step Gloo barrier over the TP CPU group when FT is enabled and `tp_size > 1`, preventing intra-TP deadlock: a sibling whose peer stopped arriving fails on the host within `cpu_distributed_timeout_seconds`, never leaving an orphaned TP collective (e.g. fused allreduce kernels without timeout) running on its device stream. The TP CPU group is rebuilt during worker `retry`. - **nixl_ep dispatcher sync.** After masking, each MoE layer's cached `num_dispatchers` is rewritten to the nixl_ep kernel's `active_rank_bound` (highest surviving EP rank + 1); the kernel sizes combine output by `active_rank_bound` and asserts the width matches. DeepEP-LL keeps a fixed num_ranks-wide layout and needs no sync. - **Dead-column neutralization.** Under the slots model the DP allreduce tensors keep their original width and a dead rank's column stays 0 after the SUM allreduce, which is not neutral for min/`all(==1)` aggregations. After the allreduce, each rank locally fills dead columns (from `get_dp_group().dead_dp_ranks`) with aggregate-neutral values. - **EPLB async state reset.** A new `ft_reset_epoch` on the EPLB state lets the async worker detect a recovery and abandon in-flight transfers instead of applying stale results; the async thread survives exceptions when FT is enabled. - **Side-stream mask query.** Worker `query_mask` runs on a separate CUDA stream so a poisoned main stream cannot block the mask read that auto-recovery and status reporting depend on. ### Known limitations - `scale_down` requires EPLB with `num_redundant_experts > 0`. - Requires an FT-capable all2all backend (`deepep_low_latency` or `nixl_ep`) and external LB mode with one engine per API server. - EPLB dynamic rebalancing stays suppressed after scale-down (lifted by a future `scale_up`). - Recovery is not guaranteed for faults occurring at certain stages (e.g. mid weight-transfer in EPLB). - Auto-recovery cannot self-heal the death of DP rank 0 (the coordination TCPStore master is hosted by rank 0); the engine fails closed and waits for an external `scale_down` command with `dp_store_port`/`dp_master_ip`. ## Test Plan E2E tests live in `tests/v1/fault_tolerance/test_fault_tolerance_e2e.py` (gated on `has_nixl_ep`): 1. `test_injected_fault_retry_recovers_all_ranks` (DP=2) — injected fault, all ranks UNHEALTHY, `retry`, all ranks HEALTHY and serving. 2. `test_scale_down_removes_dead_rank_and_recovers` (DP=4, EPLB with 32 redundant experts) — SIGKILL rank 1's worker; survivors go UNHEALTHY, victim's engine goes DEAD and rejects `retry` with a recorded `ft_error`; `scale_down(removed_dp_ranks=[1])` to all survivors; all survivors return to HEALTHY and serve completions. Manual cluster validation (external LB, 4 independent API servers, one engine each): - Model: DeepSeek-V2-Lite-Chat (MoE, 64 logical experts), `--data-parallel-size 4 --tensor-parallel-size 2` (EP world size 8), `--enable-expert-parallel`, `--all2all-backend nixl_ep`, `--enable-eplb` with 64 redundant experts (async), `--enable-fault-tolerance`, `--cpu-distributed-timeout-seconds 30`, `engine_recovery_timeout_sec=500`. - Drive baseline inference on all ranks; kill one worker of DP rank 1 (TP1 sibling) during active inference; verify survivors detect the fault and go UNHEALTHY while the victim's engine goes DEAD; send `scale_down(removed_dp_ranks=[1])` to the 3 survivors; verify expert redistribution + weight reload, Gloo group rebuilt over the surviving slots (original rank coordinates unchanged), all survivors HEALTHY and serving. ## Test Results Manual scale-down validation (2026-08-12, config above): **PASSED**. This run exercises the TP=2 single-worker-kill scenario end to end. Sanitized full log (host IP / GPU model / local paths redacted) attached. | Phase | Observation | Timestamp | |---|---|---| | Cluster ready | 4 ranks (DP=4 × TP=2, EP world 8) initialized, serving baseline traffic | 17:49:24 – 17:50:48 | | Fault injected | One worker of DP rank 1 (TP1 sibling, EP rank 3) killed during active inference | 17:50:48 | | First survivor-side error | Survivors' TP1 workers hit gloo `Connection reset by peer` on the per-TP-rank DP group | 17:50:48 | | Victim DEAD | Engine 1's executor shut down its remaining worker; busy loop cancelled → status DEAD; its API server stays up and keeps answering `/fault_tolerance/status` | 17:50:57 | | Survivors UNHEALTHY | Engines 0/2/3 raise from the nixl_ep kernel timeout with mask `[0,0,1,1,0,0,0,0]` (EP ranks 2,3 = DP rank 1, both TP workers) | 17:51:18 | | scale_down dispatched | `POST /fault_tolerance/apply` `removed_dp_ranks=[1]` → 202 on R0/R2/R3 | ~17:51:44 | | Expert redistribution | 257 (layer, expert) pairs reloaded from disk per worker (652–733 tensors); `num_dispatchers` synced to `active_rank_bound=8` | 17:51:45 | | Scale-down complete | Gloo DP group rebuilt 4→3 (dense group ranks R0→0, R2→1, R3→2; original `data_parallel_rank` frozen); Engines 0/2/3 → HEALTHY, `cumulative dead_dp_ranks=[1]` | 17:51:45 | | Post-recovery inference | All 3 survivors resume serving `/v1/chat/completions` with HTTP 200 | 17:51:45 – 17:52:07 | Key observations: - Recovery itself (mask replay + redistribution + weight reload + group reinit) completes in ~1s once dispatched; end-to-end latency is dominated by fault detection (immediate for the TP1 siblings via gloo connection-reset; ~30s for the surviving engines via the nixl_ep kernel timeout). - No hangs, no orphaned TP collectives — the TP sibling lockstep barrier keeps survivor device streams clean. E2E pytest: ``` .venv/bin/python -m pytest tests/v1/fault_tolerance/test_fault_tolerance_e2e.py -v -s ``` - **AI assistance**: this change was developed with AI assistance (kimi-k3); every changed line was reviewed by the human submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update.
", "url": "https://github.com/vllm-project/vllm/pull/46370", "createdAt": "2026-06-22T12:56:18Z", "updatedAt": "2026-08-13T08:06:06Z", "timestamp": "2026-08-13T08:06:06Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "fangyuchu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46424", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Mamba2] Fix assert crash when prefill-reclassified-as-decode occurs with no concurrent spec tokens", "text": "## Summary When a Mamba2-attention hybrid model (e.g. NemotronH) is served with both `--enable-prefix-caching` and `--speculative-config`, the server can crash with: ``` AssertionError File \"vllm/model_executor/layers/mamba/mamba_mixer2.py\", line 963, in conv_ssm_forward assert block_idx_last_scheduled_token_prev_step_d is not None ``` ### Trigger condition 1. `--enable-prefix-caching` auto-sets `mamba_cache_mode=all` for models implementing `SupportsMambaPrefixCaching` (`config.py:352`). This makes `is_mamba_cache_all=True` in the model. 2. A request arrives whose prompt is exactly `N × block_size + 1` tokens and the N-block prefix is already cached. Only 1 token remains to prefill. `mamba_attn.py` reclassifies any single-query-token prefill with prior state (`seq_len > 1`) as a decode step. 3. No other concurrent request has active speculative tokens that scheduler step, so `use_spec_decode = len(scheduled_spec_decode_tokens) > 0` is **False**. The current code gates `prev_last_scheduled_idx` inside this check, so it is never forwarded to `Mamba2AttentionMetadataBuilder`. 4. `mamba_mixer2.py:963` asserts this value is non-None whenever `has_decode=True`, `is_mamba_cache_all=True`, and `num_spec > 0`. The assert fires and kills all TP ranks. The crash is intermittent under heavy traffic: if any concurrent decode request happens to have active spec tokens in the same step, `use_spec_decode=True` and the bug is silently avoided. ### Root cause PR #41233 added `mamba_prev_last_scheduled_idx` and per-step preprocessing but left the per-step `use_spec_decode` gate in place. The fix is incomplete: `prev_last_scheduled_idx` must be forwarded whenever the buffer exists, not only on steps where spec tokens are scheduled. In v0.21.0 the crash was impossible because `config.py` forced `mamba_cache_mode=align` (not `all`) when spec decode was also active, so `is_mamba_cache_all` was always False. ### Fix Move the `prev_last_scheduled_idx` assignment outside the `use_spec_decode` gate. The guard `self.mamba_prev_last_scheduled_idx is not None` ensures this only runs for spec-decode deployments with `mamba_cache_mode=all`. `Mamba2AttentionMetadataBuilder` already handles `prev_last_scheduled_idx < 0` via the fallback `(num_computed - 1) // mamba_block_size`, so passing unconditionally is safe. ## Duplicate check Searched open PRs for `mamba_mixer2 assert`, `mamba prev_last_scheduled_idx`, `mamba cache mode spec decode` — no existing PR found. Related: #41233 (introduced the incomplete fix this PR completes). ## Test plan Requires a Mamba2-attention hybrid model implementing `SupportsMambaPrefixCaching` (NemotronH family) served with both `--enable-prefix-caching` and `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":N}'`: 1. Start vLLM. Find the effective `BLOCK_SIZE` in the startup log: `\"Setting attention block size to N tokens\"`. 2. Send a prompt of exactly `BLOCK_SIZE + 1` tokens. The server caches one full block. 3. Send the same prompt again while the server is otherwise idle. Without this fix: the assert fires immediately and all TP ranks crash. With this fix: the request succeeds. Verified on `vllm/vllm-openai:v0.22.0`, 4-way tensor parallel, NemotronH-class model. ## AI assistance disclosure This fix was developed with AI assistance (Claude Sonnet 4.6). The crash sequence, root cause analysis, and fix were verified by the human submitter against the live codebase and against a running deployment. Every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/46424", "createdAt": "2026-06-22T23:20:12Z", "updatedAt": "2026-08-13T17:47:48Z", "timestamp": "2026-08-13T17:47:48Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready", "v1" ], "author": "sytianhe", "state": "open", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:46595", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MooncakeStore] track resumed requests via scheduler's resumed_req_ids", "text": "## Summary The MooncakeStore connector tracks block tables per request so it knows whether a step's `new_block_ids` should be **appended** (normal decode) or **replace** the existing table (resumed from preemption). It was deciding this from a connector-local `_preempted_req_ids` set, which drifts from what the scheduler actually does. The scheduler already tells us the answer: `CachedRequestData.resumed_req_ids` is exactly the set of requests whose `new_block_ids` is a full replacement this step (everything else is a delta append). This PR drops the local set and reads `resumed_req_ids` directly, so the connector's append-vs-replace choice always matches how the scheduler built the block IDs. ## Symptom On a heavily warm-cache run (hybrid KV cache, multiple groups), the store send-thread throws `list index out of range` / `Store chunk out of range` from `KVCacheStoreSendingThread`. The store path indexes `block_id = block_ids[start // block_size]` while walking the logical token/hash chunks implied by `token_len`, but `token_len` is the absolute prefix length (e.g. 786432) whereas `block_ids` only covers the suffix actually held (~8K tokens), so the per-group scan walks off the end of the block table. Root cause: a request that was preempted and re-admitted stayed in `_preempted_req_ids` (nothing cleared it on that path), so on its next decode step the connector took the \"resumed\" branch and replaced `allocated_block_ids` with just the delta while `token_len` still reflected the full computed length — exactly the `token_len ≫ len(block_ids)` mismatch above. ## Fix Worker thread errors now log via `logger.exception`, so the queue-drain loop reports the full traceback and the offending `req_id` instead of just the exception message. ## Test ``` .venv/bin/python -m pytest tests/v1/kv_connector/unit/test_mooncake_store_scheduler.py -q # 14 passed ``` Added two regression tests: a running (non-resumed) request appends its delta, and a resumed request replaces a stale tracker's table — both asserting `token_len` stays covered by the block table so the store-path index can't run off the end. Before/after on the same workload at comparable cache warmth: the buggy build produced ~900 send-thread errors across a run; with this fix, zero (clean exit), while still exercising the external-store load path. ## Notes - Not a duplicate: no open PR touches the Mooncake store connector's preemption/resume tracking (searched open PRs for \"mooncake preempt resumed\" and \"resumed_req_ids connector\"). - AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/46595", "createdAt": "2026-06-24T08:31:41Z", "updatedAt": "2026-08-12T21:11:35Z", "timestamp": "2026-08-12T21:11:35Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "bug", "ready", "v1", "kv-connector" ], "author": "ivanium", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46638", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Add FlashInfer CuteDSL non-causal decode path for DFlash", "text": "## Purpose Part of the DFlash bring-up tracker #46105 (\"Attention backend → CuteDSL via FlashInfer\", contribution-welcome item). vLLM currently runs non-causal DFlash queries through the FlashInfer *prefill* wrapper (`BatchPrefillWithPagedKVCacheWrapper` with `causal=False`), because the decode/TRTLLM paths cannot express non-causal attention. FlashInfer's CuteDSL paged decode wrapper (`BatchDecodePagedCuteDSLWrapper`) supports `q_len_per_req > 1` and `is_causal=False`, which maps directly onto the DFlash decode workload (uniform `1 + num_speculative_tokens` query tokens per request). This PR adds a `CuteDSLDecode` dispatch mode to the FlashInfer backend: - `VLLM_FLASHINFER_CUTEDSL_DECODE` env flag, default off. - Static gate: SM100 only, full attention only (no SWA / soft cap / sinks), non-quantized KV cache (CuteDSL requires `kv_data_type == q_data_type`), page size in {8, 16, 32, 64}, no CUDA graph (Phase 2). - Per-batch gate: non-causal batches with a uniform query length per request; anything else falls back to the existing non-causal prefill path. ### FlashInfer version requirement The CuteDSL non-causal decode kernel had a boundary-mask bug (softmax normalized over padded KV tiles when `seq_len % 256 != 0`), fixed in flashinfer-ai/flashinfer#3717. The fix first shipped in the `0.6.14.dev20260706` nightly and landed in the `0.6.15` stable release. This PR bumps `requirements/cuda.txt` (and the Dockerfile pin) to `0.6.15` accordingly; the runtime helper only feature-detects the wrapper, so the default-off flag still guards correctness against any older FlashInfer install. Verified on B200 (SM100) with `0.6.14.dev20260706`: non-causal output matches reference for all tested `seq_len` including non-multiples of 256. ## Test Plan ```bash # Unit: CuteDSL decode vs SDPA reference (SM100 + flashinfer nightly required), # includes fallback coverage for non-uniform query lengths pytest tests/v1/attention/test_attention_backends.py -k cutedsl -v # Existing non-causal prefill path unaffected pytest tests/v1/attention/test_attention_backends.py -k non_causal -v ``` E2E (Qwen3-8B + DFlash draft, acceptance-length parity) and kernel microbench vs the non-causal prefill baseline: see Test Result below. ## Test Result Environment: NVIDIA B200 (SM100), torch 2.11.0, flashinfer 0.6.14.dev20260706, CUDA 13.0. **Unit (2026-07-07, run locally with an ungated same-architecture model, Qwen/Qwen3-0.6B, since this env has no HF token; the committed test uses Meta-Llama-3-8B as the rest of the file):** - `test_non_causal_cutedsl_decode_correctness`: 4/4 passed - `small_decode`, `small_prefill`, `medium_prefill`: routed through CuteDSL (wrapper-getter spy called), output matches SDPA reference (atol/rtol 1e-2) - `mixed_small` (non-uniform query lens): fell back to non-causal prefill, correct output, CuteDSL not invoked - seq_lens include non-multiples of 256, so the flashinfer-ai/flashinfer#3717 boundary-mask fix is exercised - Causal regression (FLASHINFER backend, flag on/off × small_decode/mixed_small): 4/4 passed - Kernel-level repro of the flashinfer bug against 0.6.14.dev20260706: ratio == 1.000 for all seq_lens (16–512) with `is_causal=False` — upstream fix confirmed **Kernel microbench (B200, DFlash shape: q_len_per_req=17, GQA 32/8, head_dim 128, page 16, bf16, HND layout, CUDA events, median of 50):** CuteDSL paged decode (`reduction=\"auto\"`) vs `BatchPrefillWithPagedKVCacheWrapper(causal=False)`: | batch | kv_len | prefill (ms) | cutedsl (ms) | speedup | |---|---|---|---|---| | 1 | 512 | 0.053 | 0.048 | 1.10x | | 1 | 2048 | 0.083 | 0.111 | 0.75x | | 1 | 8192 | 0.128 | 0.178 | 0.72x | | 4 | 2048 | 0.104 | 0.085 | 1.23x | | 8 | 2048 | 0.120 | 0.095 | 1.26x | | 16 | 2048 | 0.157 | 0.112 | 1.41x | | 16 | 8192 | 0.313 | 0.231 | 1.36x | | 32 | 8192 | 0.496 | 0.385 | 1.29x | - `reduction=\"kernel\"` (deterministic) is consistently *slower* than the prefill baseline (0.53–0.84x) — hence the integration uses `\"auto\"` (atomic; out buffer zero-initialized in `forward()`) and falls back to `\"kernel\"` only under `VLLM_BATCH_INVARIANT`. - Known weak spot: batch 1 × long context (0.72–0.75x). Everything else 1.1–1.4x. **E2E (Qwen/Qwen3-8B + z-lab/Qwen3-8B-DFlash-b16, num_spec_tokens=16, enforce_eager, 12 prompts × 512 tok, temp 0, 3 runs per config):** - CuteDSL path confirmed engaged when flag on (wrapper-getter called 362×; 0× when off). - Acceptance length: 3.7117 (off) vs 3.7214 (on) — reproducible per config across runs; drafts identical (1065). - 8/12 outputs bitwise identical; 4 diverge mid-generation into equally-plausible continuations (expected kernel-level numeric differences under temp 0). - Throughput (tok/s): off 651/644/730, on 612/709/741 — parity within machine noise (±7%) at this small scale; the kernel microbench above is the controlled performance signal, not this run. `vllm bench serve` (sonnet, 1024 in / 512 out): conc=1 median TPOT 7.92 ms (off) vs 9.79 ms (on); conc=8 742 vs 713 tok/s. Acceptance lengths differed between runs (2.75/2.53 vs 2.57/2.59) due to random sonnet sampling, so these numbers are noisy and not used as the performance evidence for this PR — included for completeness only. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/46638", "createdAt": "2026-06-24T16:29:05Z", "updatedAt": "2026-08-13T12:55:37Z", "timestamp": "2026-08-13T12:55:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ci/build", "v1", "nvidia" ], "author": "elwhyjay", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46676", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KERNEL][ROCm]Native HIP MXFP4(Compressed+Quark) (dense + MoE) for RDNA3", "text": "## What this is RDNA3 (the RX 7900 XTX and friends) currently has no way to run MXFP4 models in vLLM. Marlin is CUDA-only and AITER's MXFP4 path is CDNA4/gfx950-only, so when you point vLLM at an MXFP4 checkpoint on a 7900 XTX it either refuses to load (compressed-tensors models, which have no ROCm kernel to fall back on) or limps along on the Triton-unfused emulation (GPT-OSS). This PR adds native HIP MXFP4 kernels for gfx1100 so these models just work. It covers the three spots an MXFP4 model actually hits the GPU: - **the dense weight-only GEMM** (`mxfp4_gemm_rdna3`) behind every MXFP4 linear, - **the fused MoE GEMM** (`moe_mxfp4_gemm_rdna3`) — routing + dequant + dot in one launch, with the down-projection reduction folding in `moe_sum`, - **the plumbing** that points the MXFP4 MoE consumers at those kernels on gfx1100. It works across the three ways an MXFP4 checkpoint reaches vLLM today — **compressed-tensors**, **GPT-OSS native-mxfp4**, and **AMD Quark** (OCP-MX) — because the MoE path is registered as a backend in the existing MXFP4 MoE oracle (`fused_moe/oracle/mxfp4.py`) rather than as a bespoke quant method. All three quant methods already funnel through that oracle, so one backend (`RDNA3Mxfp4Experts`, a standard `FusedMoEExpertsModular`) serves all three; the dense linears reuse the existing `MxFp4LinearKernel` registry. MXFP4 is friendly to this: E2M1 unpacks into bf16/fp16 with what's basically a field copy, and the E8M0 block scale (group of 32, no zero point) folds in as an integer exponent add rather than a multiply. Everything here is gated on gfx1100; no other target changes. There's no FP4 tensor core on RDNA3, so the win isn't raw matmul throughput — it's the 4× smaller weights, the bandwidth-bound decode regime where reading a quarter of the bytes pays off, and frankly just *being able to run the model at all* on this card. ### Quark / W4A4 on a platform with no native FP4 AMD Quark MXFP4 checkpoints are often `w_mxfp4_a_mxfp4` (W4A4 — weights *and* activations FP4). gfx1100 has no native FP4 compute, so these degrade to weight-only (bf16 activations), exactly like the existing ROCm Triton-unfused fallback already does. This PR just makes that fallback prefer the native gfx1100 kernel: a W4A4 Quark MoE decodes its weights on `moe_mxfp4_gemm_rdna3` with bf16 activations instead of the Triton emulation. Quark's weight-only **dense** MXFP4 linears (`QuarkOCP_MX`) are likewise routed to `Rdna3MxFp4LinearKernel` per-layer (when `N%16==0 && K%32==0`, else they keep the emulation path). ### Tensor-parallel correctness The fused MoE kernel can fuse the top-k reduction into its `output_topk` epilogue, which is correct on a single GPU but wrong under tensor parallelism: each TP rank holds only an intermediate-dim shard, so its down-projection is a partial that the layer all-reduces afterwards, and writing the reduced result in-kernel ahead of that all-reduce corrupts the output. The experts backend now detects `tp_world_size > 1` and writes unreduced rows + reduces in Python (the same path already taken when a per-expert bias is present); TP1 keeps the fused fast path. Verified: the same model garbled at TP2 before this and is correct after. ## How fast it is All runs are on an RX 7900 XTX (gfx1100), ROCm 7.2.3, bf16, cudagraph on. Numbers are `vllm bench serve`, random 512-in / 256-out, `--ignore-eos` — output token throughput in tok/s. **GPT-OSS-20B (single GPU), native HIP MoE vs the Triton-unfused path it replaces.** This is the apples-to-apples one, since GPT-OSS is the only model here that *has* a ROCm fallback to compare against: | concurrency | Triton | HIP | speedup | |---|---|---|---| | 1 | 8.5 | 48.8 | 5.7× | | 8 | 82.1 | 206.8 | 2.52× | | 16 | 147.0 | 241.8 | 1.64× | | 32 | 230.0 | 382.6 | 1.66× | Single-stream latency (`vllm bench latency`, 128/256, batch 1) drops from 13.15 s to 2.43 s — about 5.4×. **`amd/Qwen3.5-35B-A3B-MXFP4` (AMD Quark, W4A4, 256-expert MoE, TP2)** — the second apples-to-apples case: this W4A4 checkpoint runs weight-only on gfx1100, native HIP vs the Triton-unfused fallback it otherwise gets. tok/s (mean TPOT ms): | concurrency | TRITON_UNFUSED | native RDNA3 | speedup | |---|---|---|---| | 1 | 26.6 (36.8) | 58.9 (14.9) | 2.21× | | 8 | 177.6 (42.1) | 255.6 (27.5) | 1.44× | | 16 | 279.9 (53.1) | 396.3 (36.4) | 1.42× | | 32 | 407.7 (72.7) | 559.0 (51.4) | 1.37× | **`pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4` (TP2)** — a real compressed-tensors MoE checkpoint (quantized from `Qwen/Qwen3.6-35B-A3B`). There's no baseline to compare against here because the model simply won't load on gfx1100 without these kernels: | concurrency | tok/s | mean TPOT (ms) | |---|---|---| | 1 | 63.3 | 15.3 | | 8 | 303.9 | 23.9 | | 16 | 434.7 | 33.0 | | 32 | 613.1 | 45.5 | **`kaitchup/Qwen3.5-27B-MXFP4A16` (TP2)** — a real third-party *dense* MXFP4 checkpoint, again with no ROCm baseline: | concurrency | tok/s | mean TPOT (ms) | |---|---|---| | 1 | 32.8 | 28.8 | | 8 | 127.4 | 55.0 | | 16 | 192.5 | 69.8 | | 32 | 251.0 | 104.4 |", "url": "https://github.com/vllm-project/vllm/pull/46676", "createdAt": "2026-06-25T01:41:16Z", "updatedAt": "2026-08-13T12:41:04Z", "timestamp": "2026-08-13T12:41:04Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "rocm", "ci/build", "gpt-oss", "quantization" ], "author": "JartX", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46690", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix UVA offload fallback copies", "text": "## Purpose Fixes the non-UVA fallback path reported in #37883. The issue was closed automatically, but received new reports on August 10 and 11. When UVA is unavailable or disabled, `UVAOffloader` copies offloaded module state to the accelerator for each forward call. The previous wrapper had several problems: - `state_dict()` inside `forward()` is not traceable by Dynamo. - Temporarily replacing `module.forward` lets exceptions and concurrent calls lose or bypass the wrapper. - State captured before post-load processing can become stale when parameters, buffers, or child modules are replaced. - Calling `functional_call` on the wrapped module dispatches module hooks a second time. - CPU-to-device copies used `non_blocking=True` even when CPU memory was not pinned. The wrapper now stays installed during every call. A private proxy invokes the saved original forward while `functional_call` supplies the current parameters and persistent buffers. Eager calls use a per-module reentrant lock because functional reparameterization is temporary. Dynamo capture bypasses the lock and traces the functional call. The copy mode now follows the actual pin-memory setting. ## Duplicate check This does not duplicate the other open offloader changes: - #43453 keeps externally cached quantization scales on the accelerator. It does not change fallback invocation or state resolution. - #48468 wires the offloader into the V2 model runner. It does not change `UVAOffloader`. - #41496 enables pinned memory on WSL2 so that WSL can use UVA. This PR fixes the fallback used when UVA remains unavailable or disabled. - Closed #29054 addressed the `state_dict()` trace failure, but not exception recovery, replaced state, hook behavior, or concurrent eager calls. ## Validation Human review and test: ```text .venv/bin/python -m pytest -q tests/model_executor/test_uva_offloader.py 10 passed ``` Repository checks: ```text pre-commit run --files tests/model_executor/test_uva_offloader.py vllm/model_executor/offloader/uva.py all applicable hooks passed ``` RTX 5090 checks also passed for in-place fullgraph compilation, recursive calls, CUDA Graph capture and replay, non-default stream copies, and 200 compiled calls across eight threads. Parameters remained on CPU between calls. A model evaluation was not run because this patch does not change model math or output selection. The focused suite compares outputs across eager and compiled calls and covers post-load state replacement. ## AI assistance OpenAI Codex assisted with the implementation and test preparation. The human submitter reviewed every changed line and ran the focused test suite before the branch update.", "url": "https://github.com/vllm-project/vllm/pull/46690", "createdAt": "2026-06-25T04:32:40Z", "updatedAt": "2026-08-13T06:56:50Z", "timestamp": "2026-08-13T06:56:50Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug" ], "author": "wgu9", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46847", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Bugfix] window-correct shuffled fp8 decode for SWA layers", "text": "## Purpose `AiterFlashAttentionImpl.forward` routes sliding-window (and sinks / multi-token spec-decode) decode through a branch that calls `flash_attn_with_kvcache` or `unified_attention`, both of which read the plain (non-shuffled) KV layout. That branch opens with `assert not rocm_aiter_ops.is_shuffle_kv_cache_enabled()`, so a sliding-window model with the shuffled fp8 cache enabled hard-fails at engine warmup: ``` AssertionError: Shuffle KV cache layout is not supported with sliding window, sinks, or speculative decoding (multi-token decode). ``` So you currently cannot run a sliding-window model with the shuffled fp8 KV layout. The non-windowed shuffled path (`paged_attention_common`) is fine; only the windowed case is blocked, because there was no kernel that reads the shuffled layout and applies a window. This replaces that blanket assert: when the shuffled layout is enabled and the case is sliding-window decode (one query token, no sinks), it reshapes the cache to the shuffled view and calls `paged_attention_decode_shuffle_swa`, which reads the shuffled fp8 layout and masks the window. Sinks and multi-token spec decode over the shuffled layout are still unsupported, so the assert is kept for those. ## Test Plan Built vLLM from source at current `main` on MI300X (gfx942), applied this patch and the companion aiter kernel (`paged_attention_decode_shuffle_swa`). Mistral-family SWA model (head 128, `sliding_window=4096`), `--kv-cache-dtype fp8`, `VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1`, `--attention-backend ROCM_AITER_FA`, `--max-model-len 8192` (above the window so SWA is actually active). Prompt longer than the window so decode crosses it. Compared against the bf16 default SWA path; decode dispatch confirmed with a per-call sentinel. ## Test Result gfx942, torch 2.10 / ROCm 7.2.1, GPU clock-locked: - Before this patch the build aborts at engine warmup with the assertion above. After: the SWA shuffled decode kernel fires on every decode step (at a 6722-token prompt the dispatch logged 2592 calls = 81 decode steps x 32 layers). A 7222-token-prompt run produces coherent, on-topic output. - fp8 shuffled vs bf16 default, batch 32: fp8 gives 2x KV-cache capacity (2,413,877 vs 1,206,938 tokens; max concurrency 294x vs 147x) at the same ~163.6 GiB reservation, with decode throughput at parity within run-to-run spread (bf16 1037-1293, fp8 1166-1226 tok/s). The point of the PR is that the windowed shuffled path now works at all instead of asserting; the deterministic win is the 2x capacity. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/46847", "createdAt": "2026-06-26T17:11:04Z", "updatedAt": "2026-08-13T16:01:37Z", "timestamp": "2026-08-13T16:01:37Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "bug", "rocm", "v1" ], "author": "reger-men", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:46994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec][V2] Support MTP speculative decoding under pipeline parallelism", "text": "## Purpose MTP speculative decoding does not currently work under pipeline parallelism on the V2 model runner. This PR makes it functional for DeepSeek-family MTP drafts (DeepSeek-V3, GLM-5.2, Qwen3.5/3.6, …). Five independent issues, all on the PP>1 path. Fixes #1-#3 are DeepSeek-family-specific; fix #4 applies to all models using sparse MLA attention; fix #5 applies to Qwen3.5/3.6 MTP draft models on the last PP rank. **1. `DeepSeekMTP` does not implement `SupportsPP`** — DeepSeek-family specific. The engine refuses to build the draft model under PP at all: ``` NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface. [DeepSeekMTPModel] ``` The MTP draft runs only on the last PP stage, so it never actually consumes PP intermediate tensors, but the interface still requires the `make_empty_intermediate_tensors` factory. (This mirrors what #39704 does for the V1 runner.) **2. `PPHandler` sampled-token broadcast width mismatch (hang)** — affects all MTP under PP. `broadcast()` sends `sampled_token_ids` at its natural width — 1 on any step with no draft tokens (prefill, first decode), `num_spec+1` once rejection sampling has run — while `receive()` always posts a fixed `[num_reqs, max_sample_len]` buffer. NCCL `broadcast` doesn't negotiate element counts, so a width-1 send against a width-`max_sample_len` receive is a count mismatch that deadlocks the receiver. Fix: pad the source to `max_sample_len` (trailing `-1`, ignored by `post_update`). **3. Proposed draft tokens are never relayed to non-last PP ranks (garbage output / ~0 acceptance)** — affects all MTP under PP. `req_states.draft_tokens` is written only on the last rank (the `propose()` path); non-last ranks keep the zero-init buffer. `combine_sampled_and_draft_tokens` then embeds zeros at the draft positions on rank 0, so the verification input is wrong. Fix: coalesce a third broadcast (the proposed draft tokens) into the existing deferred `PPHandler` sibling-group broadcast, and scatter it into `req_states.draft_tokens` on consume. No new collective; gated identically to the sampled-token broadcast so per-step op counts stay matched. **4. Stale `topk_indices_buffer` reference in sparse MLA backends (the acceptance fix)** — affects all models using sparse MLA attention. Under MTP+PP, `FlashAttnMLASparseImpl.__init__` stored `indexer.topk_indices_buffer` at construction time. When `_maybe_share_lm_head` later replaced `Indexer.topk_indices_buffer` with the target model's buffer, the impl's reference was stale — still pointing to the draft model's original (uninitialized) buffer. This caused garbage DSA attention → degenerate \"repeat-the-current-token\" drafts → ~27-33% acceptance instead of ~85%. Fix: store `self._indexer = indexer` in `__init__`, read `self._indexer.topk_indices_buffer` dynamically in `forward_mqa`. Applied to all three sparse MLA backends: `flashattn_mla_sparse.py`, `flashmla_sparse.py`, `flashinfer_mla_sparse.py`. **5. Apply `fc` projection on last PP rank for Qwen3.5 MTP** — Qwen3.5/3.6-specific. Under MTP+PP, the Qwen3.5 MTP draft model on the last PP rank was using the target model's hidden_states directly, bypassing the `fc` projection entirely. This produced essentially random predictions (~1% acceptance) because the draft model's input was not properly projected. Fix: on the last PP rank, apply the same `fc` projection as the first rank (embed `input_ids`, normalize, concat with `hidden_states`, project through `fc`). This is the same pattern used on PP0 (first rank). ## Test Plan Serve a DeepSeek-family or Qwen3.5/3.6 MTP model under PP on the V2 runner and check it boots, produces correct output, and accepts drafts at a normal rate: ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve zai-org/GLM-5.2-FP8 \\ --tensor-parallel-size 4 --pipeline-parallel-size 2 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' ``` Unit tests (`tests/v1/worker/test_pp_utils.py`): - `test_deepseek_mtp_implements_supports_pp` — verifies Fix `#1` - `test_pphandler_broadcast_pads_to_max_sample_len` — verifies Fix `#2` - `test_sparse_mla_backend_reads_topk_indices_buffer_dynamically` — verifies Fix `#4` ## Test Result Validated on GLM-5.2-FP8 (GlmMoeDsaForCausalLM, DeepSeek-Sparse-Attention MoE), TP4/PP2, on a current-main base (0.23.1rc1.dev531), serving real traffic for 5+ hours at K=3: | Metric | Value | |---|---| | Drafts | 91,088 | | Draft tokens | 273,264 | | Accepted tokens | 230,722 | | **Overall acceptance** | **84.4%** | | pos0 | 92.1% | | pos1 | 84.0% | | pos2 | 77.2% | The stale-buffer fix (fix #4) lifts acceptance from ~27-33% (pre-fix, fixes #1-#3 only) to **84.4%** at K=3 over 5+ hours of real traffic — matching the non-PP TP8 baseline (~85%). ### Cross-model validation: fix #4 applies broadly The stale-buffer fix was further validated across 8 Qwen3.5/3.6 model variants (dense BF16, AWQ, MoE BF16, MoE AWQ, MoE GPTQ-Int4), each at MTP K=1/2/3, PP=2/TP=1. All show high acceptance (83-96%) that scales gracefully with K, confirming the fix is not architecture-specific. | Model | Quant | K=1 | K=2 | K=3 | |---|---|---|---|---| | Qwen3.5-27B-AWQ | AWQ 4-bit | 95.3% | 90.5% | 85.8% | | Qwen3.5-27B (BF16) | BF16 | 95.5% | 91.1% 86.0% | | Qwen3.5-35B-A3B (MoE, BF16) | BF16 | 93.6% | 88.0% | 82.0% | | Qwen3.6-27B (BF16) | BF16 | 95.2% | 91.1% | 86.0% | | Qwen3.6-27B-AWQ | AWQ 4-bit | 95.3% | 90.5% | 85.8% | | Qwen3.6-35B-A3B (BF16) | BF16 | 94.1% | 88.6% | 83.8% | | Qwen3.6-35B-A3B-AWQ | AWQ 4-bit | 94.0% | 88.8% | 83.0% | | Qwen3.6-35B-A3B-GPTQ-Int4 | GPTQ 4-bit | 94.1% | 88.9% | 83.5% | Full per-position breakdown and GSM8K accuracy available on request. ### Remaining stale-buffer backends Fix #4 was also applied to the three remaining sparse MLA backends that were not covered by the original fix commit (`c175667db`): - `flashinfer_mla_sparse_sm120.py` — `FlashInferMLASparseSM120Impl` (SM120 variant) - `rocm_aiter_mla_sparse.py` — `ROCmAiterMLASparseImpl` (ROCm) - `xpu_mla_sparse.py` — `XPUMLASparseImpl` (Intel XPU) These backends have the same stale `topk_indices_buffer` bug but are not reachable on our hardware (H200/SM90). The fix follows the exact same pattern as the already-validated fix: store `self._indexer = indexer` in `__init__`, read `self._indexer.topk_indices_buffer` dynamically in `forward_mqa`.", "url": "https://github.com/vllm-project/vllm/pull/46994", "createdAt": "2026-06-29T05:01:54Z", "updatedAt": "2026-08-13T13:27:46Z", "timestamp": "2026-08-13T13:27:46Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "rocm", "intel-gpu", "needs-rebase", "v1", "qwen", "deepseek", "nvidia", "mrv2" ], "author": "eastwood-c", "state": "open", "assignees": [ "njhill" ] }, { "id": "github:vllm-project/vllm:pull_request:47335", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Enable fused indexer-Q RoPE+quant kernel for DeepSeek/GLM sparse attention", "text": "## Summary Enables the existing fused indexer-Q kernel (`fused_indexer_q_rope_quant`) on ROCm for the DeepSeek Sparse Attention (DSA) indexer used by GLM-5.2 / DeepSeek-V3.2-style models. On CUDA this kernel already collapses the indexer's query-side RoPE + FP8 quantization + weight-scale fold into a single launch; on ROCm the same work was falling back to a chain of small Triton/elementwise kernels per sparse layer per decode token. This is a low-risk, 2-line enablement — the kernel is already on vllm and is architecture-aware. It **removes redundant kernels and HBM round-trips** on the ROCm decode path (4 Q-side kernels → 1), yielding a small but consistent decode improvement with **no CUDA-side change** and no measured regression. ## Kernel-level evidence (torch.profiler, GLM-5.2-FP8, MI325X / gfx942, TP8, rank 0) Before/after self-CUDA over the same steady-state decode capture. The indexer decode kernels are the ones with ~21.5k calls (per decoded token × sparse layers). **Fused away** (present in baseline, gone after): | Kernel | Role | Self CUDA | Calls | |---|---|---|---| | `per_token_group_quant_8bit_kernel` | q → FP8 (ue8m0) quant | 99.8 ms | 21546 | | `triton_poi_fused_3` | RoPE/cat elementwise | 95.8 ms | 21526 | | `triton_poi_fused_2` | RoPE/cat elementwise | 89.0 ms | 20521 | | `triton_poi_fused_mul_slice_unsqueeze_view_4` | weights scale-fold | 85.3 ms | 20500 | **Introduced**: | Kernel | Role | Self CUDA | Calls | |---|---|---|---| | `_fused_indexer_q_rope_quant_kernel` | fused q RoPE + FP8 quant + weight-fold (decode) | 103.7 ms | 21546 | | `_fused_indexer_q_rope_quant_kernel_0` | same, prefill variant | 6.3 ms | 42 | | `triton_poi_fused_add_copy_index_select_mul_slice_…` | residual K-side RoPE (now its own kernel) | 90.6 ms | 20500 | The Q-side collapses from **4 dedicated kernels → 1** (the two `triton_poi_fused_2/_3` RoPE kernels did q *and* k together; after the change q's RoPE is absorbed and only a single K-RoPE kernel remains). Net ≈ **170 ms** self-CUDA reclaimed on rank 0 over the capture (≈0.7% of the ~24.9 s total). Untouched: `k_norm`, `indexer_k_quant_and_cache`, `_gluon_deepgemm_fp8_paged_mqa_logits`, `topKPerRowDecode`. ## Accuracy GLM-5.2-FP8, TP=8 on gfx942 |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9469|± |0.0062| | | |strict-match | 5|exact_match|↑ |0.9469|± |0.0062| ## End-to-end performance (GLM-5.2-FP8, TP8, MI325X / gfx942) `vllm bench serve`, random dataset, `--ignore-eos`, `--random-range-ratio 0.0`. Decode runs under CUDA graphs. | ISL | OSL | conc | TPOT_p50 base | TPOT_p50 fused | TPOT | ITL_p50 base | ITL_p50 fused | ITL | tok/s base | tok/s fused | tput | |---|---|---|---|---|---|---|---|---|---|---|---| | 1024 | 1024 | 1 | 14.98 | 14.93 | 1.004x | 14.98 | 14.94 | 1.003x | 66.32 | 66.34 | 1.000x | | 1024 | 1024 | 8 | 18.39 | 18.32 | 1.004x | 18.40 | 18.30 | 1.005x | 422.81 | 423.68 | 1.002x | | 1024 | 1024 | 32 | 24.83 | 24.71 | 1.005x | 24.31 | 24.19 | 1.005x | 1220.94 | 1222.21 | 1.001x | | 1024 | 1024 | 64 | 32.39 | 32.30 | 1.003x | 31.25 | 31.00 | 1.008x | 1835.24 | 1842.09 | 1.004x | | 8192 | 1024 | 1 | 16.01 | 15.90 | 1.006x | 15.98 | 15.88 | 1.006x | 60.18 | 60.55 | 1.006x | | 8192 | 1024 | 8 | 22.26 | 22.06 | 1.009x | 19.86 | 19.79 | 1.004x | 323.27 | 324.17 | 1.003x | | 8192 | 1024 | 32 | 38.86 | 39.73 | 0.978x* | 25.14 | 24.98 | 1.006x | 715.20 | 718.98 | 1.005x | | 8192 | 1024 | 64 | 63.69 | 63.66 | 1.001x | 29.72 | 29.80 | 0.997x | 925.56 | 925.84 | 1.000x | Consistent small improvement across the sweep (~0.3–0.9%), with no credible regression. *The single `8192/32` TPOT dip is run-to-run variance: ITL (1.006x) and throughput (1.005x) both improve at that same point. ## Related work / positioning - **#43907** (`[ROCm][Perf] DSv3.2: fuse indexer Q-RoPE+quant + K-norm/RoPE/quant/cache`) fuses Q **and** K via an **AITER** kernel, but **explicitly excludes GLM** (`model_type != \"glm_moe_dsa\"`) and depends on a companion aiter kernel behind an env flag. This PR covers `glm_moe_dsa` using the **in-tree Triton** kernel with no aiter dependency and no new flags — complementary, not overlapping. - **#44527** (`[ROCm][DSv3.2] Eliminate per-decode FillFunctor launches`) removes the `Fill` kernels in the same hot loop; stacks cleanly on top of this change. ## Risk / compatibility Low. CUDA behavior is bit-for-bit unchanged; the ROCm change is opt-in via the same static guard and only alters the previously-fragmented eager path. The end-to-end gain is intentionally modest (kernel/HBM-traffic reduction under CUDA graphs), but the change is trivial, carries no measured regression, and compounds with related sparse-indexer cleanups (#44527). Requires the fused branch to fire (`is_inplace_rope == False`), which is the default under `torch.compile`/Inductor.", "url": "https://github.com/vllm-project/vllm/pull/47335", "createdAt": "2026-07-01T20:26:21Z", "updatedAt": "2026-08-13T07:11:28Z", "timestamp": "2026-08-13T07:11:28Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "ready", "deepseek", "quantization" ], "author": "xaguilar-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47392", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MoE] Plumb swigluoai activation into FlashInfer b12x MoE", "text": "## Purpose Models with a clamped SwiGLU-OAI MoE activation (`swigluoai_uninterleave`, e.g. [nvidia/MiniMax-M3-NVFP4](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)) cannot use the FlashInfer b12x MoE backend: `FlashInferB12xExperts` handles only silu and relu2. This matters on SM120/SM121: FlashInfer TRT-LLM MoE requires SM100-family GPUs and FlashInfer CUTLASS lacks this activation, so such models have no FlashInfer MoE path there. FlashInfer supports `swigluoai_uninterleave` on this kernel since flashinfer-ai/flashinfer#3744; this PR adds it to `FlashInferB12xExperts` and passes `swiglu_alpha/beta/limit` to the `B12xMoEWrapper`, gated on the installed FlashInfer supporting them. Behavior with the current pin (0.6.13, which predates the kernel support) is unchanged; a later pin bump enables these models on b12x with no further vLLM changes. The unit test now drives the production weight-processing path instead of hand-preparing weights, covers silu and swigluoai, and uses a relative-error check that actually fails on a wrong gate/up ordering or activation (the old tolerances passed under any kernel behavior). This guards the whole b12x weight pipeline, not just the new activation. Not a duplicate of #47001, which widens b12x `_supports_activation` without plumbing the activation into the kernel call. ## Test Plan ``` pytest tests/kernels/moe/test_flashinfer_b12x_moe.py ``` Needs SM120 and FlashInfer at/past flashinfer-ai/flashinfer#3744 for the swigluoai cases; they skip on the current pin. ## Test Result RTX 5080, FlashInfer at the #3744 merge commit: 48 passed with the fix. Without the fix, all 24 swigluoai cases fail (relative error 0.60 vs the 0.45 bound) and the 24 silu cases pass. On pre-#3744 FlashInfer the swigluoai cases skip and silu passes. Developed with AI assistance (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/47392", "createdAt": "2026-07-02T05:02:35Z", "updatedAt": "2026-08-13T17:02:11Z", "timestamp": "2026-08-13T17:02:11Z", "metrics": { "reactions": 1, "comments": 5 }, "labels": [ "bug", "needs-rebase", "nvidia" ], "author": "yichengj0", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47562", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Drop incomplete tool-call markup in non-streaming to match streaming", "text": "## Purpose Fixes #47137 When generation terminates (`max_tokens`/`length` or a `stop` string) inside a `` opener that has not been promoted to a tool call, the streaming path drops the buffered markup but the non-streaming path returned it as `content` — the two paths disagreed for identical model output. Contrary to the hope in the issue thread that #46875 had fixed this, I re-ran the issue's no-server unit reproduction on current main (`978de83`) and **the divergence still reproduces** (confirming @TheDuyIT's July 1 retest): ``` 2 '\\n' stream= '' nonstream= '\\n' DIVERGE 3 '\\n<' stream= '' nonstream= '\\n<' DIVERGE 4 '\\n ruff check / ruff format / typos / mypy: Passed ``` Also re-ran the issue's part-D reproduction with the fix: all three truncation points now MATCH between streaming and non-streaming. ## Test Result All new and existing parser tests pass on macOS CPU (source build, `VLLM_TARGET_DEVICE=cpu`). --- This PR was authored with AI assistance (Claude Code); the diff was reviewed and the tests were run locally as described above. cc @bbrowning @TheDuyIT", "url": "https://github.com/vllm-project/vllm/pull/47562", "createdAt": "2026-07-03T15:21:25Z", "updatedAt": "2026-08-13T15:00:01Z", "timestamp": "2026-08-13T15:00:01Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "bug", "tool-calling" ], "author": "JaynouOliver", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47596", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: aggressively zero speculator buffers to prevent CUDA illegal memory access on H100", "text": "Fixes #47561. Root Cause: When running vLLM speculative decoding on H100s with quantized models (e.g. w8a8_fp8), the system was crashing with a CUDA illegal memory access exclusively during the speculator's prefill CUDA graph capture. During vLLM's warmup phase prior to graph capture, a \"dummy run\" eagerly executes the speculator (propose()). This leaves the AutoRegressiveSpeculator's internal state buffers (hidden_states, draft_tokens, draft_logits, idx_mapping, current_draft_step, etc.) populated with values specific to the dummy run's final execution steps. Because the prefill/decode captures operated on this stale state, the dynamic shape calculations inside _prefill were thrown out of bounds. Furthermore, the TMA descriptors (used in Hopper FP8 kernels) are extremely sensitive to uninitialized/stale data (like NaN scale factors or unaligned shape inferences), causing hard segfaults when fed this stale state. Solution: This PR introduces a robust fix that deterministically zeroes out all of the AutoRegressiveSpeculator buffers inside capture() immediately before the CUDA graphs are recorded. By aggressively running .zero_() on last_token_indices, current_draft_step, hidden_states, inputs_embeds, idx_mapping, temperature, seeds, draft_tokens, and draft_logits, we guarantee that each CUDA graph capture step executes against a pristine, zero-initialized state. This definitively prevents the Hopper architecture from crashing due to stale bounds, NaNs, or out-of-bounds embedding lookups.", "url": "https://github.com/vllm-project/vllm/pull/47596", "createdAt": "2026-07-04T08:08:49Z", "updatedAt": "2026-08-13T09:50:13Z", "timestamp": "2026-08-13T09:50:13Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "v1", "nvidia", "mrv2" ], "author": "Saurav-Gupta-9741", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47664", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature][Whisper] Native word-level timestamps (cross-attention + DTW)", "text": "Whisper accepts `timestamp_granularities[]=word` today, but `words` always comes back `null`. This fills it in. ```bash vllm serve openai/whisper-large-v3-turbo --enable-word-timestamps curl localhost:8000/v1/audio/transcriptions \\ -F file=@mary_had_lamb.ogg -F model=openai/whisper-large-v3-turbo \\ -F response_format=verbose_json -F 'timestamp_granularities[]=word' ``` **Before** ```json { \"text\": \" The first words I spoke in the original phonograph...\", \"words\": null } ``` **After** ```json { \"text\": \" The first words I spoke in the original phonograph...\", \"words\": [ {\"word\": \"The\", \"start\": 0.52, \"end\": 0.88}, {\"word\": \"first\", \"start\": 0.88, \"end\": 1.60}, {\"word\": \"words\", \"start\": 1.60, \"end\": 2.02}, {\"word\": \"I\", \"start\": 2.02, \"end\": 2.26}, {\"word\": \"spoke\", \"start\": 2.26, \"end\": 2.66} ] } ``` Off by default; nothing runs without `--enable-word-timestamps`. ## How Whisper never emits word boundaries as tokens — they live in the decoder's cross-attention. Instead of asking a fused backend for its attention probabilities, this records **Q** (and the encoder **K** at prefill) just before the fused call, then recomputes `softmax(Q·Kᵀ)` for the `alignment_heads` only, median-filters it and runs a DTW. Same method as OpenAI Whisper and WhisperX. No second model, no second forward pass, no attention-backend change, and it runs with CUDA graphs and `torch.compile` on. The recording is a registered custom op writing into static buffers, so it stays opaque to Dynamo and safe under graph replay. Code lives in `vllm/v1/worker/gpu/word_align.py` and attaches to the V2 model runner by composition, like `gpu/kv_connector.py`. The runner side is four lifecycle calls. The V1 runner is untouched. Each request gets a capture slot from a bounded pool, and a request that finds the pool full gets no timestamps. Only the layers that actually carry alignment heads are allocated, and the pool is sized by a 2 GiB budget rather than a fixed count, so checkpoints with few alignment layers cover a full batch while the larger ones stay inside the budget: | model | decoder layers | layers with alignment heads | slots within budget | | --- | --- | --- | --- | | `whisper-large-v3-turbo` | 4 | 2 | 215 | | `whisper-large-v3` | 32 | 10 | 43 | | `whisper-small` | 12 | 4 | 179 | The pool is also capped at `max_num_reqs`, so it never allocates for more requests than the server can run. ## Benchmark H200, a `large-v3-turbo`-architecture checkpoint, 40 short clips (median ~2 s) replayed at concurrency 32. `verbose_json` is the baseline because word timestamps require it — it is ~25% slower than plain `json` on its own, with or without this feature. Each row is the mean of two passes after a discarded warm-up pass; runs repeat to within ±2%. | server | request | KV cache | throughput | | --- | --- | --- | --- | | flag off | `verbose_json` | 89,033 tok | 61.6 req/s | | flag on | `verbose_json` | 85,767 tok | 38.6 req/s | | flag on | `+ word` | 85,767 tok | 37.4 req/s | Single-request latency is unchanged within measurement error (~70 ms either way); the cost only shows up under concurrency. Two things to note. Asking for words is nearly free once the flag is on (38.6 → 37.4). Turning the flag on costs ~39% throughput even for requests that never ask for words, because the capture op is compiled into every decode step. Disabling the copies inside the op does not recover it, so the cost is the op's presence in the compiled graph rather than the work it does — making the opt-in per-request instead of per-server is the obvious follow-up. ## Accuracy - vs the OpenAI/HF `find_alignment` reference: mean |Δstart| **0.019 s** over 399 words. - vs an independent forced aligner over ~36k words: median |Δstart| **0.104 s**, identical word order. Close to a dedicated aligner, not a replacement for one. - The change is additive: a new response field, no effect on sampling or on the generated text. ## Tests ```bash pytest tests/models/multimodal/generation/test_whisper_word_align.py # 15 pytest tests/entrypoints/speech_to_text/transcription/test_transcription_word_timestamps.py # 4 ``` Both pass on the default runner with CUDA graphs and `torch.compile` on. The unit tests pin the median filter and the DTW **bit-identical** to transformers' `_median_filter` and `_dynamic_time_warping`, plus the capture op's slot routing and the slot pool. The e2e tests cover populated and monotonic `words[]` that reconstruct the transcript, the per-request opt-in, and concurrent clips each keeping their own timeline. Added to the `Model Runner V2 Core Tests` step so CI runs them under `VLLM_USE_V2_MODEL_RUNNER=1`. Also checked on `whisper-large-v3`, where alignment heads span 10 layers instead of 2: same word count, monotonic, in bounds. Pool exhaustion: 100 concurrent requests against 64 slots returned 0 errors, 98 with timestamps and 2 without, none malformed. ## Notes Requested in [#13400](https://github.com/vllm-project/vllm/issues/13400), [#25750](https://github.com/vllm-project/vllm/issues/25750), [#24302](https://github.com/vllm-project/vllm/issues/24302), [#19556](https://github.com/vllm-project/vllm/issues/19556). [#24209](https://github.com/vllm-project/vllm/pull/24209) added *segment* timestamps and left word-level as future work. Not a duplicate: #13400 was closed *not planned* and no open PR implements this. One open question: carrying the per-token times from the worker to the API needs a `word_align` field on `ModelRunnerOutput` → `EngineCoreOutput` → `CompletionOutput`, ~19 lines of plumbing. I followed how `routed_experts` does it — is there something lighter I missed? Developed with AI assistance (Claude); I reviewed every line and ran every test and benchmark above.", "url": "https://github.com/vllm-project/vllm/pull/47664", "createdAt": "2026-07-05T23:16:23Z", "updatedAt": "2026-08-13T00:52:07Z", "timestamp": "2026-08-13T00:52:07Z", "metrics": { "reactions": 9, "comments": 12 }, "labels": [ "documentation", "frontend", "needs-rebase", "ci/build", "v1", "multi-modality", "mrv2" ], "author": "yusufani", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47692", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix `--data-parallel-start-rank 0` being treated as unset in `create_engine_config`", "text": "## Summary `EngineArgs.create_engine_config` uses Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None` to detect whether `--data-parallel-start-rank` was explicitly set. Since `0` is a valid, meaningful starting rank (the node owning the first slice of global DP ranks), an explicit `--data-parallel-start-rank 0` is silently treated identically to \"not specified.\" This causes `data_parallel_hybrid_lb` to never be inferred `True` for that node, even though the rest of the codebase (including the sibling field `data_parallel_rank`, and the other two call sites of this same field) already use `is not None` correctly for exactly this class of field. The corrupted flag then propagates into `ParallelConfig.local_engines_only`, `wait_for_engine_startup`'s `remote_should_be_headless` computation, and the Rust-frontend engine-indexing logic added in #45805 — misclassifying the rank-0 node's role in a hybrid-LB deployment. The fix switches both truthiness checks to `is not None`, matching the established convention already used elsewhere in the same function. ## Duplicate-work check Searched for existing/overlapping work before starting: - `gh issue list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh issue list --repo vllm-project/vllm --state open --search \"data parallel rank\"` — 30 hits, none related (skimmed all titles: Ray deadlocks, MTP, PP tool-calling, RPC port robustness, etc.) - `gh issue list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank\"` — 0 relevant hits - `gh pr list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state open --search \"DP rank 0 hybrid\"` — 26 hits, none touch this logic - `gh pr list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank closed:>=2026-06-05\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state closed --search \"hybrid_lb closed:>=2026-06-05\"` — surfaced PR #45805 (merged 2026-06-17), which _consumes_ the affected `data_parallel_hybrid_lb`/`local_engines_only` flag in `serve.py` for Rust-frontend engine indexing but does not touch or fix `arg_utils.py:1997`/`2016` Zero overlapping in-flight work found across open issues, open PRs, and PRs closed in the last 30 days. ## Changes ### `vllm/engine/arg_utils.py` Line 1997, inside `create_engine_config`: ```python # Before if self.data_parallel_start_rank and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True # After if self.data_parallel_start_rank is not None and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True ``` Line 2016: ```python # Before self.data_parallel_rank = ( self.data_parallel_start_rank or inferred_data_parallel_rank ) # After self.data_parallel_rank = ( self.data_parallel_start_rank if self.data_parallel_start_rank is not None else inferred_data_parallel_rank ) ``` ### `tests/v1/engine/test_engine_args.py` Added `test_data_parallel_start_rank_zero_infers_hybrid_lb`, asserting that `data_parallel_start_rank=0` with `data_parallel_size_local=2`, `data_parallel_size=4` (headless defaults to `False`) yields `parallel_config.data_parallel_hybrid_lb is True` and `parallel_config.data_parallel_rank == 0`. Confirmed this test **fails** against the pre-fix code (verified via `git stash` on `vllm/engine/arg_utils.py` and re-running just this test) and **passes** with the fix applied — it is a genuine regression test, not a tautology. ## Test commands run and results ```bash python -m pytest tests/v1/engine/test_engine_args.py -v # 5 passed (including the new test) python -m pytest tests/engine/test_arg_utils.py tests/entrypoints/openai/test_dp_supervisor.py -v # 111 passed, no regressions pre-commit run --files vllm/engine/arg_utils.py tests/v1/engine/test_engine_args.py # ruff check: Passed # ruff format: Passed # typos: Passed # mypy (3.10): Passed # Check SPDX headers: Passed # Check root lazy imports: Passed # Check for forbidden imports: Passed # Validate configuration has default values / docstrings: Passed # (all other applicable hooks: Passed; unrelated hooks skipped — no matching files) ``` ## AI assistance disclosure This PR includes AI-assisted contributions. Claude Code was used to trace the control flow of `create_engine_config`, identify the truthiness-vs-`is not None` inconsistency, draft the fix, and draft the regression test. The human contributor independently: - Read the full `create_engine_config` function end-to-end (not just the two changed lines) to confirm no earlier code path coerces or transforms `data_parallel_start_rank` before it reaches the affected checks - Verified the sibling-field convention (`data_parallel_rank`, handled correctly with `is not None`) and the two other call sites of `data_parallel_start_rank` elsewhere in the codebase (`serve.py`, `dp_supervisor.py`), confirming both already use the correct pattern - Traced the full downstream consumer chain (`ParallelConfig.local_engines_only` → `wait_for_engine_startup`'s `remote_should_be_headless` → the Rust-frontend engine-indexing logic added in #45805) to confirm the bug has live, current consequences - Confirmed the added regression test fails on the pre-fix code and passes on the fix by manually stashing/restoring the source change and re-running the test - Ran the full test suite and pre-commit hooks locally and reviewed all output above Closes #47691", "url": "https://github.com/vllm-project/vllm/pull/47692", "createdAt": "2026-07-06T05:45:20Z", "updatedAt": "2026-08-13T15:12:27Z", "timestamp": "2026-08-13T15:12:27Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "ready", "v1" ], "author": "syedalijaseem", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47708", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Gigachat 3.5 support", "text": "## Purpose GigaChat-3.5-432B-A28B is a 432B-total / 28B-active MoE on a DeepSeek-V3-style backbone (MLA + DeepSeek MoE) with a hybrid attention stack: most layers use a Qwen3-Next Gated-Delta-Net (GDN) linear-attention block, and a periodic subset keeps full MLA. It adds gated zero-centered RMSNorm (low-rank gating bottleneck, four-norm sandwich around attention and MLP), gated attention before the output projection, learnable MLA query/key scales (`alpha_q` / `alpha_kv`), and Multi-Token Prediction (NextN) for speculative decoding. Supports bf16 and block-fp8. The implementation reuses vLLM's DeepSeek-V3 MLA, `FusedMoE`, and Qwen3-Next GDN blocks. ## Test Plan Setup: 8×H100 + TP=8 + EP Checkpoint: `ai-sage/GigaChat3.5-432B-A28B` **Serve**: ```bash vllm serve ai-sage/GigaChat3.5-432B-A28B \\ --tensor-parallel-size 8 --enable-expert-parallel \\ --gpu-memory-utilization 0.9 --max-model-len 8192 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' ``` **MATH-500 accuracy**: ```bash CKPT=ai-sage/GigaChat3.5-432B-A28B lm_eval --model local-chat-completions \\ --model_args \"model=$CKPT,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=32,tokenizer=$CKPT,trust_remote_code=True,tokenized_requests=False,max_gen_toks=2048\" \\ --apply_chat_template --fewshot_as_multiturn --tasks minerva_math500 --batch_size 32 ``` **MTP benchmark** (ShareGPT×300, concurrency 32, greedy): ```bash SRC=\"https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json\" DST=\"./ShareGPT_V3_unfiltered_cleaned_split.json\" [ -f \"$DST\" ] || wget -c -O \"$DST\" \"$SRC\" CKPT=ai-sage/GigaChat3.5-432B-A28B vllm bench serve --model $CKPT --base-url http://localhost:8000 \\ --backend vllm --endpoint /v1/completions \\ --dataset-name sharegpt --dataset-path \"$DST\" \\ --num-prompts 300 --max-concurrency 32 --temperature 0 ``` ## Test Result **MATH-500 accuracy** | Metric | no-MTP | MTP-2 | HF card | |---|---|---|---| | MATH-500 (`math_verify`) | **87.0% ± 1.5** | **84.8% ± 1.6** | **86%** | Matches the model card, the no-MTP and MTP-2 scores agree within noise (overlapping 95% CIs). **MTP performance** | Config | Output tok/s | Speedup | req/s | Mean TPOT | Accept. len | Draft acceptance | |---|---|---|---|---|---|---| | no-MTP | 1059 | 1.00× | 4.76 | 27.1 ms | — | — | | MTP-1 | 1457 | **1.38×** | 6.55 | 20.1 ms | 1.81 | 81.5% | | MTP-2 | 1600 | **1.51×** | 7.20 | 18.9 ms | 2.28 | 64.2% (81.9% / 46.5%) | Tool calling (`--tool-call-parser gigachat35`) verified end-to-end — unit tests pass. ---
Essential Elements of an Effective PR Description Checklist - [x] Purpose — new model support (`GigaChat35ForCausalLM` + MTP draft + `gigachat35` tool parser). - [x] Test plan — serve + lm-eval (MATH-500) + `vllm bench serve` commands. - [x] Test results — MATH-500 accuracy (vs HF card) + MTP acceptance / throughput. - [x] Documentation — `docs/models/supported_models.md`, `docs/features/tool_calling.md`.
", "url": "https://github.com/vllm-project/vllm/pull/47708", "createdAt": "2026-07-06T07:57:53Z", "updatedAt": "2026-08-13T01:10:42Z", "timestamp": "2026-08-13T01:10:42Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "new-model", "speculative-decoding", "needs-rebase", "v1", "tool-calling" ], "author": "perkyfever", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47808", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] DSpark confidence-scheduled verification", "text": "Adaptively sizes the DSpark draft-verification budget from per-request confidence instead of always verifying every drafted token. Motivation: fixed-k speculation collapses at high concurrency — once the GPU saturates, verifying 7 drafts per request burns more compute than the accepted tokens return, dropping **below** non-speculative decoding (see table). ## Design - A Triton kernel ranks draft slots by survival probability (cumprod of per-position confidence) and admits the prefix maximizing estimated accepted tokens per millisecond. - Step costs come from curves profiled with dummy steps at startup: a step function at/below the cudagraph capture limit (padding is physical), linear interpolation above it. - Batch-level budget sizing runs on CPU from double-buffered *stale* confidences (no sync); per-request allocation runs on GPU from live values. - Per-request confidence is smoothed with an EMA (`adaptive_verification_ema_alpha`, default 0.8 — measured bias-zero crossing across batch sizes). - Decode cudagraphs become varlen: captured on the token grid, dispatched via `max_query_len`, every captured slot non-empty. Requires `AttentionCGSupport.ALWAYS`, which the DSV4 backends report on SM100. ## Results DeepSeek-V4-Flash-DSpark, TP=4, SM100 (B300), `speed_bench` 256 prompts, 512 output tokens; throughput measured at `32881ec177`; subsequent HEAD commits are audit fixes off the captured-graph hot path (decode `max_q_len` sourcing, CPU upper-bound restoration, non-adaptive request-ordering revert to main), accuracy re-validated at HEAD. Output tok/s: \"pareto\" Adaptive matches fixed-k within ±3% at c≤64 and preserves the spec-decode win at high concurrency where fixed-k goes underwater (at c=256, fixed 7-token is 33% *below* no-spec). Acceptance length falls from ~3.9 to 3.5 at c=256 — the budget deliberately trims drafts as verification tokens become expensive. Note: the no-spec arm ran with a co-tenant job on the other half of the node; its numbers are directionally correct but modestly depressed. Adaptive/fixed arms ran on an idle node. Accuracy (re-measured at HEAD after the audit fixes below): GSM8K **0.945** (c=16) / **0.951** (c=64), 0 invalid (bar 0.84); MTBench **80/80** completed, completions manually checked for repetition/degeneration — none found.
Reproduction commands **Server** (all measurements; ablations via the noted config deltas): ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \\ --tokenizer-mode deepseek_v4 --trust-remote-code \\ --dtype bfloat16 --max-model-len 8192 \\ --tensor-parallel-size 4 --enable-expert-parallel --block-size 256 \\ --gpu-memory-utilization 0.5 --kv-cache-dtype fp8 \\ --max-num-batched-tokens 16384 --max-num-seqs 256 \\ --compilation-config '{\"max_cudagraph_capture_size\":1024}' \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"deepseek-ai/DeepSeek-V4-Flash-DSpark\",\"attention_backend\":\"FLASH_ATTN\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"enable_adaptive_verification\":true}' ``` - fixed 7-token baseline: `\"enable_adaptive_verification\":false` - no-spec baseline: omit `--speculative-config` **Benchmark harness** — `vllm bench serve` (the Python implementation; the CLI falls back to it automatically when the optional Rust `vllm-rs` binary is absent). If your install ships a Rust binary that rejects any flag below, invoke the Python implementation directly — still repo-only: ```bash python -c 'from vllm.benchmarks.serve import add_cli_args, main from vllm.utils.argparse_utils import FlexibleArgumentParser p = FlexibleArgumentParser(); add_cli_args(p); main(p.parse_args())' ``` **GSM8K** (in-tree runner; 1319 questions, 5-shot, temperature 0): ```bash python tests/evals/gsm8k/gsm8k_eval.py --port 8000 \\ --num-questions 1319 --num-shots 5 --max-tokens 256 \\ --temperature 0 --max-concurrency 16 # and 64 ``` **MTBench coherence** (80 prompts, temperature 1; inspect the saved completions for repetition/degeneration, not just the completion count): ```bash vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \\ --base-url http://127.0.0.1:8000 --model dspark-full \\ --tokenizer deepseek-ai/DeepSeek-V4-Flash-DSpark --tokenizer-mode deepseek_v4 \\ --dataset-name hf --dataset-path philschmid/mt-bench \\ --num-prompts 80 --no-oversample --hf-output-len 256 \\ --max-concurrency 16 --request-rate inf --temperature 1 \\ --save-result --save-detailed ``` **Throughput sweep** (per concurrency; one warmup pass at `--speed-bench-output-len 256 --num-prompts 64 --max-concurrency 32` first): ```bash for c in 1 16 32 64 128 256; do vllm bench serve \\ --backend openai-chat --base-url http://127.0.0.1:8000 \\ --endpoint /v1/chat/completions --model dspark-full \\ --tokenizer deepseek-ai/DeepSeek-V4-Flash-DSpark --tokenizer-mode deepseek_v4 \\ --dataset-name speed_bench --dataset-path \\ --speed-bench-dataset-subset qualitative \\ --skip-chat-template --disable-shuffle --temperature 1.0 \\ --speed-bench-output-len 512 --num-prompts 256 --max-concurrency $c \\ --save-result --result-filename adaptive_on_c${c}.json done ``` `--disable-shuffle` plus the fixed prompt set gives every arm identical prompts in identical order; `output_throughput` from the result JSON is the tok/s reported above. The prompt set is an internal speed-bench \"qualitative\" subset; any fixed prompt set of comparable length distribution reproduces the relative behavior, e.g. `--dataset-name random --random-input-len 1024 --random-output-len 512`.
## Limitations - Output logprobs are rejected when confidence-based verification is enabled (verification compacts logits after the forward pass). - Non-SM100 falls back to PIECEWISE decode graphs: correct, but without the varlen-FULL-graph throughput win. ## Tests ``` pytest tests/v1/spec_decode/test_adaptive_verification.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py tests/v1/cudagraph/ -q ``` All pass; full pre-commit green. GSM8K eval config added at `tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml`. ## Not a duplicate Canonical PR for DSpark confidence-based verification; no other open PR touches adaptive draft budgeting. --- AI assistance was used for this change (see commit trailers). 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/47808", "createdAt": "2026-07-07T03:21:26Z", "updatedAt": "2026-08-13T00:57:35Z", "timestamp": "2026-08-13T00:57:35Z", "metrics": { "reactions": 12, "comments": 41 }, "labels": [ "documentation", "performance", "speculative-decoding", "ready", "ci/build", "v1", "qwen", "nvidia", "mrv2" ], "author": "LucasWilkinson", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:47941", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] P2P NIXL + CPU EC Connector", "text": "## Purpose This PR extends ECCPUConnector to support P2P EC sharing based on NIXL. [The original PR](https://github.com/vllm-project/vllm/pull/42998) was split into two parts; [The first part](https://github.com/vllm-project/vllm/pull/47423) is a standalone CPU-based offloading EC connector, and this PR is an extension of it, allowing CPU-offloaded EC cache to be shared between vLLM instances via NIXL. The control plane is ZMQ (consumer DEALER → producer ROUTER) carrying msgpack XferReq / XferAck. **Dependencies:** - [EC Transfer Params](https://github.com/vllm-project/vllm/pull/42433) - [CPU OffloadingEC Connector](https://github.com/vllm-project/vllm/pull/47423) ## Logic - **Producer scheduler.** Binds a ZMQ ROUTER. A single router thread owns the socket, ingests `XferReq` frames, validates `mm_hash` presence in `_local_encodings` (NACK queued otherwise), pins the matching block indices in the CPU mmap region, and posts an async NIXL `WRITE` directly to the consumer's pre-registered destination blocks (In the consumer's CPU mmap region). Completions are polled on every router-loop iteration via `_sweep_completions`, which unpins blocks and emits `XferAck(ok=True/false)`. Two-phase bookkeeping (`_pending_save` → `_local_encodings`) prevents serving pre-`save_caches` garbage: a freshly allocated block range only becomes serveable on the next `build_connector_meta` step, by which point the worker's `save_caches` has finished and synchronized. - **Producer worker.** `save_caches` copies `encoder_cache[mm_hash]` → `mmap[block_indices]` on a dedicated copy stream and CPU-syncs before returning, so the bytes are coherent for the scheduler's NIXL reads. - **Consumer scheduler.** Lazy DEALER pool keyed by `(host, port)` with ZMTP heartbeat + per-DEALER monitor sockets subscribed to `EVENT_DISCONNECTED`. `ensure_cache_available(request)` allocates blocks for unseen `mm_hash`es from `request.ec_transfer_params`, sends `XferReq`, and returns `pending=True` until ack arrives. A `_loaded` mmap cache keeps completed transfer blocks alive for local re-copy on subsequent requests (`_pending_reload`), avoiding the producer round-trip when possible. - **Consumer worker.** `start_load_caches` copies `mmap[block_indices]` → `encoder_cache[mm_hash]` on a dedicated copy stream. - **Failure path.** Producer-side NACK (unknown `mm_hash`, compat hash mismatch, evicted) → consumer frees blocks and falls through to local encode. Producer disconnect → ZMTP heartbeat fires `EVENT_DISCONNECTED` within `HEARTBEAT_TIMEOUT` (4s); `on_peer_down` evicts the peer, removes the NIXL agent, and tombstones in-flight entries so stalled requests fall back cleanly. ## End-to-End Architecture ``` ┌──────────────────────────────────────┐ │ External caller (orchestrator + │ │ request router, out of scope here) │ │ │ └─┬──▲────────────────────────┬────────┘ │ │ │ HTTP │ │ HTTP response body │ HTTP req w/ req │ │ carries top-level │ ec_transfer_params (OpenAI│ │ ec_transfer_params │ in extra_args + b64 │ │ │ + │ │ │ max=1)│ │ │ ▼ │ ▼ ┌──────────────┬──┴──────────┐ ┌──────────────────────────┐ │ Producer vLLM │ │ Consumer vLLM │ │ ec_role=ec_producer │ │ ec_role=ec_consumer │ │ mm_encoder_only=true │ │ │ │ │ │ │ │ Scheduler │ │ Scheduler │ │ ├─ ECSharedRegion │ │ ├─ ECSharedRegion │ │ │ (mmap, NIXL-reg, │ │ │ (mmap, NIXL-reg, │ │ │ alloc/free/pin) │ │ │ alloc/free) │ │ ├─ _local_encodings: │ │ ├─ _remote_encodings: │ │ │ mm_hash → │ │ │ mm_hash → │ │ │ block_indices │ │ │ block_indices │ │ ├─ NIXL agent │ │ ├─ _ready: arrived │ │ ├─ ZMQ ROUTER │ │ │ mm_hashes │ │ │ (VLLM_EC_SIDE_CHANNEL) │ │ ├─ NIXL agent │ │ ├─ router thread │ │ ├─ peer pool: │ │ └─ in-flight xfers │ │ │ (host,port) → │ │ │ │ │ (DEALER,agent,md) │ │ Worker │ │ └─ ensure_cache_ │ │ └─ save_caches: │ │ available(request) │ │ GPU → mmap │ │ │ │ at block_indices │ │ Worker │ │ from metadata │ │ └─ start_load_caches: │ │ │ │ mmap → GPU │ │ request_finished() emits │ │ at block_indices │ │ ec_transfer_params │ │ from metadata │ │ in response body │ │ │ └──────────┬─────────────────┘ └──────────┬───────────────┘ │ │ │ ROUTER ← ZMQ DEALER │ │ XferReq / XferAck │ │ ◄───────────────────────────── │ │ │ │ NIXL WRITE │ │ mmap → remote mmap │ │ ──────────────────────────────►│ └────────────────────────────────►│ ``` ## Lifecycle (cold path) 1. Caller dispatches request to producer. Producer runs with `--mm-encoder-only`. 2. Producer scheduler `build_connector_meta` allocates blocks for the newly-scheduled `mm_hash`, parks them in `_pending_save`, emits `meta.saves[mm_hash] = block_indices`. 3. Producer worker's `save_caches` copies GPU → mmap at those indices, synchronizes the copy stream. 4. Next step's `build_connector_meta` promotes `_pending_save` → `_local_encodings`. Producer's `request_finished` returns `ec_transfer_params = {mm_hash: {peer_host, peer_port, size_bytes, nixl_agent_metadata_b64}}` in the response body. 5. Caller forwards the original request to a consumer with `sampling_params.extra_args[\"ec_transfer_params\"]` populated from (4). 6. Consumer scheduler `ensure_cache_available` allocates blocks, opens a ZMQ DEALER to the producer (cached in the peer pool), `add_remote_agent`s the producer's NIXL identity, and sends `XferReq` over the DEALER. 7. Producer router thread receives `XferReq` over the ZMQ ROUTER, pins blocks, posts NIXL `WRITE` to consumer's `dst_block_indices`. On `check_xfer_state == DONE`, sends `XferAck(ok=True)` and unpins. 8. Consumer `_drain_acks` consumes the ack, moves `mm_hash` to `_ready`. Next `build_connector_meta` adds it to `meta.loads` and moves blocks to `_loaded`. Worker copies mmap → GPU on its dedicated stream. 9. Subsequent requests for the same `mm_hash` re-serve via local mmap → GPU re-copy (`_pending_reload`), no producer round-trip (until evicted when the CPU mmap region is full - FIFO policy). ## Configuration All knobs in `ec_connector_extra_config`. | Field | Scope | Default | Purpose | |---|---|---|---| | `--mm-encoder-only` (existing) | producer, required | `False` | Skip the LM entirely; only the vision encoder runs. | | `ec_connector_extra_config.num_ec_blocks` | both | `80000` | Block count in the mmap region on each side. | | `VLLM_EC_SIDE_CHANNEL_HOST` (env) | producer | pod IP | ZMQ ROUTER bind host. | | `VLLM_EC_SIDE_CHANNEL_PORT` (env) | producer | — | ZMQ ROUTER bind port; advertised to caller in response's `peer_port`. | Consumer needs no static peer config — every value to reach a producer arrives per-request via `ec_transfer_params`. ## Testing plan - **Unit** (`tests/v1/ec_connector/unit/`): `test_metadata.py` (msgpack round-trip, compat hash factors); `test_scheduler.py` (~10 tests covering producer + consumer branches: alloc/send, idempotency, ack success/failure, peer-pool invalidation, router-thread WRITE/NACK, concurrent xfers per `mm_hash`, `request_finished` shape); `test_worker.py` (3 tests for save/load copy paths); `test_ec_shared_region.py`, `test_utils.py`, `test_common.py` - **Run command:** ```bash python -m pytest tests/v1/ec_connector/unit/ -v ```", "url": "https://github.com/vllm-project/vllm/pull/47941", "createdAt": "2026-07-08T01:43:53Z", "updatedAt": "2026-08-13T10:19:02Z", "timestamp": "2026-08-13T10:19:02Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "cpu", "kv-connector", "mrv2", "rust" ], "author": "omerpaz95", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] test_batch_inference_correctness now uses batch invariance", "text": "## Purpose Partially Fixes [48038](https://github.com/vllm-project/vllm/issues/48038). An example PR fixing a test flagged by our audit. The test now enforces batch invariance to prevent accidental numerical changes causes falsely failing tests. The test is no longer coincidentally correct and enforces a strong contract: Spec decode exact matching only when the test explicitly forces batch-invariant mode/kernels. ## Test Plan N/A ## Test Result The test still passes with batch in variance turned on. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48040", "createdAt": "2026-07-08T20:17:50Z", "updatedAt": "2026-08-13T16:29:46Z", "timestamp": "2026-08-13T16:29:46Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "v1" ], "author": "morrison-turnansky", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48116", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Structured Output][Tool Calling] Use xgrammar's reasoning parser instead vLLM's", "text": "## Purpose Filter reasoning for structured output and tool calling on Xgrammar's side instead of vLLM's side to simplify logic of scheduler. Recently many bugs related to combinations of structured output/tool callings with reasoning were reported in vLLM. All these issues can be resolved on vLLM's side but requires code changes inside scheduler that makes them potentially hard to implement and test correctly. Instead Xgrammar itself can handle reasoning, so no need for complex logic on vLLM's side. For tool calling we can simply turn off existing vLLM's reasoning filtering and use instead Xgrammar's one. Structured output (SO) is a bit more different case. SO can be one of the following formats: json object, json schema, structural tag. For structural tag Xgrammar has native reasoning support, for json it currently lacks support. I will do a PR to Xgrammar to implement native reasoning support for json. But as a current workaround we can translate json to structural tag and then reasoning support works as expected. In this PR this approach was implemented for now only for Qwen3.5 as a PoC. ## Test Result ### spec decoding + tool_choice auto/required + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/44006.
server ```bash export VLLM_ENFORCE_STRICT_TOOL_CALLING=1 vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 -tp 2 \\ --language-model-only \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --max-model-len 8192 \\ --enable-log-requests \\ --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ```
client ```bash run() { # $1 = tool_choice, $2 = N local fail=0 for i in $(seq 1 \"$2\"); do code=$(curl -s -o /tmp/r.json -w '%{http_code}' \\ http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\":\"Qwen/Qwen3.5-35B-A3B\", \"messages\":[{\"role\":\"user\",\"content\":\"What is the weather in Seoul? Think about which tool to use first.\"}], \"tool_choice\":\"'\"$1\"'\",\"max_tokens\":4096,\"temperature\":0.6,\"top_p\":0.95, \"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather information\",\"parameters\":{\"type\":\"object\",\"properties\":{\"location\":{\"type\":\"string\"}},\"required\":[\"location\"]}}}] }') echo \"[$1][$i] HTTP $code\" [ \"$code\" = \"200\" ] || fail=$((fail+1)) done echo \"---- $1: failed $fail/$2\" } run required 30 run auto 30 ```
**Result**: correct; no 500 HTTP responses on server side Even though this bug has already been fixed by https://github.com/vllm-project/vllm/pull/44297 the fix itself is complicated, so probably it is better to process reasoning on xgrammar's side to keep vLLM's scheduler logic simple. This PR alone fixes the same issue but in a more simple way. ### spec decoding + structured output json schema + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/34650
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 -tp 2 \\ --language-model-only \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --max-model-len 8192 \\ --enable-log-requests \\ --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Pick a primary color and say how many letters its name has.\", } ], \"response_format\": { \"type\": \"json_schema\", \"json_schema\": { \"name\": \"color_fact\", \"schema\": { \"type\": \"object\", \"properties\": { \"color\": {\"type\": \"string\"}, \"letter_count\": {\"type\": \"integer\"}, }, \"required\": [\"color\", \"letter_count\"], }, }, }, \"chat_template_kwargs\": {\"enable_thinking\": True}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"color\\\": \\\"Blue\\\",\\n \\\"letter_count\\\": 4\\n}\"` -- correct; no 500 HTTP responses on server side ### spec decoding + structred output json object + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/48228
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Return a JSON object with keys name and age for Bob aged 30.\", } ], \"response_format\": {\"type\": \"json_object\"}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"name\\\": \\\"Bob\\\",\\n \\\"age\\\": 30\\n}\"` -- correct; no 500 HTTP responses on server side ### spec decoding + async scheduling + structred output json object + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/43388
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 \\ --reasoning-parser qwen3 \\ --async-scheduling \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":4}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Return a JSON object with keys color and letter_count for the color red.\", } ], \"response_format\": {\"type\": \"json_object\"}, \"chat_template_kwargs\": {\"enable_thinking\": True}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"color\\\": \\\"red\\\",\\n \\\"letter_count\\\": 3\\n}\"` -- correct; no 500 HTTP responses on server side ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48116", "createdAt": "2026-07-09T10:55:44Z", "updatedAt": "2026-08-12T18:21:30Z", "timestamp": "2026-08-12T18:21:30Z", "metrics": { "reactions": 4, "comments": 8 }, "labels": [ "structured-output", "ci/build", "v1", "tool-calling" ], "author": "arpera", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:48200", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Refactor]: StructuredOutputManager x Speculative Decoding Refactor", "text": "## Purpose See https://github.com/vllm-project/vllm/issues/48197. ~100 insertions, ~200 deletions of non-test changes. Then ~400 insertions, ~600 deletions for test consolidation. Moves `tests/v1/spec_decode/test_mtp_structured_output.py` and `tests/v1/structured_output/test_reasoning_structured_output.py` into `tests/v1/structured_output/test_structured_output_manager.py`. Tests are carefully picked to be more systematic while maintaining existing coverage. We can defer test consolidation to another PR to ensure that refactor pass current tests. ## Test Plan ``` venv/bin/python -m pytest \\ tests/v1/structured_output/test_structured_output_manager.py \\ tests/v1/core/test_scheduler.py \\ tests/v1/core/test_async_scheduler.py ``` Same repro as in https://github.com/vllm-project/vllm/pull/44297: https://gist.github.com/yzong-rh/b01fb276feb8f457aa68275cd44cae10 - Expected to pass both on main and on this branch Similar repro but with JSON constaints instead of STRUCTURAL_TAG constraints (see https://github.com/vllm-project/vllm/pull/44993): https://gist.github.com/yzong-rh/ed62c5f8a9fc85fb9633bb630dcc0244 - ~~Failing on main~~ (Fixed by https://github.com/vllm-project/vllm/pull/44993) - Pass on this PR ## Test Result ``` 178 passed, 1 xfailed, 31 warnings in 109.18s (0:01:49) ``` Structural Tag constraints: ``` Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Successes: 50 (100.00%) Failures: 0 (0.00%) Successful tool calls: 50 ``` JSON Constraints: ``` Main: Using model: Qwen/Qwen3.6-35B-A3B Progress: S...SS..S.S.S.S.S......S..S.......S..S....SSS.S... Summary Total requests: 50 Valid schema-constrained responses: 34 (68.00%) Failures: 16 (32.00%) invalid_json: 16 This: Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Valid schema-constrained responses: 50 (100.00%) Failures: 0 (0.00%) ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48200", "createdAt": "2026-07-10T02:24:24Z", "updatedAt": "2026-08-13T14:21:19Z", "timestamp": "2026-08-13T14:21:19Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "structured-output", "speculative-decoding", "v1" ], "author": "yzong-rh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48212", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Add CPU-tuned autotune configs for FLA (GDN) Triton kernels", "text": "> **Depends on #49583.** That PR bumps the triton-cpu pin to a commit that includes triton-cpu #275 (the `CPUOptions.hash()` runtime-only cache-key fix). Without it, the configs here recompile once per `num_cpu_threads` candidate, so the warmup speedup is only realized after #49583 lands. ## Purpose The FLA op kernels used GPU-oriented autotune configs that vary num_warps/num_stages. On the Triton CPU backend those parameters are ignored, so the search wasted many minutes recompiling identical x86 code during first-run warmup. For each FLA kernel, when current_platform.is_cpu(): - Replace the block-size search with @triton.heuristics that pick tile sizes deterministically via min(triton.next_power_of_2(dim), 64/128), keeping tiles L1-resident. - Autotune only num_cpu_threads over [16, 32, 64, 96] (the real CPU parallelism knob controlling the OpenMP launch). Guarded on current_platform.is_cpu() so GPU behavior is unchanged. Change-Id: I3edba2bd1fd5940d0f4d7ad6d9b752752c3b3d2e ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48212", "createdAt": "2026-07-10T05:36:58Z", "updatedAt": "2026-08-13T07:17:33Z", "timestamp": "2026-08-13T07:17:33Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "cpu" ], "author": "ganeshr10", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48215", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][LoRA] Add tower/connector LoRA support for Ultravox", "text": "# Purpose Part of #31479: enable `enable_tower_connector_lora` for Ultravox. This is not duplicate work. The open-PR and issue checks for #31479 show this as the only Ultravox tower/connector LoRA implementation; the other open PRs cover different multimodal model families. Ultravox previously could not apply LoRA to its audio tower and connector: - The tower/projector path was not fully built from LoRA-wrappable vLLM modules, so tower/connector adapters could silently become no-ops. - The number of tower and connector rows processed per audio item could not be recovered exactly from the multimodal placeholder count. - Ultravox's variable-length Whisper chunks require padding-aware attention, while tower/connector LoRA requires stable per-chunk row counts. # Implementation - Reuse the native `WhisperEncoder` through a thin Ultravox subclass instead of maintaining a separate encoder implementation. - Extend Whisper encoder self-attention to accept optional multimodal `cu_seqlens` metadata. Each chunk's valid frames and padding tail are separate sequences, so attention cannot cross the valid/padding boundary while all rows still pass through LoRA-wrapped linears. Decoder and cross-attention behavior is unchanged. - Use vLLM-native linear layers for the Ultravox connector. - In tower/connector LoRA mode, pad every audio chunk to the Whisper tower's full context. This makes every chunk contribute exactly `max_source_positions` tower rows and `ceil(max_source_positions / stack_factor)` connector rows. - Add exact `get_num_mm_encoder_tokens` and `get_num_mm_connector_tokens` mappings. The connector count is computed per chunk because `StackAudioFrames` pads each chunk before stacking; a single floor division over all encoder tokens would undercount multi-chunk audio. - Process the encoder batch in one pass under tower/connector LoRA so the token-to-LoRA mapping remains aligned across every chunk. - Load Whisper tower weights with `AutoWeightsLoader`, a `WeightsMapper` for q/k/v fusion and fc1/fc2 nesting, and Whisper's existing synthetic k-projection bias helper. The PR now contains implementation changes only. The test files added in earlier revisions (`tests/models/multimodal/processing/test_ultravox.py` and `tests/lora/test_ultravox.py`) were removed in `1507808e5` at reviewer request. The manual validation below was performed on `1284ff2a4`, the last revision containing those tests; it is recorded here as historical validation and is not part of the current PR's CI coverage. # Validation performed before test removal Manual validation performed on an NVIDIA A10 host: - Processor / placeholder-count boundaries (0.1s, 29.99s, 30s, 30.02s, 60s, 100s audio): actual placeholder and chunk counts compared against the tower/connector mappings and `StackAudioFrames` — **15 passed**. - Tower-only and connector-only nonzero adapters each changed deterministic output logprobs, confirming that neither code path was a no-op — **1 passed**. - Dynamic LoRA vs. an independently merged checkpoint: the same nonzero adapter was folded into the base weights with `W + (alpha / r) * B @ A`. Both runs used identical tower/connector preprocessing and kernel shapes — **1 passed**. Test-host settings, which did not change the code under test: `VLLM_ENABLE_V1_MULTIPROCESSING=0` avoided a multithreaded-pytest `fork()` deadlock, and a 40% GPU-memory budget per engine allowed the two sequential reference engines to coexist during in-process cleanup. The processing test used an equivalent cached local mirror because the test host could not access the gated Meta repository. # Tests run Local, on the two files that remain in this PR: - `.venv/bin/pre-commit run --files vllm/model_executor/models/ultravox.py vllm/model_executor/models/whisper.py` — passed. - `.venv/bin/pre-commit run mypy-3.12 --hook-stage manual --files vllm/model_executor/models/ultravox.py vllm/model_executor/models/whisper.py` — passed. # Model evaluation result The dynamic tower/connector LoRA path and the independently merged checkpoint produced identical 32-token greedy paths; every chosen token's logprob matched within `rtol=0, atol=2e-2`. The tower-only and connector-only adapters each produced nonzero logprob shifts against the same base run. --- ## AI assistance AI assistance (Claude and OpenAI Codex) was used to author and test this change. The submitter has reviewed every changed line and can explain and defend the implementation and validation.", "url": "https://github.com/vllm-project/vllm/pull/48215", "createdAt": "2026-07-10T06:09:30Z", "updatedAt": "2026-08-13T05:58:00Z", "timestamp": "2026-08-13T05:58:00Z", "metrics": { "reactions": 2, "comments": 29 }, "labels": [ "ready", "multi-modality" ], "author": "arthurgao2003", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48247", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][ROCm] Add AITER custom AG/RS", "text": "## Purpose Part of #48255. Only activated under uniform batches. Enabled by default when using DP attention. Same as ATOM. Disable with `VLLM_ROCM_USE_AITER_CUSTOM_AR=0`. **Perf gain:** **~3% improved TPOT on 1k/1k** (1-256 conc) ### Technical details - AG: Runs as 3 separate calls for hidden, topk_weights, and topk_ids. AITER doesn't have a grouped AG unfortunately. \"image\" - Zooming in we see these kernels now being run in uniform decode batches: \"image\" \"image\" - Mixed batches still use NCCL/RCCL: \"image\" ## Test Plan Tested on 8xMI300. 1. Run DSv3 with DPA+TP: ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` Compare vs nightly `vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9`. 2. Bench serve sweep, 1k/1k and 8k/1k ISL/OSL - 10x as many prompts as the concurrency level; 2x number of warmups. ```bash vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 \\ --num-warmups 256 ``` 3. GSM8k accuracy non-regression ```bash lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result ### 1. Performance Only decode path affected, so focus on TPOT. #### 1k/1k (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 262.54 | 21.64 | **+12.46%** | | 1 | nightly | 0.04 | 273.16 | 24.72 | - | | 2 | this branch | 0.08 | 439.28 | 23.60 | **+6.79%** | | 2 | nightly | 0.08 | 298.14 | 25.32 | - | | 4 | this branch | 0.16 | 481.21 | 24.02 | **+12.50%** | | 4 | nightly | 0.14 | 366.23 | 27.45 | - | | 8 | this branch | 0.29 | 701.59 | 26.51 | **+16.98%** | | 8 | nightly | 0.24 | 664.92 | 31.93 | - | | 16 | this branch | 0.44 | 1065.97 | 34.56 | **-5.37%** | | 16 | nightly | 0.46 | 851.06 | 32.80 | - | | 32 | this branch | 0.68 | 2125.91 | 44.33 | **-2.57%** | | 32 | nightly | 0.70 | 1698.74 | 43.22 | - | | 64 | this branch | 1.32 | 2901.05 | 44.66 | **+2.19%** | | 64 | nightly | 1.29 | 2528.00 | 45.66 | - | | 128 | this branch | 2.17 | 4486.44 | 53.32 | **+2.82%** | | 128 | nightly | 2.10 | 4543.42 | 54.87 | - | | 256 | this branch | 3.48 | 6351.00 | 65.49 | **+3.49%** | | 256 | nightly | 3.37 | 6386.54 | 67.86 | - | #### 8k/1k Not as clear improvement, a bit variable. Expecting 1k/1k results to be stronger since that's more decode-heavy, which this PR should affect. (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.0420 | 1690.7196 | 21.6517 | **+3.80%** | | 1 | nightly | 0.0404 | 1681.0387 | 22.5068 | - | | 2 | this branch | 0.0705 | 1872.6645 | 25.4534 | **-11.53%** | | 2 | nightly | 0.0766 | 1872.0056 | 22.8210 | - | | 4 | this branch | 0.1306 | 3409.2517 | 27.2208 | **+3.05%** | | 4 | nightly | 0.1241 | 3439.4700 | 28.0764 | - | | 8 | this branch | 0.2267 | 4008.7127 | 29.8010 | **+2.15%** | | 8 | nightly | 0.2214 | 3836.1143 | 30.4567 | - | | 16 | this branch | 0.3372 | 5931.2343 | 39.9316 | **+9.28%** | | 16 | nightly | 0.3135 | 6404.6625 | 44.0170 | - | | 32 | this branch | 0.4641 | 8843.5975 | 59.0122 | **-7.02%** | | 32 | nightly | 0.4905 | 9516.1175 | 55.1427 | - | | 64 | this branch | 0.7066 | 11619.6178 | 77.3325 | **+2.10%** | | 64 | nightly | 0.6972 | 11196.8358 | 78.9877 | - | | 128 | this branch | 0.9199 | 14325.6379 | 122.2971 | **-0.14%** | | 128 | nightly | 0.9067 | 13800.9142 | 122.1202 | - | ### 2. Accuracy #### This branch |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9469|± |0.0062| | | |strict-match | 5|exact_match|↑ |0.9462|± |0.0062| #### Nightly |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48247", "createdAt": "2026-07-10T11:36:54Z", "updatedAt": "2026-08-13T12:54:34Z", "timestamp": "2026-08-13T12:54:34Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "rocm", "nvidia" ], "author": "simondanielsson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48263", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Connector][4/N][NIXL] Recover all dedup'd HMA pool members (FA + multi-group SSM) under PP", "text": "## Purpose Fixes silent KV-cache corruption for **hybrid (Mamba/SSM) models under pipeline parallelism** in NIXL PD-disaggregated serving. Under PP>1 + the hybrid-memory allocator (HMA), the general-case allocator pools **one layer from every `kv_cache_group`** into a single shared tensor. For a multi-Mamba-group hybrid (e.g. NVIDIA Nemotron-3-Ultra-550B: 4 Mamba groups + 1 FullAttention), each pooled tensor holds *N* Mamba layers + 1 FA layer, all full-aliased at offset 0. `register_kv_caches` dedups by base address and keeps only the **group-0 Mamba representative**; the other (N−1) dedup'd Mamba members and the dedup'd FA member fall out of `registered_layer_names`. As a result `mamba_region_group_ids` collapses to `[0]*N`: only the group-0 Mamba layers transfer, and the remaining `(num_mamba_groups−1)/num_mamba_groups` of the Mamba layers run on **stale/zero SSM+conv state** on the decode side. The output is fluent but wrong-content — a silent correctness bug, not a crash. The stacked base #43368 introduced `region_members` (the per-region member list) and per-layer HMA routing, but its dedup branch only records members and `continue`s — it re-advertises **no** dedup'd member (verified against `zixi-qi:pr2/pp-disagg-nixl-hma`), so every non-representative member of a pooled tensor is dropped from the transfer set. This PR makes the producer **re-advertise every dedup'd member** (each FA member with the FA page stride; each extra-group Mamba member with the Mamba page stride) as its own NIXL region, so `registered_layer_names` becomes complete and `mamba_region_group_ids` de-collapses to `[0,1,2,…]`. The consumer applies an SSM-only region filter — gated on the new capability flags — before the Mamba descriptor builder so the SSM desc-id math stays aligned once FA regions are present. Fixes #46407. Root cause + live evidence tracked on #43765 / #43508. > **Stacked on #43368** (per-layer-name HMA routing), which is stacked on #43366 (NIXL PP support). This is [4/N] in that series. It is **Draft** until #43368 merges; the diff below is scoped to the incremental delta on top of #43368's branch and will rebase onto `main` unchanged once #43368 lands. Not related to #45880 (push-mode PP prefill, orthogonal; merged 2026-07-08) — this is the pull-mode path. ## Changes Net delta on top of #43368 — **2 files, ~2 new methods + 2 fields + one producer branch + one consumer gate** (self-contained; no change to the PP-routing, TP-mapping, or handshake wire format that #43368 owns): ``` vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py | +2 fields, version bump vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py | +2 methods, +1 producer branch, +1 consumer gate ``` **`metadata.py`** - Add `fa_members_registered: bool = False` and `ssm_members_registered: bool = False` to `NixlAgentMetadata`. Both are read defensively (`getattr(meta, ..., False)`) so a v6 producer that omits them stays wire-compatible, and a newer consumer paired with an older producer degrades to the legacy path safely. - Bump `NIXL_CONNECTOR_VERSION` and document the two new versions in the existing version-history block. - (`region_members` and `registered_layer_names` already exist from #43368 — unchanged here.) **`worker.py`** - `register_kv_caches`: in the existing dedup branch (`existing_region_idx is not None`), when a dedup'd member is a FullAttention-on-Mamba alias **or** a distinct-Mamba-group-on-Mamba alias, **re-advertise it as its own region** at the same base addr (full-alias, offset 0) with the member's own page stride (FA physical page for FA; `physical_page_size // physical_blocks_per_logical_kv_block` for SSM). Set `fa_members_registered` / `ssm_members_registered` accordingly. Two by-construction `assert`s enforce the HMA full-alias invariant (equal base, equal `page_size_bytes`) so any future layout change fails loud at registration instead of silently mis-transferring. - New `_region_fa_recovery_for_members(...)`: per producer region, recover a dedup'd FA member hidden behind a non-FA representative, resolving its kv-group by layer name (PP-stable). - New `_ssm_layer_names(...)`: filter advertised layer names to SSM-group layers only. - `_mamba_region_group_ids_for_layer_names(...)`: route through `_ssm_layer_names` so the Mamba desc bookkeeping covers only SSM layers (attention layers no longer inflate the count / shift SSM offsets). - Consumer (`add_remote_agent`, local + remote descriptor layouts): apply the SSM-only region filter to the meta the Mamba builder sees, gated on `(fa_members_registered or ssm_members_registered)`. **Inert on unaffected paths (byte-unchanged):** PP=1 (each layer is already its own representative), all-attention, single-Mamba-group, and MLA producers never set either flag, so the re-advertise branch never fires and the consumer filter is a no-op — matching the #43366/#43368 gating convention. Verified by the +486 MB PP1 control being unchanged. ## Test Plan **Unit** (deterministic, no GPU — the CI gate): ``` pytest -q tests/v1/kv_connector/unit/test_nixl_connector_pp_hma_multimember.py ``` New test constructs a producer whose HMA pool shares one tensor across *N*>1 Mamba groups + 1 FA group (multi-group, mirroring Nemotron-3-Ultra), runs `register_kv_caches`, and asserts: - every dedup'd member (all *N* Mamba reps + the FA member) appears in `registered_layer_names`; - `fa_members_registered is True` and `ssm_members_registered is True`; - `_mamba_region_group_ids_for_layer_names(...)` de-collapses from the pre-fix `[0]*4N` to `[0,1,…,N-1]` (×4 for conv0/1/2+ssm); - a single-Mamba-group / PP1 / all-attention config leaves both flags `False` and the region set byte-identical to pre-fix (guards the inert paths). **E2E accuracy** (registered into CI): add a multi-Mamba-group hybrid PP config to `tests/v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh` (`HYBRID_SSM` group) and wire the label in `.buildkite/test_areas/disaggregated.yaml`, running GSM8K 5-shot parity between an aggregated baseline and the PP-disaggregated deployment (0 transfer errors, accuracy equal within noise). Reproducible serve commands (producer PP=2 / consumer PP=2, `kv_both`, LIBFABRIC backend over EFA) and the aggregated control are in the PR's `tests/.../nixl_integration/` harness. ## Test Result **Unit:** _to be pasted from the PR branch after `pytest` on the rebased tree (expected: N passed)._ **E2E — live on 2× p5en (8× H200) over EFA, real Nemotron-3-Ultra-550B (hybrid Mamba2 + attention), TP8×PP2 both roles, NIXL/LIBFABRIC, 2026-06-26 → 2026-07-07:** | Metric | Pre-fix (PP2) | This PR (PP2) | Aggregated PP1 control | |---|---|---|---| | `mamba_region_group_ids` | `[0]*24` (collapsed) | `[0×24, 1×24, 2×24, 3×24]` | n/a | | `region_group_ids` | all-SSM | `[0..4 ×12]` (5 groups) | n/a | | `num_regions` | 24 | 60 | n/a | | KV over EFA (decode) | degenerate | **~487 MB/req**, 4221–4917 MB/s, 804 descriptors, 48 transfers, 0 failures | +486 MB/req | | Decode output | fluent but **wrong-content** (stale SSM) | **byte-parity vs control** | ground truth | - Correctness parity vs the aggregated PP1 control (same image, temp 0, seed 42, identical prompts): **6/6 answers correct**; **5/6 strict byte-identical** (the one diff is a reasoning-preamble prose difference with the answer \"Paris\" identical — cross-parallelism fp reduction-order nondeterminism, TP8×PP2 vs TP8×PP1). Both verbatim needle recalls (`MAROON-8817-QUARTZ`, `73912`) are byte-perfect — a stale-SSM/wrong-KV bug corrupts random-string recall first, and it did not. - Spot checks post-fix: \"capital of France\" → Paris; \"17 plus 25\" → 42; monotonic sequence continuation correct.
Essential Elements of an Effective PR Description Checklist - [x] Purpose — fixes the multi-member HMA region-group collapse left by #43368; links #46407 / #43368 / #43366 - [x] Test plan — new deterministic unit test + GSM8K PD accuracy config wired into `disaggregated.yaml` - [x] Test result — de-collapse table + ~487 MB/req KV-over-EFA + GSM8K/byte parity (unit counts pasted on the PR branch pre-merge) - [ ] (Optional) Documentation update — n/a (internal connector path; no user-facing model/API/example change)
## Related PRs / Issues - Fixes #46407 (bug report + root cause, ours). - Stacked on #43368 (per-layer-name HMA routing) → #43366 (NIXL PP support). Credits @zixi-qi for the PP + `region_members` foundation this extends. - Context: #43765 (hybrid SSM/GDN PD feature), #43508 (Mamba `_mamba_block_aligned_split` assertion). - Not #45880 (push-mode PP prefill; orthogonal, merged) — this is the pull-mode path. Signed-off-by: Anton Alexander <dmvevents@gmail.com> This PR was AI-assisted; the submitter has reviewed every line and reproduced the results above on live hardware.", "url": "https://github.com/vllm-project/vllm/pull/48263", "createdAt": "2026-07-10T16:52:01Z", "updatedAt": "2026-08-13T14:00:54Z", "timestamp": "2026-08-13T14:00:54Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "needs-rebase", "v1", "kv-connector" ], "author": "dmvevents", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48282", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Emit added/done lifecycle events for zero-delta streaming items", "text": "Fixes #48274 ## What's going on There's a known bug in `emit_previous_item_done_events` where items that complete with zero deltas get silently dropped from the SSE stream. The code already acknowledges it: ```python if not state.sent_output_item_added and not state.is_first_function_call_delta: # Suppress done events for items had no delta and thus had no # added/in-progress lifecycle events. This is a bug. # TODO: Ensure added/in-progress events are emitted for zero-delta items. return [] ``` This means any client tracking output items by lifecycle events just never sees these items at all. ## The fix I added a small helper `_emit_zero_delta_added()` that synthesizes the missing `response.output_item.added` event when an item completes without any prior deltas. It covers all five item types: function calls, code interpreter, MCP calls, reasoning, and text messages. The helper sets up the correct `StreamingState` flags so the downstream `emit_*_done_events` functions work as expected. The existing `emit_previous_item_done_events` now calls this helper instead of bailing out early. If the helper can't identify the item type (unknown channel, no recipient), it still returns early to avoid emitting orphaned done events. Also removed the `xfail` marker from `test_zero_delta_items_should_preserve_streaming_lifecycle` since the test passes now. ## Testing The xfail test that was already written for this bug now runs green. It covers all the zero-delta item types (text, function call, code interpreter, MCP, reasoning) and validates the full added->done lifecycle for each. ## AI disclosure This PR was developed with assistance from Cursor (AI coding tool). I reviewed and validated all changes.", "url": "https://github.com/vllm-project/vllm/pull/48282", "createdAt": "2026-07-10T21:45:25Z", "updatedAt": "2026-08-12T22:27:36Z", "timestamp": "2026-08-12T22:27:36Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "frontend" ], "author": "mimran-khan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48290", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunner v2] Enable MRV2 for pooling models by default", "text": "## Purpose Upstream defaults every pooling model to Model Runner V1 even though the V2 pooling path now supports both encoder and decoder text tasks. This enables MRV2 by default for **native text-only pooling models**, including Qwen3 embeddings, rerankers, and reward/PRM models, while retaining V1 for multimodal models such as ColQwen3, the Transformers backend, and the existing MoE/hybrid exclusions. The MTEB sweep also exposed that FA4's SM100 head-dim-256 encoder kernel cannot use split KV, so that exact configuration now uses a single split. ## Reproducer (inline) ```bash .venv/bin/python -c 'from types import SimpleNamespace as N; from vllm.config import VllmConfig; model=N(architectures=[\"Qwen3ForCausalLM\"], runner_type=\"pooling\", is_multimodal_model=False, using_transformers_backend=lambda: False, is_moe=False, is_quantized=False); print(VllmConfig._is_default_v2_model_runner_model(N(model_config=model)))' ``` ## Output on main / on branch (verbatim) ```text main: False enable-mrv2-pooling-default: True ``` ## Test Plan ```text .venv/bin/python -m pytest tests/test_config.py::test_is_default_v2_model_runner_model -q 22 passed, 14 warnings in 8.79s PATH=\"$PATH:$PWD/.venv/bin\" CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/models/language/pooling/test_all_pooling_plus_chunked_prefill.py::test_decoder_token_embed_model_runner_v2 tests/models/language/pooling/test_all_pooling_plus_chunked_prefill.py::test_sequence_embed_model_runner_v2 tests/models/language/pooling/test_jina_reranker_v3.py tests/models/language/pooling/test_reward.py::test_prm_models_with_golden_outputs -v 5 passed, 17 warnings in 118.85s PATH=\"$PATH:$PWD/.venv/bin\" CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/models/language/pooling/test_colbert.py::test_colbert_hf_comparison -v 4 passed PATH=\"$PATH:$PWD/.venv/bin\" CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/models/language/pooling_mteb_test/ -q 1 failed, 31 passed, 95 skipped, 49 warnings in 1665.22s # The full MTEB sweep exposed the FA4 SM100 split-KV issue fixed here. PATH=\"$PATH:$PWD/.venv/bin\" CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest 'tests/models/language/pooling_mteb_test/test_st_projector.py::test_embed_models_mteb[model_info1]' -q 1 passed, 19 warnings in 30.05s PATH=\"$PATH:$PWD/.venv/bin\" CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/entrypoints/pooling/ -q 311 passed, 37 warnings in 1782.18s .venv/bin/pre-commit run --files vllm/config/vllm.py tests/test_config.py vllm/v1/attention/backends/flash_attn.py All hooks passed ``` The full `tests/test_config.py` run also completed with 167 passes and one unrelated failure because this environment lacks access to gated `meta-llama/Llama-3.2-1B-Instruct` (HTTP 403). No new CI cost: no new test files or test functions were added; the existing lightweight gate parameterization was extended by one multimodal exclusion case. ## AI assistance disclosure OpenAI Codex (GPT-5) assisted with drafting the code.", "url": "https://github.com/vllm-project/vllm/pull/48290", "timestamp": "2026-08-12T16:45:53Z", "metrics": { "reactions": 3, "comments": 23 }, "labels": [ "ready", "mrv2" ], "author": "taneem-ibrahim", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:48556", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump astral-sh/setup-uv from 7.6.0 to 9.0.0", "text": "Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 9.0.0.
Release notes

Sourced from astral-sh/setup-uv's releases.

v9.0.0 🌈 Change prune-cache default to false

Changes

This release disables the default cache cache pruning to ease the load on the PyPi infrastructure. Since users might experience more GitHub Actions cache usage which might result in higher costs this is marked as a breaking change. To read more on why we did this (now) you can read the detailed analysis and reasoning in #967

Besides this big breaking change we also have a small bugfix while building caches for linux distributions that behave a big different than the "big ones" and a speed up in version resolution by only reading the version manifest until a matching version is found saving runtime and network bandwith.

🚨 Breaking changes

🐛 Bug fixes

  • fix: fall back to distribution ID when os-release has no version field @​cxzhong (#961)

🚀 Enhancements

🧰 Maintenance

📚 Documentation

⬆️ Dependency updates

v8.2.0 🌈 New inputs quiet and download-from-astral-mirror

Changes

This release brings two new inputs and a few bug fixes.

New inputs

Lets talk about the new inputs first.

quiet

Pretty simple. It turns of all info loggings. Useful if you use this in a composite action and are not interested in all the details. In the upcoming releases we will add log groups to fully implement support for "less noise"

[!NOTE]
Warnings and errors are always logged.

... (truncated)

Commits
  • c771a70 chore(deps): roll up Dependabot updates (#970)
  • 2f537ca chore: update known checksums for 0.11.30 (#968)
  • 2269552 Speed up version client by partial response reads (#807)
  • 47a7f4f Change prune-cache default to false (#967)
  • 71966ef chore(deps): roll up Dependabot updates (#962)
  • f12b1f0 fix: fall back to distribution ID when os-release has no version field (#961)
  • ecd24dd chore: update known checksums for 0.11.29 (#960)
  • 6a19136 docs: update version references to v8.3.2 (#949)
  • 11f9893 chore: roll up Dependabot updates (#948)
  • f798556 docs: update version references to v8.3.1 (#946)
  • Additional commits viewable in compare view

> **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days.", "url": "https://github.com/vllm-project/vllm/pull/48556", "createdAt": "2026-07-14T04:16:57Z", "updatedAt": "2026-08-13T16:41:44Z", "timestamp": "2026-08-13T16:41:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ci/build", "github_actions", "dependencies" ], "author": "dependabot[bot]", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48560", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump fsspec from 2024.12.0 to 2026.6.0", "text": "Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2024.12.0 to 2026.6.0.
Commits
  • a245700 changelog (#2049)
  • 652325d Fix async cat ranges on error (#2045)
  • 073aac8 Forward kwargs from cat_ranges to cat_file (#2044)
  • 2818611 FTP: preserve filenames containing whitespace in _mlsd2 (#2043)
  • c879384 Prevent attribute error by checking for 'forced' attribute before flushing ca...
  • 71b8a6b Reflect async _walk correctly (#2040)
  • 5eec9f9 Fix: Omit threading.get_ident() from cache token for async implementations ...
  • 8643878 Small safety improvements (#2039)
  • a1c16ab Honour simple_templates everywhere in referenceFS (#2029)
  • 85facaa Fix infinite recursion in expand_path when resolving paths with glob magic ch...
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fsspec&package-manager=pip&previous-version=2024.12.0&new-version=2026.6.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) You can trigger a rebase of this PR by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
> **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days.", "url": "https://github.com/vllm-project/vllm/pull/48560", "createdAt": "2026-07-14T04:58:48Z", "updatedAt": "2026-08-13T05:45:15Z", "timestamp": "2026-08-13T05:45:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ci/build", "nvidia", "dependencies" ], "author": "dependabot[bot]", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48666", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Gemma-4 FA4 FP8 Kernel", "text": " ## Purpose Gemma-4 uses 256-wide heads in `sliding_attention` and 512-wide heads in `full_attention`. On SM90, the full-attention layers upgrade from FA3 to the FA4 CuTeDSL kernel. This PR wires the FA4 FP8-KV-dequant path from [vllm-project/flash-attention#164](https://github.com/vllm-project/flash-attention/pull/164) into vLLM, allowing Gemma-4 to use FP8 KV cache across both FA3 sliding attention and FA4 full attention. It also preserves FP8 KV-cache scales for Gemma-4 MTP. The BF16 draft model uses Q-only attention over target-model KV caches; aliasing only the cache tensor left the draft attention modules with default K/V scales of `1.0`. The fix copies the target attention module's device and host K/V scales when wiring each shared cache. After this PR, when FLASH_ATTN is used, BF16 would need explicitly set version=4. FP8 should **not** explicitly set version=4, instead leave it unset or version=3. The reason is SWA FP8 layer should run with version=3, only FP8 full attention head dim 512 layer will automatically use FA4. ## Test Plan - **Accuracy:** `gemma-4-31B-it`, 4x H200, TP=1/DP=4, max model length 262144. Compare BF16 TRITON, FP8 TRITON, and FP8 FA4 with AIME 2025, GPQA Diamond, MMLU-Pro, and GSM8k. - **Serving performance:** 4x H200, TP=4, 25k shared prefix + 3k unique input tokens, OSL=200, concurrency 1-64. - **MTP:** 2x H200, TP=2, four-layer BF16 Gemma-4 assistant, three draft tokens, concurrency 32, and all 512 entries from the same SPEED-Bench dataset. Compare BF16 weights/BF16 KV against FP8 weights/FP8 KV with the scale-sharing fix. - `pre-commit run --files vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py` ### Raw MTP serve commands Set the checkpoint and dataset locations for your environment: ```bash export BF16_MODEL= export FP8_MODEL= export DRAFT_MODEL= export DATASET= ``` BF16 weights and BF16 KV: ```bash vllm serve \"$BF16_MODEL\" \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 2 \\ --max-num-seqs 64 --max-num-batched-tokens 8192 --max-model-len 8192 \\ --gpu-memory-utilization 0.9 --trust-remote-code \\ --async-scheduling --enable-prefix-caching \\ --attention-backend FLASH_ATTN --attention-config.flash_attn_version=4 \\ --max-cudagraph-capture-size 256 \\ --speculative-config \"{\\\"model\\\":\\\"$DRAFT_MODEL\\\",\\\"num_speculative_tokens\\\":3}\" \\ --compilation-config '{\"use_inductor_graph_partition\":1,\"pass_config\":{\"eliminate_noops\":true,\"fuse_allreduce_rms\":true}}' \\ --hf-overrides '{\"text_config\":{\"use_bidirectional_attention\":null}}' ``` FP8 weights and FP8 KV: ```bash vllm serve \"$FP8_MODEL\" \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 2 \\ --max-num-seqs 64 --max-num-batched-tokens 8192 --max-model-len 8192 \\ --gpu-memory-utilization 0.9 --trust-remote-code \\ --async-scheduling --enable-prefix-caching \\ --attention-backend FLASH_ATTN --kv-cache-dtype fp8 \\ --max-cudagraph-capture-size 256 \\ --speculative-config \"{\\\"model\\\":\\\"$DRAFT_MODEL\\\",\\\"num_speculative_tokens\\\":3}\" \\ --compilation-config '{\"use_inductor_graph_partition\":1,\"pass_config\":{\"eliminate_noops\":true,\"fuse_allreduce_rms\":true}}' \\ --hf-overrides '{\"text_config\":{\"use_bidirectional_attention\":null}}' ``` Both MTP runs used: ```bash aiperf profile \\ --model \"$MODEL\" --tokenizer \"$MODEL\" --tokenizer-trust-remote-code \\ --endpoint-type chat --endpoint /v1/chat/completions --streaming \\ --use-legacy-max-tokens --url http://localhost:8000 \\ --input-file \"$DATASET\" \\ --extra-inputs ignore_eos:true --extra-inputs skip_special_tokens:false \\ --use-server-token-count --concurrency 32 --request-count 512 \\ --num-dataset-entries 512 --random-seed 1234 ``` ## Test Result ### Accuracy | Benchmark | BF16 TRITON | FP8 TRITON | FP8 FA4 | |---|---:|---:|---:| | AIME 2025 (avg@64) | 87.29 | 87.14 | 86.88 | | GPQA Diamond (avg@16) | 85.42 | 85.39 | 86.55 | | MMLU-Pro | 86.44 | 86.25 | 86.64 | | GSM8k (flexible-extract) | 94.62 | 94.62 | 94.47 | All configurations are within 1.2 percentage points on every benchmark. ### Serving performance Decode throughput in output tokens/s/GPU: | Concurrency | FP8 FA4 | BF16 FA4 | FP8 TRITON | BF16 TRITON | |---:|---:|---:|---:|---:| | 1 | 27.3 | 24.3 | 17.9 | 15.0 | | 16 | 186.5 | 168.4 | 105.0 | 102.5 | | 64 | 261.6 | 241.1 | 118.5 | 127.5 | FP8 FA4 is 8-12% faster than BF16 FA4 and up to 2.21x faster than FP8 TRITON across this sweep. ### MTP speculative decoding Both runs completed 512/512 requests without errors. | Configuration | Mean acceptance length | Draft-token acceptance | Output tok/s | Avg ITL (ms) | |---|---:|---:|---:|---:| | BF16 weights + BF16 KV | 3.6543 | 88.48% | 4,370.34 | 7.13 | | FP8 weights + FP8 KV | 3.6370 | 87.90% | 4,693.33 | 6.64 | The fixed FP8 run is within 0.0173 accepted tokens and 0.58 percentage points of the BF16 acceptance baseline while delivering 7.4% higher output throughput. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/48666", "createdAt": "2026-07-14T23:28:39Z", "updatedAt": "2026-08-13T03:26:49Z", "timestamp": "2026-08-13T03:26:49Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "documentation", "speculative-decoding", "ci/build", "v1", "mrv2" ], "author": "jhaotingc", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48672", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: reject required/named tool_choice for Harmony models without a tool parser", "text": "### Summary In the OpenAI-compatible chat path, when no tool-call parser is configured, `tool_choice=\"required\"` and named tool choices were accepted for Harmony (gpt-oss) models and then produced empty or incorrect tool-call results. ### Fix Only exempt Harmony for `tool_choice=\"auto\"` (best effort). Named and `required` now fail closed with a 400 when no tool parser is available. Adds a regression test. Fixes #48207", "url": "https://github.com/vllm-project/vllm/pull/48672", "createdAt": "2026-07-14T23:55:39Z", "updatedAt": "2026-08-13T03:54:48Z", "timestamp": "2026-08-13T03:54:48Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "gpt-oss" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48673", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "test(quantization): cover tied lm_head/embed_tokens when lm_head excluded from ModelOpt", "text": "### Summary Adds regression coverage for tying an excluded `lm_head` to `embed_tokens` through the quantization method path (`QuantizeMethodBase.tie_weights`), the behavior that fixed the original NVFP4 crash on main. Test only. Related to #48238", "url": "https://github.com/vllm-project/vllm/pull/48673", "createdAt": "2026-07-14T23:59:09Z", "updatedAt": "2026-08-13T14:47:20Z", "timestamp": "2026-08-13T14:47:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48684", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Remove `override_attention_dtype`", "text": "## Purpose `override_attention_dtype` is only used for V0 and has been removd from https://github.com/vllm-project/vllm/pull/25351/ long time ago. It's safe to remove it now. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48684", "createdAt": "2026-07-15T02:47:30Z", "updatedAt": "2026-08-13T15:35:29Z", "timestamp": "2026-08-13T15:35:29Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "ready" ], "author": "wangxiyuan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48693", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Make Gemma 4 suppress-token masking CUDA-graph safe", "text": "## Purpose Gemma 4 CUDA graph capture used Python suppress-token IDs and advanced indexed assignment with a Python scalar. Both can introduce host-to-device work during capture. Store validated IDs as nonpersistent model buffers and use `index_fill_` for graph-safe masking in the target, unified, and MTP paths. The centroid MTP proposer also returned sparse token IDs without applying the same suppression policy as full logits. Sparse argmax now masks those IDs and, if every selected candidate is suppressed, returns the same lowest eligible fallback token that full-vocabulary argmax selects. Invalid dtype/range values and suppression of the entire vocabulary fail at model initialization. Addresses #48503. Use `Fixes #48503` after the exact affected-model evaluation below passes. #48509 covered only part of the failure; #48515 closed unmerged when its source repository disappeared. Mandatory duplicate searches found no competing open PR. AI assistance was used. The human submitter must review every changed line and the recorded model result before readiness. ## Validation Current head `79d3a4c7f` is based on upstream `1a659a0c3`. ```text pytest -q tests/model_executor/test_gemma4_suppress_tokens.py 29 passed, 3 skipped # local host has no CUDA pre-commit run --files all applicable hooks passed, including mypy git diff --check passed ``` The suite covers direct dense masking, sparse/full argmax parity for none, partial, and all-selected suppression, invalid types and bounds, scalar zero, empty values, nonpersistent registration, actual LM-head placement, module and meta-device movement, and separate dense and sparse CUDA-graph capture/replay tests. On an earlier H100 revision with the same dense masking mechanism, `Gemma4MTP.compute_logits` captured and replayed twice successfully and the then-current focused suite passed. The newly strengthened sparse/fallback CUDA tests exercise both partial and all-selected suppression, but have not yet run on H100, so that historical result is not claimed as a current-head GPU pass. Still required before readiness: - Run the complete current-head focused suite on H100. - Start and generate under CUDA graphs with `google/gemma-4-12B-it-qat-w4a16-ct` and the affected Gemma 4 MTP assistant. - Compare eager and graph greedy token IDs or run the repository-appropriate deterministic model evaluation because this changes proposal/logit output. - Human review and reviewer-triggered full CI. ## Release note Gemma 4 with MTP speculative decoding no longer performs suppress-token host-to-device copies during CUDA graph capture, and sparse centroid proposals honor the same validated suppression policy as full logits. After affected-model evaluation and mainline merge, this crash/correctness fix is a v0.26 backport candidate.", "url": "https://github.com/vllm-project/vllm/pull/48693", "createdAt": "2026-07-15T06:22:39Z", "updatedAt": "2026-08-12T23:22:17Z", "timestamp": "2026-08-12T23:22:17Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "speculative-decoding", "needs-rebase", "v1", "nvidia" ], "author": "dumko2001", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48712", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Only run FP8 AITER MLA prefill when using FP8 KV", "text": "## Purpose Even when KV is not FP8, the FP8 PS ASM prefill kernel's persistent buffers are both (a) initialized/reserved and (b) filled with metadata (through `_build_fp8_prefill_ps_metadata`) in every forward. Specifically the latter leads to performance degradation when using BF16 as `_build_fp8_prefill_ps_metadata` is a host-side function with several sync memcpys in it, causing a unnecessary H->D sync before every forward. Co-authored with @olka-amd. ### Before: big bubble in start of every forward, even when using BF16 \"image\" ### After: bubble gone \"image\" ## Test Plan 1. Runs as expected on gsm8k with BF16 KV ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` ``` lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result This branch: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9466|± |0.0065| | | |strict-match | 5|exact_match|↑ |0.9469|± |0.006| Nightly: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48712", "createdAt": "2026-07-15T07:24:10Z", "updatedAt": "2026-08-13T12:20:28Z", "timestamp": "2026-08-13T12:20:28Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "bug", "rocm", "v1" ], "author": "simondanielsson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48761", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[WIP][XPU][Test]add xpu yaml", "text": "## Purpose only for xpu UT, please do not merge ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48761", "createdAt": "2026-07-15T17:34:17Z", "updatedAt": "2026-08-13T03:29:48Z", "timestamp": "2026-08-13T03:29:48Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "intel-gpu", "ci/build", "nvidia", "rust" ], "author": "wincent8", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48768", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add Inkling multi-depth MTP support [5/N]", "text": "## Summary This draft now contains the multi-depth MTP work that remains after the following model slices landed: - #48799: core Inkling model and frontend support - #48822: breakable PIECEWISE CUDA graph support - #48858: Hopper FA4 relative attention - #48869: CUDA-graph-compatible MTP=1 support Additional independent changes were extracted into draft PRs: - #48880: select the NVSHMEM symmetric-memory backend for multi-node groups - #48881: show rank-zero CuTeDSL warmup compilation progress - #48884: Inkling LoRA support The standalone QKVR preparation benchmark was removed. The remaining model changes: - Extend the merged MTP=1 implementation to multiple checkpoint depths. - Add the multi-step speculative prefill, verification, and KV-cache plumbing required by MTP>1. - Retain the checked Rust multimodal placeholder-token conversion. The core model, LoRA support, MTP=1 foundation, breakable CUDA graphs, and Hopper attention support are no longer introduced by this diff. ## Duplicate-work check This does not duplicate another open PR. It is rebased directly onto current `main` and builds on the merged slices #48799, #48822, #48858, and #48869. The changes shown here are specifically the delta from one draft depth to multi-depth MTP. LoRA is isolated in #48884, and the unrelated infrastructure changes are isolated in #48880 and #48881. ## Validation - `.venv/bin/python -m pytest tests/config/test_speculative_draft_hf_overrides.py tests/models/inkling/test_mtp_input_fusion.py -v`: 38 passed - `git diff --name-only -z origin/main...HEAD | xargs -0 .venv/bin/pre-commit run --files`: passed, including Python 3.10 mypy and Rust formatting - `git diff --check origin/main...HEAD`: passed ## Model evaluations The source PR reports the following evaluations for the complete Inkling implementation. These were not rerun while extracting the multi-depth MTP delta. | Benchmark / metric | vLLM NVFP4 | Reference NVFP4 | Delta vs Reference | | --- | ---: | ---: | ---: | | MMAU overall | 76.10% (761/1,000) | 75.50% | +0.60 pp | | BFCL exact calls | 78.61% (1,062/1,351) | 78.16% | +0.45 pp | | BFCL All-Live macro | 75.86% | 73.54% | +2.32 pp | | MMMU-Pro overall micro | 71.12% (3,691/5,190) | 70.52% (3,660/5,190) | +0.60 pp | | MMMU-Pro Standard 10-option | 70.23% (1,215/1,730) | 70.00% (1,211/1,730) | +0.23 pp | | MMMU-Pro Standard 4-option | 76.47% (1,323/1,730) | 76.30% (1,320/1,730) | +0.17 pp | | MMMU-Pro Vision | 66.65% (1,153/1,730) | 65.26% (1,129/1,730) | +1.39 pp | | HLE | 29.33% (633/2,158) | 26.65% | +2.68 pp | | NIAH (2K-221K) | 99.09% (436/440) | 99.09% (436/440) | 0.00 pp | | NIAH (294K-513K) | 95.68% (421/440) | 96.82% (426/440) | -1.14 pp | | NIAH (586K-805K) | 81.36% (358/440) | 84.09% (370/440) | -2.73 pp | | NIAH (878K) | 70.91% (78/110) | 80.91% (89/110) | -10.00 pp | ## AI assistance AI assistance (OpenAI Codex) was used to split and rebase the original branch, preserve contributor attribution, isolate LoRA from multi-depth MTP, run validation, and update this draft PR. The human submitter must review every changed line, understand the change end-to-end, and confirm the model evaluations before marking this ready for review.", "url": "https://github.com/vllm-project/vllm/pull/48768", "timestamp": "2026-08-12T16:45:26Z", "metrics": { "reactions": 14, "comments": 8 }, "labels": [ "performance", "new-model", "needs-rebase", "ci/build", "v1", "tool-calling", "nvidia", "mrv2", "rust" ], "author": "WoosukKwon", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:48804", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode][V1] Warm Eagle and DFlash/DSpark spec-decode Triton kernels at startup", "text": "## Summary Triton specializes integer arguments whose runtime value is 1 into compile-time constants, producing a separate cubin per \"which params are 1\" combination. Triton also specializes `tl.constexpr` arguments (one cubin per distinct value) and `tl.num_programs` grid axes when they equal 1. Without warmup, the first request in each shape pays a JIT latency spike. This PR adds two standalone warmup modules under `vllm/model_executor/warmup/` that enumerate all relevant parameter combinations for the spec-decode Triton kernels: ### Eagle warmup (`eagle_spec_decode_warmup.py`) 1. **`eagle_prepare_next_token_padded_kernel`** — 2³ int combos × len(BLOCK_SIZES) cache entries 2. **`eagle_prepare_inputs_padded_kernel`** — 2 entries (single-req vs multi-req) 3. **`_mtp_shared_head_rmsnorm_kernel`** — 1 entry (only `tl.constexpr` params, grid dim is not specialized) 4. **`eagle_step_slot_mapping_metadata_kernel`** — len(n_blocks_candidates) × 2² int combos, covering CP × hybrid block factors No-op when Eagle spec decoding is not configured (`num_speculative_tokens` is `None` or 0) or when not on CUDA. ### DFlash/DSpark warmup (`dflash_spec_decode_warmup.py`) 5. **`_prepare_dflash_inputs_kernel`** — shared by the DFlash and DSpark speculators. Its cache key is driven by: - `BLOCK_SIZE` constexpr — computed at runtime as `min(256, next_power_of_2(max_tokens_per_req))`, so it varies with batch composition (small for pure decode, 256 for prefill chunks). This is the main source of multiple cubins; without warmup the first prefill-heavy request pays a JIT latency spike (the reported JIT warning references `BLOCK_SIZE=256, SAMPLE_FROM_ANCHOR=True` for DSpark). - `SAMPLE_FROM_ANCHOR` constexpr — `False` for DFlash, `True` for DSpark. - `PAD_SLOT_ID` constexpr — always `-1`. - Grid `(num_reqs, num_blocks)` — `tl.num_programs` is specialized when an axis equals 1. The remaining i32 scalars (`block_size`, `block_table_stride`, `num_speculative_steps`, `num_query_per_req`, `parallel_drafting_token_id`, `max_num_reqs`, `max_num_tokens`, `max_model_len`) are constant per deployment, so Triton only specializes them when their value happens to be 1 — passing the configured value covers both branches automatically. The warmup enumerates `BLOCK_SIZE` powers of two `{1,2,4,8,16,32,64,128,256}` crossed with the four grid axis==1 combinations `(1,1)/(1,8)/(50,1)/(50,8)` = 36 entries, reading deployment-fixed values from the live `DFlashSpeculator` (covers `DSparkSpeculator` via subclassing). No-op when DFlash/DSpark is not configured. ## Design Both warmups are called from the central `kernel_warmup()` dispatcher in `kernel_warmup.py`, each wrapped in its own `try/except` so a failure does not prevent other warmups from running. Each kernel invocation inside the modules is also individually wrapped in `try/except` so a single shape failure does not abort the rest. Deployment-fixed values are read from the live speculator so the warmup matches the runtime specialization exactly. ## Supersedes #48393 #48393 embedded the warmup logic inside `LLMBaseProposer.dry_run_helper_kernels()` and was stacked on #41481 (still open). This PR takes a different approach: - **Standalone module** under `vllm/model_executor/warmup/` instead of proposer class methods - **No dependency on #41481** — imports kernels directly from `vllm.v1.spec_decode.utils` and `vllm.v1.worker.gpu.spec_decode.dflash.speculator` - **Broader coverage** — also warms `eagle_step_slot_mapping_metadata_kernel` (not in #48393) and adds the DFlash/DSpark warmup - **Central dispatch** — called from `kernel_warmup()` alongside other warmups, consistent with mHC / sparse-MLA / DeepGEMM patterns ## Test commands ```bash .venv/bin/python -m pytest tests/v1/spec_decode/test_helper_kernel_warmup.py -v ``` DFlash/DSpark model evals should be run to confirm the `jit_monitor` no longer reports `_prepare_dflash_inputs_kernel` JIT compilation during inference (the reported warning referenced `BLOCK_SIZE=256, SAMPLE_FROM_ANCHOR=True` for DSpark). ## Pre-commit `ruff check`, `ruff format`, `mypy`, `check-torch-cuda-call`, `check-spdx-headers`, `check-forbidden-imports`, and `signoff-commit` all pass on the new files. ## AI assistance This PR was prepared with AI assistance (opencode). Every changed line was reviewed by a human.", "url": "https://github.com/vllm-project/vllm/pull/48804", "createdAt": "2026-07-16T03:42:03Z", "updatedAt": "2026-08-13T01:17:04Z", "timestamp": "2026-08-13T01:17:04Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "speculative-decoding", "needs-rebase", "v1", "cpu", "mrv2" ], "author": "leihuang-sketch", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48857", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][Triton] Reassociate softmax scale onto Q in unified attention", "text": "## Purpose Reassociate the softmax scale so ``score_scale * Q`` is loop-invariant, letting the compiler hoist the multiply out of the tile loop instead of scaling the BLOCK_M x TILE_SIZE S matrix on every iteration. Skip the reassociation for the fp8 query descale path, where folding the tiny score_scale into fp8 Q would requantize the product and destroy precision. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48857", "createdAt": "2026-07-16T16:18:46Z", "updatedAt": "2026-08-12T21:05:33Z", "timestamp": "2026-08-12T21:05:33Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "v1" ], "author": "quinnlp", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:48870", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Quant] Fused WNA16 GEMM for tied quantized lm_head logits", "text": "## Purpose Follow-up to #45535 (compressed-tensors WNA16 input embeddings + tied `lm_head`). This PR speeds up the **logits matmul** when a quantized embedding is reused as a tied `lm_head`. In #45535, `CompressedTensorsEmbeddingWNA16Int.apply()` dequantizes the entire packed `[vocab, hidden]` table into a dense weight on every call and runs `F.linear`. This PR routes that matmul through vLLM's existing WNA16 **Linear** kernel (Marlin/Machete) instead — a fused dequant-GEMM that never materializes the dense weight. Key insight: the embedding's packed weight is **already** in the compressed-tensors layout those kernels expect (packed along the input/hidden dim, identical to a WNA16 Linear), so `process_weights_after_loading` sets up the fused kernel directly. It's set up **only** for tied embeddings (flagged in `tie_weights`), keeping a gather-format copy for the input-lookup path. When no fused kernel is available it falls back to a full-table dequant (no index tensor) + `F.linear`. > [!NOTE] > **Stacked on #45535** — the tied-embedding `apply()` this optimizes only exists there, so the net-new change here is `compressed_tensors_embedding.py`; the rest of the diff belongs to #45535. Please review/merge after #45535. Kept as a separate PR to avoid growing the (already large) plumbing PR. **Not a duplicate:** quantized-`lm_head`-via-Marlin already exists for the *untied / separately quantized* case — a `ParallelLMHead` independently quantized as a GPTQ/AWQ/compressed-tensors **Linear** already dispatches to Marlin (cf. #40999). This PR covers the distinct *tied* case, where the embedding **is** the `lm_head` and its quant method is the embedding method (`CompressedTensorsEmbeddingWNA16Int`), not a Linear method. The open ModelOpt lm_head/embedding PRs (#35660, #42791, #44671, #41000) target a different backend (NVFP4/FP8 in `modelopt.py`) and address loading/dispatch, not this fused tied-logits path. No open PR touches `compressed_tensors_embedding.py`. ## Correctness The fused path matches the dequant reference numerically (max relative diff **0.025%**, fp16). The existing tied fixture test (`tests/quantization/test_quantized_embedding.py::test_tied_quantized_embedding`) exercises the fused Marlin path end-to-end and generates coherently. ## Perf evals RTX 5080 (sm120, CUDA 13), `apply()` microbenchmark, µs/call (lower is better): | dims | M | current (dequant + `F.linear`) | no-gather dequant | **fused (Marlin)** | fused vs current | | --- | --- | --- | --- | --- | --- | | vocab 151936 × 1024, W4-g64 | 1 | 1329 | 1317 | **240** | **5.5×** | | | 32 | 1703 | 2395 | **619** | **2.8×** | | | 256 | 3247 | 2460 | **2266** | 1.4× | | vocab 151936 × 1024, W8-g128 | 1 | 2859 | 1374 | **329** | **8.7×** | | | 32 | 1442 | 575 | **575** | 2.5× | | | 256 | 2971 | 2088 | **2088** | 1.4× | | vocab 50304 × 512, W4-g64 | 1 | 141 | 142 | **19** | **7.6×** | | | 32 | 192 | 190 | **123** | 1.6× | Biggest win at decode (`M=1`), the common serving case for logits. The no-gather dequant (kept as the fallback path) is marginal versus the current dequant, as expected — the fused GEMM is the real lever. ## Test Plan ``` pytest tests/quantization/test_quantized_embedding.py ``` ## Test Result - Both fixture tests pass on an RTX 5080; the tied test confirms dispatch to `MarlinLinearKernel` for the logits path (verified `logits_kernel` is set and gather copies are preserved). - `pre-commit` (ruff + mypy) clean on the changed file. --- This change was developed with AI assistance (Claude Code). All changed lines were reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/48870", "createdAt": "2026-07-16T18:13:30Z", "updatedAt": "2026-08-13T16:48:00Z", "timestamp": "2026-08-13T16:48:00Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "speculative-decoding", "needs-rebase", "llama", "qwen", "deepseek", "gpt-oss", "quantization", "mistral", "kimi", "cohere" ], "author": "KKothuri", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48882", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Connector] Validate HMA children for MultiConnector subclasses", "text": "## Summary - Validate HMA support from child connectors for every `MultiConnector` subclass. - Add auto-configuration coverage for custom wrapper subclasses with all-HMA and mixed-HMA child configurations. ## Root cause `KVConnectorFactory.supports_hma_config()` special-cased the literal connector name `\"MultiConnector\"`. A subclass inherited `SupportsHMA`, so it passed the plain-connector check without validating its children. A mixed child configuration could therefore leave HMA enabled and fail later during connector initialization. ## Impact Custom `MultiConnector` wrappers now have the same effective HMA contract as the built-in wrapper: HMA remains enabled only when every configured child supports it. ## Validation - `.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_hma_auto_config.py -v` — 7 passed - `.venv/bin/ruff check vllm/distributed/kv_transfer/kv_connector/factory.py tests/v1/kv_connector/unit/test_hma_auto_config.py` — passed ## Duplicate-work check This is a narrow follow-up to #41847. No open PR was found for HMA validation of `MultiConnector` subclasses; the broad hybrid-offload work in #38261 does not cover this factory classification bug. ## AI assistance AI assistance was used to implement and validate this change.", "url": "https://github.com/vllm-project/vllm/pull/48882", "timestamp": "2026-08-12T16:27:05Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "v1", "kv-connector" ], "author": "jthomson04", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:48915", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses", "text": "## Purpose Speculative decoding guesses several tokens ahead, then checks them. Some guesses get accepted, some don't. How many get accepted tells you how well spec decode is working. Today you can only see this as a server-wide average on the `/metrics` page. You can't tell how any single request did. Tools like AIPerf want that per-request number, and right now they'd have to wrap the engine to get it. This PR puts the numbers right in the response. It's off by default; you turn it on at server start with `--per-request-spec-decode-stats summary` (or `detailed`). When it's off, the response is unchanged. The stats go on each choice: ```json { \"choices\": [ { \"index\": 0, \"message\": {\"role\": \"assistant\", \"content\": \"...\"}, \"finish_reason\": \"stop\", \"speculative_decoding_stats\": { \"mean_acceptance_length\": 2.5, \"draft_acceptance_rate\": 0.5, \"acceptance_histogram\": {\"0\": 8, \"2\": 6, \"3\": 6}, \"num_spec_steps\": 20, \"num_accepted_draft_tokens\": 30, \"num_draft_tokens\": 60, \"num_spec_tokens\": 3 } } ] } ``` `summary` gives the fields above. `detailed` also adds `per_step_accepted` and `per_step_drafted` arrays, one entry per step. This builds on #47322, which does the same job with a different shape. That PR attaches one stats object to the response as a whole, taken from the first sequence, and leads with per-step arrays instead of a histogram. Putting the stats on each choice matters when you ask for `n>1`: every choice reports its own acceptance, instead of a single blob that only reflects one of them. ## Relationship to prior work This builds directly on @reed-meyerson's #47322 — it's an adaptation and continuation of their work, and thanks to them for the original design and groundwork. See the Purpose section above for how the shape differs (per-choice + histogram vs one response-level blob + per-step arrays). The closest other relative is #43310, so to be clear on why this isn't a duplicate of it: #43310 also returns per-request spec-decode counts in the response, but on a different surface for a different user. It targets vLLM's native `/inference/v1/generate` endpoint, attaches raw counts at the request level, and is aimed at RL rollout tools (verl). This PR targets the OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`), puts stats per choice so `n>1` works, and adds derived numbers like mean acceptance length and the histogram. The two don't overlap in endpoint, placement, or consumer. A few other efforts are nearby but doing something else: - RFC #48202 (with impls #48204 and #48692) is about *changing* how many tokens get drafted per request — adaptive, variable-length drafting. This PR doesn't change any behavior. It just reports what happened. And `detailed` already records how many tokens were drafted each step, so it keeps working if drafting becomes variable. - #44487 adds a per-request acceptance histogram to the `/metrics` Prometheus page. That's server-side scraping, not fields in the response body. ## Test Plan Unit: ``` .venv/bin/python -m pytest tests/v1/spec_decode/test_request_acceptance.py -v .venv/bin/python -m pytest tests/v1/core/test_scheduler.py -k spec_decod -v ``` E2E (H100, n-gram spec decode, no draft weights required): ``` vllm serve Qwen/Qwen2.5-0.5B-Instruct --port 8000 \\ --speculative-config '{\"method\":\"ngram\",\"num_speculative_tokens\":3,\"prompt_lookup_max\":3,\"prompt_lookup_min\":1}' \\ --per-request-spec-decode-stats detailed --max-model-len 4096 ``` - Non-streaming chat: `choices[0].speculative_decoding_stats` populated. - `n=2` completion: each `choices[i]` carries independent stats. - Streaming: stats present only on the terminal chunk's choice. - `--per-request-spec-decode-stats summary`: `per_step_*` arrays absent; default `none`: field is `null` on every choice. - Reconciliation: summed per-request fields vs the aggregate `/metrics` spec-decode counters on a fresh server. ## Test Result Unit: `11 passed` (accumulator math, summary/detailed `to_dict`, msgspec round-trip, `CompletionOutput` field). Scheduler integration (`tests/v1/core/test_scheduler.py -k spec_decod`): passed — the per-request accumulator is asserted alongside the aggregate stats in `test_schedule_spec_decoding_stats`, plus the `detailed` per-step recording, invalid-draft subtraction, and off-by-default cases. E2E on NVIDIA H100 80GB (CUDA 13.2): all checks passed — per-choice emission, independent `n=2` stats, streaming terminal-chunk placement, `detailed`→`summary` gating, and default-`none` omitting the field. Representative `detailed` choice (n-gram, `num_speculative_tokens=3`): ```json {\"mean_acceptance_length\":1.2325581395348837,\"draft_acceptance_rate\":0.07751937984496124,\"acceptance_histogram\":{\"0\":39,\"1\":1,\"3\":3},\"num_spec_steps\":43,\"num_accepted_draft_tokens\":10,\"num_draft_tokens\":129,\"num_spec_tokens\":3,\"per_step_accepted\":[3,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0],\"per_step_drafted\":[3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3]} ``` Self-consistent: histogram counts sum to `num_spec_steps` (43); `sum(j·count) = num_accepted_draft_tokens` (10); `mean_acceptance_length = 1 + 10/43`; `num_draft_tokens = 43×3`. Prometheus reconciliation (fresh server, summed across all choices of all requests vs the aggregate counters): | Per-request field (summed) | `/metrics` counter | Result | | --- | --- | --- | | `num_spec_steps` | `vllm:spec_decode_num_drafts_total` | 134 = 134 | | `num_draft_tokens` | `vllm:spec_decode_num_draft_tokens_total` | 402 = 402 | | `num_accepted_draft_tokens` | `vllm:spec_decode_num_accepted_tokens_total` | 126 = 126 | ## Model evaluation Purely observational — no change to sampling, scheduling, or generated tokens. Output is byte-identical across `none`/`summary`/`detailed`. No accuracy eval applicable. ## AI assistance AI assistance (Claude) was used for implementation and test authoring. All changed lines were reviewed by the submitting human, and the tests above were run and verified.", "url": "https://github.com/vllm-project/vllm/pull/48915", "createdAt": "2026-07-17T05:17:50Z", "updatedAt": "2026-08-12T22:29:13Z", "timestamp": "2026-08-12T22:29:13Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "documentation", "frontend", "speculative-decoding", "ready", "needs-rebase", "v1", "verified" ], "author": "matthewkotila", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48922", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Guard tool call argument JSON parsing in chat message postprocessing", "text": "## Purpose ​ Fixes #47761. ​ `_postprocess_messages` in `vllm/entrypoints/chat_utils.py` calls `json.loads()` on the assistant tool-call `arguments` string that clients replay in conversation history, with no error handling: ​ ```python if content := function.get(\"arguments\"): if not isinstance(content, (dict, list)): parsed = json.loads(content) # raises on malformed input ``` ​ When that string is not valid JSON, the `JSONDecodeError` propagates out of `parse_chat_messages_async` and the request fails with an opaque `400 Unterminated string starting at: line 1 column N`. ​ Because the malformed string lives in **conversation history**, the failure is sticky. Every subsequent turn replays the same message and gets the same 400, so the session is unrecoverable rather than degraded, and the error message gives the user nothing to act on. ​ ### How the malformed string gets there ​ Two independent reports in #47761 reach this line by different routes: ​ | Reporter | Surface | Upstream cause | Payload | | --- | --- | --- | --- | | @rnogueira-tech | Chat Completions, Qwen3.6-35B-A3B-FP8, `qwen3_coder` | `max_tokens` cap hit mid-string; `finish_reason` still reported as `tool_calls`, so the client persisted the partial call | 18,356 chars, fails at char 13 | | @iMMIQ | Responses API, non-Harmony Simple path, DeepSeek-V4-Flash | Generation stopped at EOS with the string still open; `emit_simple_tool_call_done` wrote `state.accumulated_text` into `ResponseFunctionToolCall.arguments` with `status: \"completed\"` and no validation | 54 chars, 297 output tokens, fails at char 8 | ​ In both cases the reported offset is exactly the index of the opening quote of the value that never closes. Both were observed on 0.24.0 and 0.25.1. ​ Ruled out during investigation: ​ - **Not the tool parser.** `qwen3_coder`, `qwen3_xml` and `mimo` all resolve to `Qwen3EngineToolParser`, and `_qwen3_arg_converter` in `vllm/parser/qwen3.py` returns `json.dumps(params, ensure_ascii=False)` on both the full and partial paths, so it cannot emit invalid JSON. - **Not client mis-escaping.** Reconstructing the arguments from the SSE deltas and diffing against what the client replayed gives an identical 18,356 bytes, md5 `d1be3623820564162172c1f5ebff5893`. - **Not a config or version issue.** The reporter tried `hermes` and `qwen3_xml` parsers, patched chat templates, disabling the reasoning parser, and non-streaming mode; all still fail. Raising `max_tokens` to 8192 reproduces at the new cap, and 16384 overflows the 32768 context window. ​ ### Scope of this PR ​ This PR deliberately does not fix either upstream cause. Neither cause is fixable in one place: a `finish_reason` fix does nothing for the EOS case, and Responses-path validation does nothing for the Chat Completions case. `_postprocess_messages` is the only point the two have in common, and it is where vLLM turns data it emitted itself into an unhandled exception. ​ Two follow-ups will be filed separately so this PR stays single-purpose: ​ 1. Report `finish_reason: \"length\"` when a tool call is cut off by the token cap, so clients can discard the partial call instead of persisting it. 2. Validate JSON in `emit_simple_tool_call_done` on the Responses Simple path, and emit the done event that is currently missing for that item. ​ ### Design note ​ On the failure path this keeps the original string and logs a warning, rather than substituting `{}` or attempting a repair. ​ The tradeoff is real and worth stating. Passing through a truncated 18k-char file body preserves useful context. Passing through something like `{\"cmd\": \"mkdir -p /tmp/x && cat > /tmp/x` hands a half-written shell command back to the model as a completed `exec` call, which is arguably worse than an empty object. ​ Raw passthrough is chosen because it is tool-agnostic and non-destructive, and because substituting `{}` silently discards data in the much larger truncation case. This is a one-line change if reviewers prefer `{}` or a best-effort repair; say which and I will push it. ​ ## Test Plan ​ **Unit tests** — `tests/entrypoints/test_chat_utils_tool_call_arguments.py` (new): ​ ``` pytest tests/entrypoints/test_chat_utils_tool_call_arguments.py -q ``` ​ Cases covered: ​ - valid JSON object string is parsed to a dict (unchanged behaviour) - `arguments` already a dict or list is passed through untouched (unchanged behaviour) - absent / empty `arguments` becomes `{}` (unchanged behaviour) - truncated payload in the shape reported by @rnogueira-tech no longer raises - 54-byte `exec` payload reported by @iMMIQ no longer raises - a warning is logged on the failure path ​ **Manual reproducer** — contributed by @iMMIQ in #47761. No GPU work and no generation is involved, because `_postprocess_messages` runs before sampling, so `max_tokens: 1` against any served model is enough: ​ ```bash curl -sS http://:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{ \"model\": \"\", \"max_tokens\": 1, \"messages\": [ {\"role\": \"user\", \"content\": \"hi\"}, {\"role\": \"assistant\", \"content\": null, \"tool_calls\": [ {\"id\": \"c1\", \"type\": \"function\", \"function\": {\"name\": \"foo\", \"arguments\": \"{\\\"cmd\\\": \\\"mkdir -p /tmp/x\"}} ]} ] }' ``` ​ On `main` this returns: ​ ```json {\"error\":{\"message\":\"Unterminated string starting at: line 1 column 9 (char 8)\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}} ``` ​ The equivalent request against `POST /v1/responses` with a `function_call` input item fails the same way. The tool name is irrelevant. ​ ## Test Result ​ REPLACE THIS LINE with the real output of `pytest tests/entrypoints/test_chat_utils_tool_call_arguments.py -q`, pasted inside a code fence. ​ REPLACE THIS LINE with the curl result only if you actually ran it against a server. Otherwise delete this line and leave the reproducer as reported by @iMMIQ in the issue. ​ ## (Optional) Documentation Update ​ None required. Behaviour change is limited to no longer raising on malformed input that vLLM previously accepted into history. ​", "url": "https://github.com/vllm-project/vllm/pull/48922", "createdAt": "2026-07-17T07:15:19Z", "updatedAt": "2026-08-12T17:54:22Z", "timestamp": "2026-08-12T17:54:22Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "frontend" ], "author": "VBS2004", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:48944", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Context-length-aware K in DSD (RFC #48627): extend num_speculative_tokens_per_batch_size with a ctx axis", "text": "## What this PR does Extends `num_speculative_tokens_per_batch_size` (Dynamic SD, #32374) with an optional ctx-length axis, so that a schedule can select K based on both batch size and per-request context length instead of batch size alone. The change is a backward-compatible schema extension — a 3-item entry `[bs_lo, bs_hi, K]` parses and behaves exactly as before, a 5-item entry `[bs_lo, bs_hi, ctx_lo, ctx_hi, K]` opts in to the ctx axis. ## Why Under memory-bound decode with a shared prefix (RAG serving, agentic sessions, batched summarization), the optimal K depends on ctx as well as batch. A batch-only lookup has to pick one K per BS, either leaving ctx-amortization on the table for long sequences or over-drafting short ones. This PR gives operators a per-(bs, ctx) K table that opts in to this axis when it's useful and stays inert (3-item form) otherwise. The concept was previously formalized by **MagicDec (Sadhukhan et al., arXiv:2408.11049, 2024)** — *\"Leveraging our observation that there is a sequence length threshold above which decoding becomes memory bound, we propose choosing the amount of speculation as a function of the sequence length (longer sequence length → more speculated tokens).\"* MagicDec's instantiation was self-speculation with StreamingLLM sparse KV in a batch-centric framing; this PR is the general draft-target integration into vLLM's DSD lookup surface. See RFC #48627 for the design discussion. ## Evidence (2026-07-27 measurement) Position-balanced 2-trial, `prefix_repetition` c=256, H100 NVL 94GB, `prithivMLmods/gemma-4-31B-it-qat-FP8` + assistant draft, vLLM `c5d967c23`. Full methodology + per-cell data in [decomposition comment](https://github.com/vllm-project/vllm/pull/48944#issuecomment-5091663057). Per-cell stdev <2%, order-bias max 1.72% (signal 29-36× larger). **Primary contrast — C′ (6-cell 2D schedule) vs A′ (3-item batch-only), same DSD-mode cost:** | ctx | A′ | C′ | C′/A′ | |-----:|---:|---:|:-:| | 400 | 1875.6 | 1890.7 | 1.01× (K=0 tier, tie by design) | | 900 | 1453.5 | 1874.6 | **1.29×** | | 1900 | 1416.6 | 1848.2 | **1.30×** | | 4000 | 1232.8 | 1680.4 | **1.36×** | **Zero cost when unused:** the original A/B comparison in the PR body (retracted headline, but this specific check reproduces) — 3-item vs 5-item-same-K on spec-bench aggregate: 2627.2 vs 2642.3 (+0.6%, within noise). Users who don't opt in to the ctx axis pay nothing. **Absolute crossover vs no-spec (measured, not projected):** above ctx ~2k the 2D schedule pays the full DSD-mode tax and still beats no-speculation in absolute throughput (ctx 1900: 1.02×, ctx 4000: 1.09× vs no_spec). This is the direct mechanism measurement. ## Regime guardrail (documented usage) **The ctx axis is intended to raise K for long-ctx buckets, not to lower K for short-ctx buckets in mixed traffic.** Using it in the latter direction (as an earlier iteration of my C schedule did) can regress aggregate throughput — the previous −4.0% spec-bench result was an anti-pattern usage plus the DSD-mode baseline tax at short ctx compounding. The docstring for the 5-item schema states this explicitly. ## Changes - `vllm/config/speculative.py` — 3/5-tuple type union widening, `ctx_agg: Literal[\"median\",\"mean\",\"max\"] = \"mean\"` field - `vllm/v1/spec_decode/dynamic/utils.py` — `DynamicSDLookup(dense, ctx_boundaries)` NamedTuple, rectangular grid validation, mixed-arity rejection, 1-indexed ctx range convention - `vllm/v1/core/sched/scheduler.py` — 2D dispatch (`dense[bs][ctx_bucket]`), decode-only pool filter for ctx representative (`num_computed_tokens >= num_prompt_tokens` with fallback to full pool), aggregator dispatch (median/mean/max) - `tests/v1/spec_decode/test_dynamic_sd.py` — 10 new tests for 5-item parsing, 2D routing, rectangular validation, capture-set K-invariance ## Known limitations - **DSD-mode baseline tax**: all DSD-mode arms (including this PR's schedule) pay a substantial throughput tax vs no-spec at short ctx (up to −31% at ctx=400 in the measurements above). The `PIECEWISE` cudagraph downgrade for DSD is one identified factor; a full decomposition (K=0 fast-path opportunity, drafter forward on K=0, admission cost, tier-boundary ramp) is filed as #49986. This is orthogonal to this PR — it applies to the original 3-item DSD API from #32374 too — but is worth noting because the ctx=400 tie in the C′/A′ table is a consequence of it (both arms fire K=0 at that cell, so both eat the same tax). - **Evidence range**: measured for ctx ∈ {400, 900, 1900, 4000}. Extension to 8k+ requires re-designing the batch tier to fit within achievable concurrency at longer ctx (per-request KV budget scales with ctx) — a separate measurement. - **Single model/drafter pair**: measured on gemma-4-31B-it-qat-FP8 + assistant. Generalization to other pairs (Llama-3.1-8B + EAGLE-3, Qwen, etc.) is a natural next validation. - **APC-high regime**: the measurement uses `prefix_repetition` (num_prefixes=1) which represents high-shared-prefix serving (RAG, agents, batched summarization at ~98% APC hit). Under low-APC-hit workloads the story is different — the gain is bounded by how much of the KV read the amortization can save. - **Aggregator choice**: `mean` is a defensible default (mean × B ≈ Σ ctx_i ≈ verify KV read cost — a cost signal). `median` is the majority-amortization signal and wins under specific batch-shape skew (documented in the `ctx_agg` field). Per-sequence K is out of scope for this PR (would require gathered-K verify batching + straggler control) and noted as a longer-term direction in RFC #48627 Alternatives. ## Roadmap (why this PR is worth landing as the schema base) Filed as #49986: 1. **K=0 fast-path** — when the schedule selects K=0 for a given (bs, ctx), route through the no-spec path entirely for that step. Target: recover C′ ctx=400 from 1890.7 → 2711.7 tok/s (+43%), which would make C′ strictly dominant across all four cells (tie-to-win vs no-spec, +26% over static-K3, +29-35% over batch-only). Separate PR, 2-3 weeks. 2. **K-keyed FULL cudagraph capture** — since the schedule enumerates the K set at startup, (batch bucket × K)-shape graphs are statically capturable, and the DSD-mode PIECEWISE downgrade can be lifted for declarative schedules. Follow-on to the fast-path. 3. **8k+ evidence extension**, **second model/drafter pair validation**, **per-bucket AR reporting** — natural follow-ups. ## Tests 32 unit + integration tests pass on H100 (driver 570, CUDA 12.8): 28 in `test_dynamic_sd.py` + 4 CUDA-graph regression. E2E verified on H100 (driver 580, CUDA 13.0) via the measurement setup above. ## References - vLLM RFC #48627 — Context-length-aware K in DSD (design + Prior Art including MagicDec anchor) - vLLM PR #32374 — original DSD API (ekagra-ranjan) - MagicDec (Sadhukhan et al., arXiv:2408.11049) - Follow-up: #49986 — DSD baseline tax decomposition - Companion: SGLang PR sgl-project/sglang#31716 (sibling 2D routing implementation) --- ## AI assistance This change was drafted with the help of an AI coding assistant (Anthropic Claude) and reviewed line-by-line before this PR was opened. All code, tests, commit message, and PR text were reviewed by me; the test commands above were run by me on this branch. Filing this disclosure per vLLM's `AGENTS.md` requirement for AI-assisted contributions.", "url": "https://github.com/vllm-project/vllm/pull/48944", "createdAt": "2026-07-17T11:40:09Z", "updatedAt": "2026-08-13T07:54:50Z", "timestamp": "2026-08-13T07:54:50Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "documentation", "speculative-decoding", "v1", "nvidia", "mrv2" ], "author": "seongyun1104", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:48970", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] e2e hybrid-Mamba prefix-cache corruption regression tests (#43559)", "text": "## Purpose End-to-end output-correctness regression tests for **#43559** — hybrid-Mamba prefix caching silently corrupting output under `mamba_cache_mode=\"align\"` + MTP/EAGLE speculative decoding. These are the e2e tests split out of #48361 (per review) and supersede the test-only **#48198**. Two graded arms, each compared against an `enable_prefix_caching=False` control using deterministic greedy needle recall: - **cold-race** (write-side): concurrent cold prefills fragment a prefill chunk mid Mamba block; the chunk-end state can be cached as the boundary snapshot. - **multi-turn** (read-side): reuse of blocks written during (speculative) decode, where the eagle-lookahead lookup can let the Mamba hit length overrun the attention-verified hit. Parametrized on `Nemotron-Super-120B-A12B-BF16` (TP4) and `Qwen3.6-27B-FP8`. Corruption raises `CorruptionDetected`; geometry / control-quality / cache-liveness problems hard-fail with dedicated exceptions (never `pytest.skip`, since the fork runner reports in-body skips as PASS). An **anti-vacuity liveness gate** hard-fails a \"clean\" run whose prefix cache is not demonstrably live (and now also hard-fails if the ≥3-block liveness probe cannot fit `max_model_len`, rather than silently disabling itself). Registered as a **non-gating optional** Buildkite step. ## What these tests actually guard (scope) These are **in-process** (`LLM`) tests; they exercise the `HybridKVCacheCoordinator` read path and the align-mode scheduler, **not** the Mooncake KV-connector. Concretely: - The **corruption arms** are a regression guard for the already-merged in-process #43559 fixes (**#46384** kv_cache_coordinator eagle-margin gate + **#47782** selective retention). On current `main` these arms are green because those fixes are already in; the RED baseline in the table below is against a **pre-#46384/#47782 tree**. If a future change regresses the in-process fix, the corruption arms go red. - The **cold-race liveness gate** additionally requires **#48668** — it reads `vllm:prefix_cache_hits`, which the offline `LLMEngine.step()` path under-reports to 0 (hits applied on non-output-bearing prefill steps are dropped) without #48668, which would make the gate fire even though the cache is live. **Dependency:** requires **#48668** to pass. It does **not** exercise #48361's Mooncake connector path (`_find_hit_blocks`) — #48361 is covered by its own unit test, not here. (Kept in the #43559 stack for context; merge order relative to #48361 does not matter for these tests.) ## Why not a duplicate Supersedes #48198 (test-only, draft). The production fixes live in #48361 (Mooncake connector) and #48668 (stats); this PR is the in-process e2e regression coverage for #46384/#47782 + the #48668 liveness gate. ## Test Plan / Test Result ```bash pytest -v -s -rA tests/v1/e2e/test_hybrid_mamba_prefix_cache_correctness.py ``` Validated on GB200 (Nemotron-Super-120B-A12B-BF16, TP4, current `main` + #48668; #48361 present but not exercised): | Arm | Result | Engagement | |---|---|---| | cold-race | **1 passed** | `liveness_probe_hits=4224`, no corruption | | multi-turn | **1 passed** | wave-2 hit ratio 0.477 (67584 hits), no corruption | RED baseline (pre-#46384/#47782 main) reproduced corruption on both arms (documented in issue #43559). ## AI assistance disclosure Developed with AI assistance (Claude Code), including GB200 validation and an adversarial self-review that corrected the dependency scope above; every line reviewed by the human submitter.", "url": "https://github.com/vllm-project/vllm/pull/48970", "createdAt": "2026-07-17T17:59:01Z", "updatedAt": "2026-08-12T17:48:40Z", "timestamp": "2026-08-12T17:48:40Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready", "ci/build", "v1" ], "author": "puririshi98", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:48995", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Helion] Route fusion-only kernels to Helion during CUDA-graph capture", "text": "vLLM's post-grad fusion passes emit native fused ops (RMSNorm+quant, SiLU+mul+quant, QK-norm+RoPE) that have no eager model call site, so the capture-time call-site router used for per_token_group_fp8_quant (#47799) cannot reach them. Instead, add a single post-fusion FX pass that runs after fix_functionalization and retargets schema-compatible native fused ops to CUDA-graph-aware routed Helion ops. - vllm/kernels/helion/routing.py: build native->routed op map for the four fusion-only ops, gated on matching mutation signatures. Each routed op dispatches to Helion only while a CUDA graph is capturing and falls back to the native op otherwise. - vllm/compilation/passes/fusion/helion_routing.py: HelionFusionRoutingPass retargets both direct calls and remaining auto_functionalized nodes. - pass_manager.py: run the pass last (after defunctionalization) so vLLM's copy-elimination rules are preserved; gated on VLLM_USE_HELION_KERNELS. - Skip fused_qk_norm_rope on B200 pending the known numerical mismatch. - silu_mul_fp8 is intentionally excluded: its functional Tensor->Tensor schema is incompatible with native silu_and_mul_quant(out, ...). Tested on H100: focused routing tests (native fallback + cold CUDA-graph capture for all 4 ops) plus test_fusion, test_silu_mul_quant_fusion, and test_qk_norm_rope_fusion pass with routing enabled. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48995", "createdAt": "2026-07-17T23:31:43Z", "updatedAt": "2026-08-13T02:17:53Z", "timestamp": "2026-08-13T02:17:53Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm", "needs-rebase", "v1", "nvidia", "quantization" ], "author": "yushangdi", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Support sequence parallelism for block fp8 on XPU", "text": "## Purpose Enable sequence parallelism (SP) support on XPU for the block-quant W8A8 FP8 model SP is enabled with: ```json {\"pass_config\":{\"enable_sp\":true},\"use_inductor_graph_partition\":true} ``` ## Test Plan Model: `RedHatAI/Qwen3-32B-FP8-block` (dense, block-quant W8A8 FP8, `weight_block_size=[128,128]`), tp=4, `--linear-backend xpu`. ### Server (one of the three, per case) ```bash # sp_on VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 \\ --compilation-config '{\"pass_config\":{\"enable_sp\":true},\"use_inductor_graph_partition\":true}' # sp_off VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 # eager VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 --enforce-eager ``` ### Client (bench serve, run twice per case; better of the two reported) ```bash timeout -s KILL 6000000s python3 -m vllm.entrypoints.cli.main bench serve \\ --model RedHatAI/Qwen3-32B-FP8-block \\ --ready-check-timeout-sec 1 --temperature=0 \\ --dataset-name random --random-input-len=4096 --random-output-len=1024 \\ --ignore-eos --port=8072 --host 0.0.0.0 \\ --num-prompt 15 --request-rate inf --backend vllm --trust-remote-code \\ --max-concurrency 3 ``` ## Test Result Model: `RedHatAI/Qwen3-32B-FP8-block`, tp=4, `--linear-backend xpu`, input_len=4096, output_len=1024, num_prompts=15, max_concurrency=3. ### Accuracy (GSM8K, 250 questions) Case | Accuracy | Invalid rate -- | --: | --: `eager` (baseline) | 0.584 | 0.000 `sp_off` | 0.648 | 0.000 `sp_on` | 0.668 | 0.000 ### Benchmark Throughput: higher raw value is better. TTFT/TPOT: lower raw value is better. % shown is the improvement vs. `eager` baseline (positive % = better than eager, negative % = worse than eager). Case | Output throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) -- | --: | --: | --: | --: | --: | --: | --: `eager` (baseline) | 21.06 | 12729.52 | 14200.37 | 20602.96 | 130.09 | 129.23 | 136.14 `sp_off` | 22.34 (+6.08%) | 12891.37 (-1.27%) | 14516.25 (-2.22%) | 20284.13 (+1.55%) | 121.78 (+6.39%) | 121.62 (+5.89%) | 129.17 (+5.12%) `sp_on` | 22.34 (+6.08%) | 12344.39 (+3.03%) | 13671.84 (+3.72%) | 20345.60 (+1.25%) | 122.35 (+5.95%) | 121.68 (+5.84%) | 128.27 (+5.78%)", "url": "https://github.com/vllm-project/vllm/pull/49009", "createdAt": "2026-07-18T04:03:57Z", "updatedAt": "2026-08-13T13:47:52Z", "timestamp": "2026-08-13T13:47:52Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu", "ci/build" ], "author": "chaojun-zhang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49038", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Make `load_weights` completely optional", "text": "Part of https://github.com/vllm-project/vllm/issues/48972 Infrastructure changes: - Add `autoload_weights` and use it in weight loaders and runners. This creates it's own `AutoWeightsLoader` if the model doesn't have a `load_weights` method - When the model has no `load_weights` method, auto apply any `hf_to_vllm_mapper` in `AutoWeightsLoader._load_module` as the model is recursed into - Remove `skip_prefix` and `skip_substrs` from `AutoWeightsLoader`. These arguments duplicate the behaviour achievable with the mapper. Convert all uses of these arguments to mapper mappings - Add automatic `lm_head` skipping for models with `tie_word_embeddings` so every model doesn't need to specify this manually - Add `fused_qkv_interleaved` to `QKVParallelLinear` so that it can de-interleave a fused weight itself", "url": "https://github.com/vllm-project/vllm/pull/49038", "createdAt": "2026-07-18T13:28:20Z", "updatedAt": "2026-08-13T16:48:54Z", "timestamp": "2026-08-13T16:48:54Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "speculative-decoding", "ready", "needs-rebase", "v1", "llama", "qwen", "deepseek", "gpt-oss", "ready-run-all-tests", "mistral", "kimi", "cohere" ], "author": "hmellor", "state": "open", "assignees": [ "sangstar" ] }, { "id": "github:vllm-project/vllm:pull_request:49086", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][feature] Add new moe backends supporting int4/int8 weight-only…", "text": "## Purpose 1. Add Int8EmulationTritonExperts and TritonWNA16OTFExperts to EMULATION MoE backend to support int4/int8 quantization for WNA16; 1.1 Int8EmulationTritonExperts: dequant at load time; 1.2 TritonWNA16OTFExperts: dequant on-the-fly, supporint sym/asym/static/dynamic int4 quantization schemes, and also int8 quantization; 1.3 TritonWNA16OTFExperts is added to replace legacy MoeWNA16Method (same functionality with different structure to enable LoRA/EP/DP); 2. Update moe backend selection logic to honor --moe-backend cmd option for WNA16 case for ROCm branch", "url": "https://github.com/vllm-project/vllm/pull/49086", "createdAt": "2026-07-19T09:39:15Z", "updatedAt": "2026-08-13T06:47:05Z", "timestamp": "2026-08-13T06:47:05Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "needs-rebase", "quantization" ], "author": "qli88", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49104", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Bump `openai` to `>=2.25.0` to support namespace tools types", "text": "## Purpose Fix the dependency lower bound issue #49103. We have recently added OpenAI responses API namespace tools support, but the corresponding openai types are only supported for `openai>=2.25.0`. Hence the lower bound bump. Note that the dependency bump is necessary in the sense that any client with `openai<2.25.0` got an inevitable `ImportError` from the very beginning of vllm online serving process (see the error traceback in the linked issue). ## Test Plan No additional tests required - it just resolves the missing dependency lower bound bump. ## Test Result N/A ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49104", "createdAt": "2026-07-19T15:41:00Z", "updatedAt": "2026-08-13T03:52:31Z", "timestamp": "2026-08-13T03:52:31Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready", "ci/build" ], "author": "cjackal", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49139", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kernel] Fix persistent top-k histogram reuse after short rows", "text": " ## Purpose Fix a correctness bug in `persistent_topk` when one persistent CTA group processes a radix row (`seq_len > 32768`), followed by a short or medium row (`seq_len <= 32768`), and then another radix row. The kernel previously used the outer row iteration counter to rotate its triple-buffered radix histograms. Short rows advanced that counter without executing `radix_topk`, causing the next radix row to reuse a histogram that could still contain counts from an earlier row. This PR introduces a separate `radix_iter` counter that advances only when `radix_topk` executes. It also adds a CUDA regression test that constructs the long-short-long scheduling pattern in one persistent CTA group. Related to #41748, which addresses workspace initialization between kernel launches. This PR fixes histogram-ring misalignment within a single launch. ## Test Plan - Compare the long-short-long case against `torch.topk`. - Run a consecutive-radix case as a control. - Repeat both cases ten times on an RTX 3080. - Exercise the adaptive CTA-group layout used by the regression test. - Run the checked-in regression test in CI: ```bash python -m pytest \\ tests/kernels/test_top_k_per_row.py::test_persistent_topk_reused_group_after_short_row \\ -v ``` - Run static checks: ```bash python -m py_compile tests/kernels/test_top_k_per_row.py ruff check tests/kernels/test_top_k_per_row.py git diff --check origin/main...HEAD git clang-format --diff origin/main -- \\ csrc/libtorch_stable/persistent_topk.cuh ``` ## Test Result RTX 3080 results before the fix: ```text middle_len=32768: match=False, set_diff=8 middle_len=32769: match=True, set_diff=0 ``` Results after rebuilding the production `topk.cu` with the fix: ```text middle_len=32768: failures=0/10 middle_len=32769: failures=0/10 ``` Adaptive regression layout: ```text ctas_per_group=7, num_groups=9, rows=(0, 9, 18) set_diff=0 ``` Python compilation, Ruff, `git diff --check`, and clang-format checks passed. The complete `_C_stable_libtorch` extension and checked-in pytest were not run locally; they are left for CI. Local GPU validation compiled the production `topk.cu` and modified `persistent_topk.cuh` for SM86. No documentation update is required because this is an internal kernel correctness fix with no public API or configuration changes. AI assistance disclosure: OpenAI Codex was used for source analysis, implementation, regression-test construction, and validation. I reviewed and understand all submitted changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] No documentation update is required for this internal kernel fix.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/49139", "createdAt": "2026-07-20T02:07:09Z", "updatedAt": "2026-08-12T19:54:46Z", "timestamp": "2026-08-12T19:54:46Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "ready" ], "author": "fxfxfxfxfxfxfxfx", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:49171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Skip logits and sampling for unfinished prefills", "text": "## Purpose Model Runner V2 currently produces one sampling-logit row for every request in the common non-speculative path, including chunked-prefill requests whose prefill does not finish in the current step. Although these sampled tokens are discarded later by setting `num_sampled` to zero, the model has already performed the vocabulary projection and the sampler has already applied logits processors and sampling kernels. This introduces unnecessary GPU work, especially for workloads with multiple concurrent long prompts and a small `max_num_batched_tokens`. This PR avoids that work by: - Determining whether each request finishes prefill before constructing the logits batch. - Assigning zero logits rows to unfinished chunked-prefill requests. - Compacting mixed prefill/decode batches so output projection and sampling run only for requests that can produce an output token. - Short-circuiting `compute_logits` and sampling when the entire batch consists of unfinished prefill chunks. - Expanding compact sampled-token and NaN-count results back to the original per-request layout with a small Triton kernel, preserving existing downstream interfaces. - Preserving the existing pure-decode fast path. The change is limited to Model Runner V2's common non-speculative path. Speculative decoding remains unchanged. LoRA batches retain the existing dense logits layout because the per-request mapping is reused by prompt-logprob projection. This should reduce unnecessary vocabulary-projection and sampling work for chunked-prefill-heavy serving workloads without changing generated outputs. ## Test ### start server ``` VLLM_USE_V2_MODEL_RUNNER=1 \\ vllm serve Qwen3.5-4B \\ --served-model-name Qwen3.5-4B \\ --host 0.0.0.0 \\ --port 8000 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 2048 \\ --max-num-seqs 32 \\ --gpu-memory-utilization 0.90 \\ --disable-log-stats ``` ### vllm bench ``` vllm bench serve \\ --backend vllm \\ --host 127.0.0.1 \\ --port 8000 \\ --endpoint /v1/completions \\ --model Qwen3.5-4B \\ --tokenizer Qwen3.5-4B \\ --dataset-name random \\ --random-input-len 8192 \\ --random-output-len 32 \\ --random-range-ratio 0 \\ --ignore-eos \\ --request-rate inf \\ --max-concurrency 32 \\ --num-warmups 16 \\ --num-prompts 256 \\ --seed 0 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,99 ``` ## Test Result ### before \"IMG_3103\" ### after \"IMG_3104\" - Output token throughput increased from **174.72 tok/s** to **191.97 tok/s**, an improvement of **9.87%**. - Mean TTFT decreased from **4311.56 ms** to **3904.83 ms**, a reduction of **9.43%**. - Mean TPOT decreased from **43.97 ms** to **40.15 ms**, a reduction of **8.69%**. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49171", "createdAt": "2026-07-20T08:40:18Z", "updatedAt": "2026-08-13T15:16:32Z", "timestamp": "2026-08-13T15:16:32Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "multi-modality", "nvidia", "mrv2", "rust" ], "author": "jesse996", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49209", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU", "text": "## Purpose This is the second (2/2) PR introducing batch invariance to Intel XPU devices. You can find the first one [here](https://github.com/vllm-project/vllm/pull/41934). This PR registers the missing matmul and linear kernels. It also adds a new matmul_kernel_descriptor_persistent Triton kernel for better matmul performance on XPU. ## Test Plan Tested with unit tests covering batch invariance on XPU. ## Test Result All tests pass. ### Note The code was co-developed with GitHub Copilot. ---
Matmul and linear kernels were skipped in the previous PR, because they required more work. I tested batch invariance in Reinforcement Learning context and there I hit an error, because backward kernels were not registered for the \"XPU\" dispatch key. Also, I wanted to introduce a new Triton matmul kernel based on Tensor Derscriptors, because that implementation has shown ~3x speedup on XPU vs. the one based on pointers.
", "url": "https://github.com/vllm-project/vllm/pull/49209", "createdAt": "2026-07-20T14:45:22Z", "updatedAt": "2026-08-13T15:22:06Z", "timestamp": "2026-08-13T15:22:06Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "intel-gpu", "ready", "v1" ], "author": "tzielinski-habana", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49218", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build][The Rock] Use model_class_overrides so spawned worker can use test PredictableLlamaForCausalLM class when worker spawned using Python 3.14", "text": "## Purpose This PR changes the test to use the \"`module:class`\" string format with `model_class_override`, which lets `ModelConfig.model_class_override` re-register the model in each worker process. The following warning is seen when running this test: WARNING 07-20 15:41:58 [system_utils.py:157] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. The following failure occurs because the worker class can not find the class that was registered in the parent due to the spawn. `FAILED v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_with_predictable_dummy_model - RuntimeError: Engine core initialization failed. See root cause above. Failed ` Python 3.14 changed the default multiprocessing start method from \"`fork`\" to \"`forkserver`\", and CUDA initialization forces \"`spawn`\" when launching this test. With this PR the test now passes with both Python 3.12 and Python 3.14. ## Test Plan `pytest -sv v1/kv_connector/extract_hidden_states_integration/test_extraction.py` ## Test Result `==== 3 passed, 131 warnings in 211.66s (0:03:31) ====` ---
Essential Elements of an Effective PR Description Checklist - [ X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ X] The test plan, such as providing test command. - [ X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49218", "timestamp": "2026-08-12T16:49:34Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "ready", "ci/build", "v1", "llama", "kv-connector" ], "author": "rasmith", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:49225", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV-offload][FS]: Batching for read/write threads", "text": "PR #49152 introduced batched C versions of load_block / store_block. The batching was request-level, i.e. each thread in the thread_pool receives all the keys for a request and loads/stores them one after another. If there are 16 requests, each with say 100 keys to load, and we have 16 threads. Each thread will work on a separate request. This is less desirable as read jobs for different requests are interleaved on the disk, increasing any single request's total read time. This PR: Batch load/stores with the intent of parallelizing per-request loads/stores. i.e. Care about FCFS at the request level. In this PR, a single request is split into as many batches as there are threads and each thread gets an even distribution of work. ### Benchmarks Benchmark the following configurations, offload block size | n_read_threads / n_write_threads | comment -- | -- | -- 1024 | 16 | less keys to load per request (tens of keys) 1024 | 8 16 (default) | 16 | lots of keys to load per request (thousands of keys) 16 (default) | 8 #### Setup ``` MODEL=\"openai/gpt-oss-120b\" TP_SIZE=2 CPU_BYTES=25769803776 KV_TRANSFER_CONFIG=$(cat < offload block size | rw threads | main Total tokens / s | PR Total tokens / s | improvement x -- | -- | -- | -- | -- 1024 | 16 | 57490.4 | 59455.2 | 1.03x 1024 | 8 | 57923.7 | 60093.6 | 1.03x 16 | 16 | 52987.7 | 55501.7 | 1.04x 16 | 8 | 54061 | 56137 | 1.03x ### Eval ``` vllm serve openai/gpt-oss-120b --tensor-parallel-size=2 --kv-transfer-config '{ \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"spec_name\": \"TieringOffloadingSpec\", \"cpu_bytes_to_use\": 150930000000, \"eviction_policy\": \"lru\", \"secondary_tiers\": [{ \"type\": \"fs\", \"root_dir\": \"/mnt/nvme-storage/\", \"n_read_threads\": 16, \"n_write_threads\": 16 }] ,\"enable_cross_layers_blocks\" : \"True\" } }' --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --port 8000 ``` ``` TARGET_URL=\"http://127.0.0.1:8000\" MODEL=\"openai/gpt-oss-120b\" LM_EVAL_NUM_CONCURRENT=1000 LM_EVAL_TASKS=\"gsm8k\" lm_eval \\ --model local-completions \\ --model_args \"base_url=${TARGET_URL}/v1/completions,model=${MODEL},tokenized_requests=False,num_concurrent=${LM_EVAL_NUM_CONCURRENT},trust_remote_code=True\" \\ --tasks ${LM_EVAL_TASKS} \\ --seed 42 \\ --num_fewshot 25 \\ --gen_kwargs temperature=0.0 ``` ### Eval results #### Run1 : cold disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6353|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4200|± |0.0136| ``` #### Run2: warm disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6323|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4481|± |0.0137| ``` #### Run3: warm disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6164|± |0.0134| | | |strict-match | 25|exact_match|↑ |0.4253|± |0.0136| ```", "url": "https://github.com/vllm-project/vllm/pull/49225", "createdAt": "2026-07-20T16:29:10Z", "updatedAt": "2026-08-12T22:30:14Z", "timestamp": "2026-08-12T22:30:14Z", "metrics": { "reactions": 2, "comments": 20 }, "labels": [ "ready", "v1" ], "author": "varun-sundar-rabindranath", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49233", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][V1] Reserve CUDA graph memory in V2 GPU model runner", "text": "## Purpose Fixes #49224. Model Runner V2 became the default for **dense** models in 0.25.1, but its `profile_cudagraph_memory()` was a placeholder returning `0`: ```python def profile_cudagraph_memory(self) -> int: # NOTE(woosuk): It is TBD whether we keep this API or not. return 0 ``` `Worker.determine_available_memory()` subtracts this estimate before sizing the KV cache (gated by `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS`, default on), so with V2 no headroom was reserved for CUDA graph capture: the KV cache claimed the whole `gpu_memory_utilization` budget and `capture_model()` OOMed at startup (e.g. `Llama-3.1-70B-Instruct` FP8, TP=8 on 8× L40S). The legacy runner is unaffected because it implements the method for real, and before 0.25.1 dense FP8 models used the legacy runner anyway. This PR implements `profile_cudagraph_memory()` for the V2 runner, reusing its own `initialize_kv_cache()` / `capture_model()`: 1. Return `0` early when `cudagraph_mode == NONE`. 2. Bootstrap a **minimal** KV cache (`_init_minimal_kv_cache_for_profiling`) — just enough blocks to capture every graph, matching the legacy runner. 3. Capture graphs into a **throwaway** graph pool (`current_platform.graph_pool_handle()`) so their memory is reclaimed on teardown and does not pollute the persistent global pool the real capture reuses. 4. Measure the free-memory delta via the existing `capture_model()`. 5. Release the profiling KV cache / attention groups / captured graphs while keeping model weights (`_teardown_profiling_state`), so the real `initialize_kv_cache()` / `capture_model()` path starts from a clean slate. **Note for reviewers:** the bulk (PIECEWISE) graphs are effectively captured once — in the e2e run below the real capture pass added only 0.07 GiB after profiling — so the extra startup cost is small. Happy to add legacy-style sampling (capture ~2 graphs per mode and extrapolate) if preferred, since the original `# NOTE(woosuk)` flagged this API as \"TBD\". ## Test Plan - **Unit** (`tests/v1/worker/test_gpu_model_runner_v2_cudagraph_profiling.py`, CPU-only): covers the orchestration — disabled-cudagraph short-circuit, the no-graphs path, the capture path (asserting a throwaway pool is used and the measured value is returned), and teardown-on-error. ```bash pytest -q tests/v1/worker/test_gpu_model_runner_v2_cudagraph_profiling.py ``` - **End-to-end:** `Llama-3.1-70B-Instruct` FP8, `tensor_parallel_size=8`, `gpu_memory_utilization=0.90` on 8× L40S (no NVLink), with CUDA graphs enabled and the V2 runner as default (i.e. **without** `VLLM_USE_V2_MODEL_RUNNER=0` or `enforce_eager`) — the exact config that OOMs on 0.25.1. ## Test Result **Before (0.25.1, unpatched):** OOM during `capture_model()` at startup; server never becomes ready. Users on released 0.25.1 can work around this by forcing the legacy runner with `VLLM_USE_V2_MODEL_RUNNER=0` (keeps CUDA graphs), or via `enforce_eager` / a lower `--gpu-memory-utilization`. **After (this PR):** startup succeeds with CUDA graphs enabled and V2 as default. Key startup log: ``` Graph capturing finished in 39 secs, took 6.05 GiB # profiling capture (throwaway pool) [gpu_worker.py] Available KV cache memory: 19.46 GiB # headroom now reserved Graph capturing finished in 31 secs, took 0.07 GiB # real capture, no OOM INFO: Application startup complete. ``` Config confirms graphs were active (not silently disabled): `enforce_eager=False`, `cudagraph_mode=FULL_AND_PIECEWISE`, `tensor_parallel_size=8`, `quantization=modelopt`. Peak memory ~36.9/46 GiB per L40S (fits with headroom). Live checks: - `GET /v1/health/ready` → `200` - base model: `\"The capital of France is\"` → `\"a city of love, art, fashion, and cuisine.\"` - LoRA adapter: `\"Where is the Eiffel Tower?\"` → `\"Paris\"` Unit tests pass. --- *AI assistance disclosure (per the contributing guidelines): this change was developed with AI assistance (Cursor). All lines were reviewed and validated by the submitter, including the e2e run above.*
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49233", "createdAt": "2026-07-20T19:14:21Z", "updatedAt": "2026-08-13T11:15:23Z", "timestamp": "2026-08-13T11:15:23Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "v1", "nvidia", "mrv2" ], "author": "anhtra3889", "state": "open", "assignees": [ "njhill" ] }, { "id": "github:vllm-project/vllm:pull_request:49287", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][UT] Fix OOM and skip graph case", "text": "Fix UT on XPU: 1. test_mamba_cache_cg_padding can only run with graph enabled. Otherwise, UT will fall into an infinite loop. 2. To address the issue of insufficient VRAM caused by consecutive use cases (where there is insufficient free memory upon engine startup), the approach is to reuse the existing ROCm VRAM waiting mechanism and provide an appropriate default `gpu_memory_utilization` for the XPU.", "url": "https://github.com/vllm-project/vllm/pull/49287", "createdAt": "2026-07-21T06:24:20Z", "updatedAt": "2026-08-13T07:46:10Z", "timestamp": "2026-08-13T07:46:10Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "intel-gpu", "multi-modality", "kv-connector", "mistral" ], "author": "mayuyuace", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49357", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] bound FlashMLA sparse decode intermediate tensors size", "text": "## Purpose Fix #44545 - not in an ideal way though. When FlashMLA sparse kernel is selected with FP8 KV cache dtype, FlashMLA sparse decode kernel(https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_decode.h#L184) is called in mixed batch mode. This kernel internally allocates intermediate tensors (`o_accum`) on call, which quickly dominates the usual unallocated workspace size (~2GB per the comment in `flashmla_sparse.py`) for long prefill: For 32k input on H200, the bytesize of `o_accum` is ``` # b = 1 # num_sm_parts = 1 # s_q = 32768 # h_q = 64 # d_v = 512 (b + num_sm_parts) * s_q * h_q * d_v * 4 bytes = 8 GiB ``` For BF16 KV cache, a different kernel (FlashMLA sparse prefill kernel - https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_fwd.h#L101) is called, which does not allocate a massive VRAM on kernel call and thus does not suffer an OOM contrary to the FP8 code branch. (Note the additional `d_v` factor in the formular for `o_accum`, no other scratch tensors in FlashMLA have this factor thus negligible in terms of VRAM occupancy) Hence we chunk the mixed-batch input into manageable bytes (`VLLM_FLASHMLA_SPARSE_MAX_SCRATCH_MB`) and loop over the chunks to aggregate the output attention tensor, which is in the same spirit as the indexer budget management logic in #36178. Due to the mechanism (looping the kernel over sub-chunks instead of single big shot), it may incur a throughput penalty as a tradeoff, but I have no better idea yet. As the context window for MLA model families keep growing, FP8 KV cache is kind of inevitable, so I think this PR can work as a bandaid. ## Test Plan No OOM for the reproducer in #44545 ## Test Result pass ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49357", "createdAt": "2026-07-21T18:06:42Z", "updatedAt": "2026-08-13T16:20:26Z", "timestamp": "2026-08-13T16:20:26Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "v1" ], "author": "cjackal", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49365", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Detect ROCm wheel variant from environment for precompiled wheels.", "text": "Fixes **AMD: Python-only Installation** failing because ROCm precompiled wheels on `wheels.vllm.ai` use a different path layout than CUDA. - **`setup.py`**: Detect installed ROCm at runtime, match against published variants on `wheels.vllm.ai/rocm/{commit}/`, fall back to AMD PyPI. - **`python_only_compile.sh`**: Same variant resolution for the preflight metadata check.", "url": "https://github.com/vllm-project/vllm/pull/49365", "createdAt": "2026-07-21T19:40:03Z", "updatedAt": "2026-08-13T06:33:27Z", "timestamp": "2026-08-13T06:33:27Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "aarushjain29", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49381", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method", "text": "## TL;DR ModelOpt linear quantization is implemented today as **six near-duplicate `LinearMethod` classes**, one per format (FP8 per-tensor, FP8 per-channel/per-token, FP8 block-weight-only, NVFP4 W4A4, NVFP4 W4A16, MXFP8). This PR replaces all six with **one generic `ModelOptLinearMethod`**, composed from per-`QuantKey` schemes and driven by a `QuantSpec(weight, activation)` pair. Adding or changing a format becomes *data* (a `resolve()` row + reusable schemes), not a new class. Behavior-preserving for existing checkpoints (byte-level parity + GSM8K on all six formats). **MoE and mixed-precision are untouched.** Two long-standing issues get fixed as a side effect of unifying. 📄 **Design doc:** https://docs.google.com/document/d/14ao-WqOeeMIi0XPvjXXm4baxD4xQm_Jb/edit ## Motivation The six per-format linear classes share ~80% of their structure but each re-implements `create_weights` / `process_weights_after_loading` / `apply` inline, with subtle per-format differences that are **load-bearing and fail silently** — a wrong assumption produces garbage output with no error. Concretely: - **Duplication.** NVFP4 W4A4 and W4A16 are two entire classes that differ only in whether activations are quantized. The three FP8 classes differ mainly in scale shape. - **Scattered kernel selection.** Some classes build their kernel in `__init__`, others in `create_weights` — no single place decides. - **Hidden residue.** Each class silently does its own transpose / requant / reshape, so the format-specific quirks are buried in imperative code and easy to break during unrelated edits. - **Hard to extend.** A new format means a new class, wired into every dispatch site (base config + mixed-precision) and the weight-loader-v2 list. The design question (detailed in the doc): **which parts of a quantized linear layer are derivable purely from the format's numeric description, and which genuinely need per-format code?** The answer is that almost everything is derivable from the `(weight, activation)` QuantKey pair; the true residue is small and can be named explicitly. ## Design ### Flow — before vs after **Before — six self-contained classes** (the config picks one; each re-implements the whole lifecycle): ```text checkpoint (quant_algo) │ ▼ ModelOpt*Config.__init__ ──picks──► self.LinearMethodCls │ ┌─────┬─────┬─────┼─────┬─────┬─────┐ ▼ ▼ ▼ ▼ ▼ ▼ FP8 PcPt PbWo NVFP4 W4A16 MXFP8 ← 6 *LinearMethod classes │ │ │ │ │ │ └─────┴─────┴─────┴─────┴─────┘ │ ▼ each class re-implements its OWN: create_weights · process_weights · apply · kernel-selection (~80% duplicated; per-format quirks inline & silent-failing) ``` **After — one method composed from shared schemes** (each lifecycle phase calls the weight scheme, then the activation scheme, in order): ```text checkpoint (quant_algo) │ ▼ ModelOpt*Config.linear_algo() → \"NVFP4\" │ ▼ resolve(algo) → QuantSpec(weight_key, activation_key) CkptCtx(serialized, group_size) │ ▼ ModelOptLinearMethod (one generic class) ├─ wkey = SCHEME_FOR[weight_key] └─ akey = SCHEME_FOR[activation_key] │ ▼ fixed 3-phase lifecycle — each phase runs wkey THEN akey, in order: create_weights(layer) 1. wkey.create_weights(WEIGHT) register weight + scales 2. akey.create_weights(ACT) register activation params (if any) 3. select_linear_kernel(spec) pick fp4 / mxfp8 / fp8 kernel 4. expose_input_quant_key process_weights_after_loading(layer) 1. wkey.process(WEIGHT) transpose / requant / reshape 2. akey.process(ACT) 3. maybe_fuse_global_scales alpha = weight_gs × input_gs (if both present) 4. kernel.process_weights_after_loading apply(layer, x) 1. kernel.apply_weights shared schemes: NVFP4 W4A4 & W4A16 → same KNvfp4Static (differ only by activation: kNvfp4Dynamic vs None) ``` **Dispatch identity is the `(weight_key, activation_key)` pair.** A ModelOpt linear format is fully described by two `QuantKey`s. `resolve(algo, config, prefix)` maps the checkpoint's algo string to a `QuantSpec(weight, activation)` plus a `CkptCtx(serialized, group_size)`. `resolve` is strictly read-only over the config. **`QuantKeyScheme` — one scheme per QuantKey, content-keyed and role-parameterized.** Each scheme knows how to allocate and post-process *its key* in either the weight or activation slot, and rejects any role it hasn't validated (no silent wrong-role fall-through). Schemes are shared across formats — e.g. NVFP4 W4A4 and W4A16 use the *same* `KNvfp4Static` weight scheme. Seven schemes cover all six formats. **One generic `ModelOptLinearMethod`** holds `(wkey, akey)` from the pair and runs a fixed lifecycle: - `create_weights` → `wkey.create_weights(WEIGHT)` → `akey.create_weights(ACT)` → `select_linear_kernel(spec)` → expose input-quant-key - `process_weights_after_loading` → `wkey.process` → `akey.process` → `maybe_fuse_global_scales` → `kernel.process_weights_after_loading` - `apply` → one line to the kernel **`select_linear_kernel(spec, …)`** picks the kernel family from the weight key (fp4 / mxfp8 / fp8) in a single place. **Front-end.** Each config now exposes `linear_algo()` (just the algo string), and `get_quant_method`'s `LinearBase` arm calls `build_linear_method(config, algo, prefix)`. The config no longer names a method class — it names a format, and the behavior is *derived*. **Only three things don't fit a single key** (everything else composes): the cross-key `alpha = weight_gs × input_gs` (`maybe_fuse_global_scales`), the W4A16 placeholder-`input_scale` drop, and the MXFP8 idempotency guard. **Two extension seams** so future formats have tools, not forks: - **`FormatScheme`** — optional per-format hooks (`extra_weights` / `pre_process` / `post_process`) that compose *around* the key schemes, for residue that belongs to the layer as a whole. Default no-op. - **`LINEAR_METHOD_BUILDERS`** — a registry (empty by default) so a format that genuinely can't be a `(weight, activation)` pair registers its own `LinearMethodBase` by algo, via the single `build_linear_method` indirection point — without re-growing per-format branching in either dispatch arm. **MoE and mixed-precision are unchanged.** MoE keeps its per-format methods (the `ModelOpt*` class-name coupling that `routed_experts.py` gates on is preserved), and the mixed-precision MoE fork is byte-identical. Only the linear arm changed; a mixed checkpoint runs new-linear beside old-MoE with no new plumbing. ## Behavior preservation + two intended fixes Behavior-preserving for all six formats (evidence below). Two deltas fall out of unifying, both improvements: - **FP8 PbWo bug fix.** The old class skipped the block kernel's post-load via a misnamed guard, so under compiled serving it ran the GEMM on un-repacked weights → **garbage** (acc 0.000, invalid-rate 1.0). The generic method runs the post-load unconditionally → correct (0.897). (Note: old PbWo works under *eager*, so this only surfaces in real compiled serving.) - **MXFP8 → weight_loader_v2.** MXFP8 was the one linear format missing from `WEIGHT_LOADER_V2_SUPPORTED` (v1 loader); the generic method self-registers for v2. Proven byte-identical. ## W4A16 NVFP4 checkpoints that carry an on-disk `input_scale` (called out loud) A `W4A16_NVFP4` (weight-only) checkpoint can still ship activation `input_scale` tensors on disk — e.g. a W4A4 checkpoint relabeled `W4A16_NVFP4`, or scales injected by a downstream tool. Because `resolve(\"W4A16_NVFP4\")` sets `activation=None`, no `input_scale` param is registered and the on-disk tensor is orphaned. **This PR drops the orphan on load.** Simple linears (`down_proj`, `o_proj`) skip the unmatched key via `_ignore_unexpected_suffixes`; merged linears (`qkv_proj`, `gate_up_proj`) load shards through `LinearBase.load_weights`, which previously looked the param up unconditionally and crashed — now guarded by a `param is self` skip (the one shared-`linear.py` touch here). Native W4A16 (no scale on disk) is unaffected. Verified: a qwen3 W4A16 checkpoint carrying 108 `input_scale` loads clean weight-only (Marlin) with no behavior change vs the already-validated weight-only path. > A commit earlier on this branch also explored two follow-ons — a Scope-A > `_DropInputScale` FormatScheme (register a placeholder, then discard) as an alternative > to the load-time drop, and an `activation_dtype: nvfp4` config override that runs such a > checkpoint as *pure W4A4* (consuming the `input_scale`; validated at GSM8K 0.89310). > Both are intentionally **excluded from this PR's diff** and left for separate proposals; > the tip carries only the drop above. ## Adding a format (developer guide, in-code) - **Composes as a `(weight, activation)` pair** → add a `QuantKeyScheme` per new key to `SCHEME_FOR` + a `resolve()` row. No new method class. (How all six formats are built.) - **Needs format-wide residue, same lifecycle** → also return a `FormatScheme` from that `resolve()` row. - **Genuinely can't be a key pair** → write a `LinearMethodBase` and register it in `LINEAR_METHOD_BUILDERS` by algo. ## Testing Kept current with upstream `main` by rebasing; `modelopt.py` and `linear.py` have had no upstream changes, so the cutover applies cleanly. **Unit** — `pytest tests/quantization/test_modelopt.py -k \"not checkpoint_setup\"`: **20 passed.** Dispatch/config tests updated to assert the resolved `QuantSpec` + `ModelOptLinearMethod` (instead of the old classes), a new builder-registry test for the extension seam, and the existing upstream mixed-precision/mapper tests. **Lint**: `ruff check` + `ruff format --check` (0.14.0) clean on all changed files. **Per-format evidence** — Qwen3-8B. *Parity* is the deterministic gate (`VLLM_BATCH_INVARIANT=1` + `--enforce-eager`): per-layer weight/kernel/expose byte-hash + per-token prefill logit-diff (old vs generic) against a proven-`0.0` old-vs-old baseline. *GSM8K* (1319 Q · 5-shot · T=0 · 512 tok, batch-invariant) is the coarse end-to-end check. | Format | weight/kernel/expose hash | logit Δ | GSM8K generic | GSM8K old | Verdict | |---|---|---|---:|---:|---| | NVFP4 W4A4 | byte-identical | `0.0` | 0.89765 | 0.89765 | ✅ identical | | W4A16 NVFP4 | byte-identical | `0.0` | 0.87794 | 0.87642 | ✅ identical (+2 Q autotune) | | FP8 per-tensor | byte-identical | `0.0` | 0.88476 | 0.88628 | ✅ identical (−2 Q autotune) | | FP8 PcPt | byte-identical | `0.0` | 0.89689 | 0.89689 | ✅ identical | | MXFP8 | byte-identical | `0.0` | 0.89538 | 0.89538 | ✅ identical | | FP8 PbWo | weights differ ¹ | `0.0` (eager) | 0.89689 | **0.00000** | ✅ **bug fix** | Logits are **byte-identical for all six** → behavior-preserving. GSM8K deltas are ≤2 Q (torch.compile cross-load autotune noise, not math — weights are provably identical), invalid-rate 0. NVFP4 was re-confirmed **after rebasing onto current main** (`0.89765`, bit-identical) — upstream changes don't affect the cutover's output. ¹ PbWo stored weights differ *by design*: the generic method runs the block-kernel post-load the old class skipped (a misnamed-guard bug). Output is identical under eager; under real compiled serving the old class produces garbage (`0.000`, invalid-rate 1.0) while the generic path is correct — this is the fix noted above. ## Follow-up (not in this PR) To keep this diff reviewable, `modelopt.py` stays a single file. A natural next step is to split it into a package by tier — schemes/method vs MoE: ```text modelopt/ ├── configs.py # base + Fp8/NvFp4/MxFp8/Mixed configs (+ linear_algo) ├── kv_cache.py ├── linear/ # types.py · schemes.py · kernels.py · method.py │ # (ModelOptLinearMethod, FormatScheme, resolve, builders) └── moe/ # fp8.py · nvfp4.py · mxfp8.py ``` Constraints: keep the `ModelOpt*` class names (`routed_experts.py` coupling) and re-export the config classes from `modelopt/__init__.py` for import stability. ## Not duplicating existing work This **replaces** the in-tree ModelOpt linear methods; it is not a parallel implementation. #48422 targets the FP8_PB_WO post-load bug specifically; this PR fixes it as a consequence of unifying the linear methods. If #48422 merges first, the PbWo behavior here becomes a no-op vs the fixed class and we rebase accordingly. #50617 (serve `modelopt_mixed` checkpoints with FP8_PB layers) edits the same `modelopt.py` in the pre-refactor structure, so it will conflict; whichever lands first, the other rebases. Its two fixes — `has_blocked_weights()` and loading the 2-D `weight_scale_inv` block scale — are complementary and will be carried into the generic path on rebase. ## Notes - `mypy` currently hits an environmental internal-crash in the dev container (unrelated to this diff; reproduces with the changes stashed). CI's `mypy-3.12` hook runs it properly. ## AI assistance AI assistance (Claude Code) was used for this change; the diff and results were human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49381", "createdAt": "2026-07-21T23:53:04Z", "updatedAt": "2026-08-13T16:42:50Z", "timestamp": "2026-08-13T16:42:50Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "ready", "quantization", "verified" ], "author": "juhi10071998", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49406", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Add PARD-2 parallel draft model support", "text": "## Purpose This PR adds support for PARD-2 ([AMD-AGI/PARD](https://github.com/AMD-AGI/PARD)) target-aligned parallel draft models as a speculative-decoding method in vLLM. ## Test Plan **Serve** (PARD-2, target-dependent): ``` vllm serve --max-model-len 4096 --no-enable-prefix-caching --port 8008 --host 0.0.0.0 \\ --speculative-config '{\"model\": \"\", \"num_speculative_tokens\": 5}' ``` **Client** (concurrency sweep): ``` vllm bench serve --port 8008 --model --temperature 0 --backend openai-chat --endpoint /v1/chat/completions --dataset-name spec_bench --dataset-path question.jsonl --spec-bench-category math_reasoning --spec-bench-output-len 256 --chat-template-kwargs '{\"enable_thinking\": false}' --num-prompts 80 --max-concurrency <1..64> ``` - Hardware: **AMD EPYC 9755 128-Core Processor** (single socket, 128 threads), CPU backend, bfloat16. ## Test Result - Target: `Qwen/Qwen3-14B` · PARD-2 draft: `amd/PARD2-Qwen3-14B` · PARD-1 draft: `amd/PARD-Qwen3-0.6B` - Dataset: **GSM8K** (spec_bench `math_reasoning` category), no-thinking - Draft length `K = 5` , greedy (temp 0), 80 prompts. ## Performance Highlights \"image\" - Acceptance length : **PARD-2 = 4.30** (rate 65.9%), PARD-1 = 4.00 (rate 60.0%) ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49406", "createdAt": "2026-07-22T04:36:08Z", "updatedAt": "2026-08-13T07:13:31Z", "timestamp": "2026-08-13T07:13:31Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "new-model", "speculative-decoding", "v1", "llama", "qwen" ], "author": "ganeshr10", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49458", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Hardware-agnostic model definition via HF transformer backend (1/N)", "text": "## Purpose This PR is an alternative approach to realize hardware-agnostic model definitions based on the HF transformer backend. In particular, the idea is that the modeling code of tail models resides in HF transformers and can be executed in vLLM through the help of the `transformers` backend, i.e., `--model-impl transformers`. The way this backend currently works is by replacing / patching particular layers from HF transformers with native vLLM implementations. This first PR introduces a lightweight infrastructure to overrides the native vLLM implementations with hw-agnostic definitions instead through an environment variable `VLLM_USE_HW_AGNOSTIC`. If set to VLLM_USE_HW_AGNOSTIC=1, the hw_agnostic layers/activations are tried to be imported first and only if failing, the default vLLM layers are imported. In addition to the basic infrastructure, this PR also introduces the SiluAndMul activation function as well as the RMSNorm layer, to sketch the usage of the new infrastructure (originally prepared as a separate [PR](https://github.com/bohnstingl/vllm/pull/5)). The basis for the hw-agnostic layer implementation is the DSv4 migration effort from #45470 ## Test Plan In the future, we might need to have dedicated tests for the hardware-agnostic models, see last point for discussion above. Currently we only performed rudimentary verification tests, i.e., serving the model via and then prompting the model via ``` VLLM_USE_HW_AGNOSTIC=1 vllm serve \\ ibm-granite/granite-3.3-8b-base \\ --model-impl transformers \\ --tensor-parallel-size 1 \\ --data-parallel-size 1 \\ --trust-remote-code \\ --max-model-len 4096 \\ --gpu-memory-utilization 0.85 \\ --enforce-eager ``` ``` curl -s http://localhost:8000/v1/completions -H \"Content-Type: application/json\" -d '{\"model\":\"ibm-granite/granite-3.3-8b-base\",\"prompt\":\"What is vLLM?\",\"max_tokens\":13,\"temperature\":0}' ``` yielding the response ``` {\"id\":\"cmpl-a3fffb08bcf8b243\",\"object\":\"text_completion\",\"created\":1784727174,\"model\":\"ibm-granite/granite-3.3-8b-base\",\"choices\":[{\"index\":0,\"text\":\"\\n\\nvLLM is a new way to learn languages.\",\"logprobs\":null,\"finish_reason\":\"length\",\"stop_reason\":null,\"token_ids\":null,\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"routed_experts\":null}],\"service_tier\":null,\"system_fingerprint\":\"vllm-0.23.1rc1.dev955+gc2a8e0b45.d20260707-nohash\",\"usage\":{\"prompt_tokens\":6,\"total_tokens\":19,\"completion_tokens\":13,\"prompt_tokens_details\":null},\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null} ``` ## Test Result None yet cc @tdoublep ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49458", "createdAt": "2026-07-22T14:15:34Z", "updatedAt": "2026-08-13T10:21:50Z", "timestamp": "2026-08-13T10:21:50Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "ready", "verified" ], "author": "bohnstingl", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49506", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Carry chunk index in OffloadKey", "text": "## Purpose Re-scopes this PR to a single, minimal change agreed with the maintainer in review: carry the chunk index inside the `OffloadKey` so the self-describing KV-event path can recover a chunk's provenance from the key alone. `OffloadKey` becomes: ``` block_hash | group_idx:u32 (big-endian) | chunk_idx:u32 (big-endian) ``` where `chunk_idx` is the absolute, group-relative chunk index. This removes the need to thread a separate `chunk_idx` through the events tracker, so the offloading manager, tiering, and worker layers keep treating the key as an opaque identity. This is the design @orozery proposed in review ([suggestion](https://github.com/vllm-project/vllm/pull/49506#discussion_r3655466611)) and I [agreed to re-scope this PR to](https://github.com/vllm-project/vllm/pull/49506#discussion_r3661314636): coupling `chunk_idx` to the key makes single-key `prepare_store` unnecessary for KV events, so that admission work moves to a follow-up. ### What changes - `vllm/v1/kv_offload/base.py`: `make_offload_key(block_hash, group_idx, chunk_idx)` now requires `chunk_idx` (no default, so no caller can silently pack the wrong position); add `get_offload_chunk_idx()`; `get_offload_block_hash()` / `get_offload_group_idx()` slice the widened key. - `.../offloading/scheduler.py`: `update_offload_keys()` numbers each group's new chunks from its current key count (first pass `0, 1`; a later append continues `2, 3`; every group starts from 0). The events tracker no longer takes a separate `chunk_idx`, so `_maximal_prefix_lookup()` drops its `start_chunk_idx` parameter and `_build_store_jobs()`'s `record_store()` call drops it too. `_build_store_jobs()` otherwise keeps `main`'s batch store logic and still computes an internal `chunk_idx` for GPU block indexing. - `.../offloading/events.py`: `record_store()` / `record_lookup()` decode `chunk_idx` from the key via `get_offload_chunk_idx()`. ### Intentionally unchanged - **KV-event wire format** — `BlockStored` / `BlockRemoved` schema and payloads are unchanged; `chunk_idx` lives in the key, not on the wire. - **Store admission, allocation atomicity, worker transfer** — the CPU/tiering managers are byte-for-byte unchanged from `main`, and `_build_store_jobs()` keeps its batch behavior (batch `prepare_store`, batch allocation/eviction/cascade, existing `pending_primary_stores` pairing); its only change is the `record_store()` call dropping the `chunk_idx` argument. - **Dedup** — unchanged under fixed chunk geometry: a given (hash, group) always maps to the same `chunk_idx`, so keys are stable. - **FS layout** — `FileMapper` still keys file names on `(hash, group)`; a non-zero `chunk_idx` does not change the file name. ### P2P compatibility (breaking) P2P transports the full `OffloadKey` bytes opaquely, so producer and consumer peers must run the **same** key format — deploy in lockstep. This PR does not implement mixed-version P2P compatibility; a key-format/version handshake is a follow-up. ### Out of scope (follow-ups) - Single-key `prepare_store` admission (k-way merge, `OffloadReject`) — @orozery agreed this can follow. - Worker-facing transfer reshape — #44865 (sequenced after this). - Full promotion provenance and key-only removal. Not a duplicate: the nearest open neighbor is #44865 (worker transfer data-model reshape), which the review thread sequences *after* this change and which consumes rather than duplicates it. Other open KV-offload PRs (#48798 tiering metrics, #49850 HIT_PENDING bound) are unrelated. ## Test Plan ``` .venv/bin/python -m pytest \\ tests/v1/kv_offload/ \\ tests/v1/kv_connector/unit/offloading_connector/ ``` New/updated behavior tests: - `OffloadKey` round-trip and same-(hash, group) chunk disambiguation (`tests/v1/kv_offload/test_base.py`). - `update_offload_keys()` per-group numbering: `[0, 1]` then a later append `[2, 3]`, and each KV group numbering independently from 0 (`tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py`). - Events tracker selects chunk metadata by key alone (existing order-independence test) and a placeholder decodes the tail hash / group from an extended key without leaking `chunk_idx` onto the wire (`.../test_events.py`). - FS file name unchanged under a non-zero `chunk_idx` (`tests/v1/kv_offload/test_file_mapper.py`). - P2P opaque full-key round-trip over the ZMQ transport (`tests/v1/kv_offload/tiering/p2p/test_zmq_transport.py`). ## Test Result All offloading suites pass locally on an L4 (torch 2.13.0+cu130, Python 3.12): 498 passed across `tests/v1/kv_offload/` and `tests/v1/kv_connector/unit/offloading_connector/`. `ruff check`, `ruff format`, and `mypy` (3.12) are clean on the changed files. --- AI assistance was used to prepare this PR (Cursor). A human author reviewed every changed line and ran the tests above.", "url": "https://github.com/vllm-project/vllm/pull/49506", "createdAt": "2026-07-23T03:22:48Z", "updatedAt": "2026-08-13T05:43:55Z", "timestamp": "2026-08-13T05:43:55Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "needs-rebase", "v1", "kv-connector" ], "author": "Change72", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49539", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Quantization] Keep unquantized routed experts unquantized under AutoGPTQ", "text": "# Issue #49141 — Fused_moe dimension mismatch for Qwen MXFP4 model on ROCm ## 1. Root cause The issue has two reports. Both are the same underlying defect: **a MoE layer whose expert weights are stored *unquantized* in the checkpoint is still given a *quantized* MoE method**, so the allocated parameters do not match the checkpoint tensors. The trigger in both cases is the MTP (speculative-decoding drafter) layer. AMD's Quark and community GPTQ quantizations of `Qwen3.5-35B-A3B` quantize the 40 transformer layers but leave the MTP layer's experts in bf16. The main model loads fine; only the drafter fails. * **Original MXFP4/Quark report** (`RuntimeError: The size of tensor a (256) must match the size of tensor b (512) at non-singleton dimension 1`, in `routed_experts.py::_load_w2`). `hidden_size=2048`, `moe_intermediate_size=512`. Quark MXFP4 allocates `w2_weight` as `(E, hidden, intermediate // 2)` = `(E, 2048, 256)` (fp4 packs two values per byte), while the checkpoint's MTP `down_proj.weight` is the unquantized `(2048, 512)`. `amd/Qwen3.5-35B-A3B-MXFP4`'s `quantization_config.exclude` lists `mtp.layers.0.mlp.experts..{gate,up,down}_proj` for all 256 experts, but `should_ignore_layer()` only compared the *parent* layer name `mtp.layers.0.mlp.experts` against those *child* entries. **This half is already fixed on main** by the `check_children` argument added to `quark/utils.py::should_ignore_layer` (the amd/GLM-5.2-MXFP4 fix), which is why a commenter on the issue could not reproduce it on main. No change needed. * **Still-broken GPTQ report** (issue comments: `AttributeError: 'RoutedExperts' object has no attribute 'w2_weight'. Did you mean: 'w2_qweight'?`, same drafter-loading path). `AutoGPTQConfig` tracks which modules the checkpoint actually quantized in `modules_in_block_to_quantize` — auto-derived in `maybe_update_config()` from the safetensors dtypes, so it contains only the quantized layers' per-expert module names. `get_linear_quant_method()` consults it via `is_layer_gptq_quantized()`, but the `RoutedExperts` branch of `AutoGPTQConfig.get_quant_method()` **never consulted it at all** — neither on the Marlin path nor on the `MoeWNA16Config` fallback path taken here. The MTP layer therefore got a WNA16 MoE method allocating `w13_qweight`/`w2_qweight`, while the checkpoint supplies `...down_proj.weight`, and `RoutedExperts.load_weights()`'s `getattr(self, \"w2_weight\")` raised. ## 2. The fix and why Consult `modules_in_block_to_quantize` for routed-expert layers, the same way it is already consulted for linear layers. * New `is_moe_layer_gptq_quantized(prefix, quantized_layers)` in `gptq_utils.py`. `prefix` is the fused layer (`...mlp.experts`) while the config lists individual expert projections (`...mlp.experts.0.down_proj`), so a plain substring test in either direction is wrong. The helper normalizes each entry to its `...experts` parent and matches that as a suffix of `prefix`, which works for both absolute names (the auto-derived case) and the block-relative names optimum writes. * Called from `AutoGPTQConfig.get_quant_method()` *before* the Marlin-support check, so both the Marlin path and the `MoeWNA16Config` fallback are covered by the single early `return UnquantizedFusedMoEMethod(layer.moe_config)`. Deliberately conservative: when the config enumerates no expert module at all it carries no information about experts, so the helper returns `True` and behaviour is byte-for-byte unchanged for every config shape that does not list experts. ## 3. Files changed * `vllm/model_executor/layers/quantization/utils/gptq_utils.py` — add `is_moe_layer_gptq_quantized()`. * `vllm/model_executor/layers/quantization/auto_gptq.py` — import it and use it in the `RoutedExperts` branch of `get_quant_method()`. * `tests/quantization/test_auto_gptq.py` — add `test_moe_layer_quantized_only_when_its_experts_are_in_the_checkpoint`. ## 4. Risk / uncertainty * The behaviour change is gated on `modules_in_block_to_quantize` containing at least one `...experts...` entry. Fully-quantized MoE GPTQ checkpoints list every layer's experts, so they keep the quantized method. The residual risk is a checkpoint that lists experts for *some* layers under a naming scheme where the `...experts` suffix does not line up with the vLLM module prefix; that layer would silently fall back to unquantized (loud failure on the following weight load rather than silent wrong numerics, since the parameter names would then mismatch). * Block-relative configs (`mlp.experts.0.down_proj`) still cannot distinguish the MTP layer from the main layers, so the fix does not help there. That is the pre-existing behaviour, asserted in the test so the limitation is explicit. * I could not obtain the exact GPTQ checkpoint the reporter used, so the shape of its `modules_in_block_to_quantize` is inferred from `maybe_update_config()`'s derivation logic plus the `w2_weight`-vs-`w2_qweight` error, not observed. * Not touched: the Quark/MXFP4 path (already fixed on main) and the AWQ path, which uses a different exclusion mechanism (`modules_to_not_convert`). ## 5. How I verified it * **Root cause**, traced from the traceback through `routed_experts.py` (`_load_w2`, `weight_loader`, `build_expert_params_mapping`), `quark/quark_moe.py::create_weights` (`get_packed_dim`), `quark/utils.py`, `auto_gptq.py`, `moe_wna16.py` and `gptq_utils.py`, cross-checked against `Qwen/Qwen3.5-35B-A3B/config.json` (`hidden_size=2048`, `moe_intermediate_size=512` — exactly the 2048×256 vs 2048×512 mismatch) and `amd/Qwen3.5-35B-A3B-MXFP4/config.json` (the `mtp.layers.0.mlp.experts.*` exclude entries). * **Confirmed the MXFP4 half is already fixed** by diffing `quark/utils.py::should_ignore_layer` at the reporter's commit `97c162676` (no `check_children`) against `main` (has it), matching the issue comment reporting it no longer reproduces. * **Helper logic**: executed `is_moe_layer_gptq_quantized` in isolation over 8 cases (main-model layer, MTP layer, block-relative names, no-experts config, fused-expert entry, nested `list[list[str]]` form, empty config) — all pass. * **Lint**: `ruff check` and `ruff format --check` clean on all three files; `py_compile` clean. * **Not run**: the pytest suite. This environment has no `.venv` and no torch installed, so `tests/quantization/test_auto_gptq.py` could not be executed, and no end-to-end load of the affected checkpoints was performed. ## Duplicate check GitHub search for open PRs referencing `49141` in the body returned `total_count: 0`. No open PR touches `is_layer_gptq_quantized` / `get_moe_quant_method`.", "url": "https://github.com/vllm-project/vllm/pull/49539", "createdAt": "2026-07-23T06:59:04Z", "updatedAt": "2026-08-13T03:54:49Z", "timestamp": "2026-08-13T03:54:49Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "rocm", "qwen", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49540", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MM] Profile the multimodal embedding merge during memory profiling", "text": "Automated draft from oss-work.", "url": "https://github.com/vllm-project/vllm/pull/49540", "createdAt": "2026-07-23T06:59:19Z", "updatedAt": "2026-08-13T03:54:51Z", "timestamp": "2026-08-13T03:54:51Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "v1", "qwen", "nvidia", "mrv2" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49541", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] Cover detokenize=False in offline beam search", "text": "Offline counterpart of #46422. Sets `detokenize=False` on the per-step sampling params in the offline beam search path, and threads it through the structured-output beam path as well, so intermediate beam candidates are not detokenized on every step. This does not fix the Whisper slowdown reported in #49197. That one is the per-beam encoder recompute called out in the FIXME in `vllm/entrypoints/generate/beam_search/utils.py`, which is a separate change. Not benchmarked on GPU. ## Test `tests/samplers/test_beam_search_offline.py` covers both the plain and the structured-output beam paths and runs without a GPU.", "url": "https://github.com/vllm-project/vllm/pull/49541", "createdAt": "2026-07-23T06:59:22Z", "updatedAt": "2026-08-13T03:54:37Z", "timestamp": "2026-08-13T03:54:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "frontend", "ready" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] gfx942: use FlyDSL fp8 MQA logits kernel (ROCm/aiter#3913)", "text": "On gfx942, replace the vendored Triton `fp8_mqa_logits` with `aiter.ops.flydsl.flydsl_fp8_mqa_logits` from ROCm/aiter#3913. Drop-in replacement (identical args/semantics), gated behind `_ON_GFX942`. gfx950 and other paths are untouched. ## Results: GLM-5.2-FP8, 8× MI325X TP8 ### ISL=128K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 42,839 | 21,871 | **−49%** | | Median TPOT (ms) | 109.11 | 68.76 | **−37%** | | Output tok/s | 52.44 | 85.34 | **+63%** | ### ISL=8K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 918 | 863 | −6% | | Median TPOT (ms) | 19.79 | 19.68 | −0.6% | | Output tok/s | 382.2 | 386.3 | +1.1% | No regression at short context; gains scale with input length as seen above ## Accuracy | Test | Filter | Result | |------|--------|--------| | GSM8K (5-shot, full 1319 samples) | flexible-extract | 0.9416 ± 0.0065 | | | strict-match | 0.9424 ± 0.0064 | | NIAH (128K tokens, 10 needles, 10 trials) | — | 8/10 perfect, 2/10 miss one needle |
Serve command 
export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export HSA_NO_SCRATCH_RECLAIM=1 vllm serve zai-org/GLM-5.2-FP8 --block-size 1 --compilation-config '{\"pass_config\": {\"fuse_norm_quant\": true, \"fuse_act_quant\": true}, \"custom_ops\": [\"+rms_norm\", \"+silu_and_mul\"]}' --disable-log-stats --enable-auto-tool-choice --no-enable-expert-parallel --gpu-memory-utilization 0.80 --hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": 8}' --host 0.0.0.0 --kv-cache-dtype fp8 --max-model-len 263168 --max-num-batched-tokens 16384 --max-num-seqs 128 --no-enable-prefix-caching --port 8000 --reasoning-parser glm45 --tensor-parallel-size 8 --tool-call-parser glm47 --trust-remote-code
Bench command 
vllm bench serve --backend vllm --model zai-org/GLM-5.2-FP8 --host localhost --port 8000 --dataset-name random --random-input-len 131072 --random-output-len 1024 --max-concurrency 8 --num-prompts 20 --num-warmups 2 --seed 4 --ignore-eos --trust-remote-code
", "url": "https://github.com/vllm-project/vllm/pull/49544", "createdAt": "2026-07-23T07:20:08Z", "updatedAt": "2026-08-13T14:57:49Z", "timestamp": "2026-08-13T14:57:49Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "v1" ], "author": "akii96", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49577", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Mask Replay", "text": "## Summary This PR adds experimental support for **sampling distribution replay**. A sampling mask represents the vocabulary support retained after top-k/top-p filtering. It is not an attention mask and does not affect causal attention or KV-cache behavior. When enabled, vLLM returns the sampling support for each generated token in a CSR-style representation: ```python SamplingMask( token_ids=[...], offsets=[0, ..., len(token_ids)], ) ``` The feature is opt-in and does not change default generation behavior. ## Motivation With top-p sampling, rollout probabilities are normalized over a truncated vocabulary, while training commonly recomputes probabilities over the full vocabulary. This creates a systematic mismatch in importance ratios and KL estimates. Sampling replay returns the rollout-time support so training can recompute log-probabilities over the same distribution. ## Experiment We compare GRPO training with sampling replay enabled and disabled under top-p sampling. For each generated token: - `logp_diff = training_logprob - rollout_logprob` - `importance_ratio = exp(logp_diff)` Ideally, the rollout and training distributions match, so `logp_diff` is close to `0` and `importance_ratio` is close to `1`. ![Sampling replay metrics](https://github.com/user-attachments/assets/baa591e0-7a21-4b0a-9a62-488685ec2789) ### Mean Importance Ratio With replay enabled, the mean importance ratio remains tightly centered around `1`. Without replay, it is consistently below `1` and shows larger fluctuations. This indicates that replay restores the rollout-time top-p normalization during training, while the non-replay baseline compares the truncated rollout distribution with a full-vocabulary training distribution. ### Mean Log-Probability Difference With replay enabled, the mean log-probability difference remains close to `0`. Without replay, it has a persistent negative bias. This is expected: a token probability normalized over the full vocabulary is generally lower than the probability normalized over the retained top-p support. ### Standard Deviation of the Log-Probability Difference Replay lowers the standard deviation of the log-probability difference. This means it improves not only the average alignment, but also token-level consistency between rollout and training probabilities. Lower variance produces more stable importance weights and reduces the impact of probability-ratio outliers. ### Approximate KL Replay produces a lower and more stable approximate KL estimate, with fewer large spikes. This indicates that the training-time distribution stays closer to the rollout-time distribution throughout optimization. ### Conclusion Across all four metrics, sampling replay improves rollout/training distribution consistency: - `importance_ratio_mean` stays close to `1`; - `logp_diff_mean` stays close to `0`; - `logp_diff_std` is lower; - `approx_kl` is lower and more stable. These results demonstrate the intended algorithmic effect of sampling replay: improved probability alignment and lower-variance importance weights. This experiment does not, by itself, establish an improvement in downstream reward or task accuracy. ## Limitations The current implementation requires: - Model Runner V2; - `logprobs_mode=\"processed_logprobs\"`; - `SamplingParams.logprobs=1`; - `temperature > 0`. Speculative decoding, custom samplers, and custom logits processors are not currently supported.", "url": "https://github.com/vllm-project/vllm/pull/49577", "createdAt": "2026-07-23T10:22:03Z", "updatedAt": "2026-08-13T08:44:42Z", "timestamp": "2026-08-13T08:44:42Z", "metrics": { "reactions": 3, "comments": 33 }, "labels": [ "documentation", "frontend", "ready", "v1", "mrv2", "rust" ], "author": "vx120", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49585", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] Added Build Connector Worker Meta for EC Connector", "text": "## Purpose **Why this is still needed after #38390.** The PR implemented the V2 model runner EC Connector, but EC still has no worker -> scheduler metadata channel. KV connectors have a complete one: `build_connector_worker_meta()` -> `KVConnectorOutput.kv_connector_worker_meta` -> `KVOutputAggregator` -> scheduler-side KV connector. EC had none of those three pieces, so a worker-side`ECConnector` has no way to report per-step state — e.g. \"this `mm_hash` is now durably stored\" — back to its scheduler-side EC connector. That blocks the CPU-offload based on CUDA events and NIXL P2P EC connectors, which need exactly this report (We are retiring the StepTracker in the CPU EC Connector, because using CUDA events is the right way to handle \"copy done\" notifications, instead of waiting until the batch is done). This PR adds that channel, and fixes the exit paths where the worker's `ec_connector_output` was computed and then dropped before reaching the scheduler. ### The channel - `ec_connector/base.py`: `ECConnectorWorkerMetadata` ABC with an abstract `aggregate()`, plus `ECConnectorBase.build_connector_worker_meta()`. - `v1/outputs.py`: `ec_connector_worker_meta` field on `ECConnectorOutput`. - `ec_connector/utils.py`: `ECOutputAggregator`, mirroring `KVOutputAggregator` - merges the EC output of the rank that ran the connector onto the rank whose `ModelRunnerOutput` reaches the scheduler. - `engine/core.py`, `executor/abstract.py`, `executor/multiproc_executor.py`: wiring so the aggregator runs when an EC connector is configured. - Both runners (MRv1, MRv2) populate the field via `build_connector_worker_meta()`. Default behavior is unchanged for existing connectors: nothing overrides `build_connector_worker_meta()` yet, so it returns `None`. ## Test Plan ```bash # Existing EC connector unit tests (unaffected areas) .venv/bin/python -m pytest tests/v1/ec_connector/unit/test_metadata.py \\ tests/v1/ec_connector/unit/test_ec_example_connector.py \\ tests/v1/ec_connector/unit/cpu/test_connector.py -v ## Test Result ``` tests/v1/ec_connector/unit/test_metadata.py ..................... PASSED tests/v1/ec_connector/unit/test_ec_example_connector.py ......... PASSED tests/v1/ec_connector/unit/cpu/test_connector.py ................ PASSED 36 passed, 2 warnings pre-commit: ruff check / ruff format / mypy / SPDX headers / etc. — all Passed ```", "url": "https://github.com/vllm-project/vllm/pull/49585", "createdAt": "2026-07-23T12:33:50Z", "updatedAt": "2026-08-13T16:42:37Z", "timestamp": "2026-08-13T16:42:37Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "ready", "ci/build", "v1", "kv-connector", "mrv2" ], "author": "omerpaz95", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49596", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Enable XPU blockfp8 for DSv3", "text": "## Purpose Enable DeepSeek-V3 on Intel XPU through the oneDNN block-scaled FP8 GEMM (XPUFp8BlockScaledMMKernel). Three issues blocked correct execution: 1. oneDNN requires N to be a multiple of block_n. Some DeepSeek-V3 block-FP8 projections have a per-partition N that isn't 128-aligned. We now pad the weight's N up to the next multiple of block_n in `process_weights_after_loading`, record the unpadded size, and drop the padded output columns after the GEMM in `apply_block_scaled_mm`. 2. Block-scale layout mismatch during weight dequant. The XPU kernel transposes the block scale to `[K/block_k, N/block_n]` layout at load time. `get_and_maybe_dequant_weights` (used by MLA to recover kv_b_proj for weight absorption) assumes the checkpoint `[N/block_n, K/block_k]` layout and asserts on the mismatch. We record `layer.weight_scale_transposed = True` when the kernel repacks the scale and undo the transpose in the dequant helper before scaled_dequantize. 3. Make the shared block base padding-agnostic. `Fp8BlockScaledMMLinearKernel.apply_weights` previously computed the output shape from `weight.shape[0]`, which is the padded N on XPU. It now derives the output shape from the actual GEMM output width (output.shape[-1]) after the matmul. This is a no-op for all non-padding backends (Triton / DeepGEMM / Cutlass / FlashInfer / Aiter-ROCm / CPU), where the returned width already equals `weight.shape[0]`, and it lets the XPU kernel return a sliced result that reshapes correctly. ## Test Plan Run a block-FP8 DeepSeek-V3 checkpoint on Intel XPU Max 1550 and confirm it (a) loads without the oneDNN N-divisibility failure and without the scaled_dequantize shape assertion in the MLA process_weights_after_loading, and (b) produces coherent generations. This did require using two workarounds which were outside the scope of this PR 1. Replace `torch.ops._C_cache_ops.getMemoryInfo(device)` with `torch.xpu.get_mem_info(device)` due to outdated L0 drivers on Max 1550 installation. 2. Topk workaround as detailed in https://github.com/vllm-project/vllm-xpu-kernels/pull/451#discussion_r3626844247 ## Test Result Before: DeepSeek-V3 block-FP8 failed to run on XPU. After: The model loads and generates on XPU (TP×PP×EP, --enforce-eager); the padded kv_a_proj layers run through oneDNN and the MLA kv_b_proj dequant succeeds. A partial run of gsm8k (limit: 40) shows high accuracy: ``` vllm ({'pretrained': 'deepseek-ai/DeepSeek-V3', 'tensor_parallel_size': 4, 'pipeline_parallel_size': 3, 'enable_expert_parallel': True, 'enforce_eager': True, 'max_model_len': 2048, 'max_num_seqs': 32, 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 1487067239, 'dtype': 'bfloat16'}), gen_kwargs: ({'max_gen_toks': 320}), limit: 40.0, num_fewshot: 5, batch_size: auto |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ | 0.95|± |0.0349| | | |strict-match | 5|exact_match|↑ | 0.95|± |0.0349| ```", "url": "https://github.com/vllm-project/vllm/pull/49596", "createdAt": "2026-07-23T14:21:04Z", "updatedAt": "2026-08-13T05:54:02Z", "timestamp": "2026-08-13T05:54:02Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "intel-gpu", "quantization" ], "author": "frost-intel", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49602", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Hoist $defs/definitions in Cohere parser tool schema composition", "text": "## Purpose Fix a bug where any `/v1/chat/completions` request with tools whose JSON Schema contains `$defs`/`definitions` fails with HTTP 500 on Cohere models served with `--reasoning-parser cohere_command3|cohere_command4`. Resolves the bug class reported in #16467 (closed as stale, never fixed) for the Cohere reasoning parser path specifically. Other parsers that compose their own grammars are out of scope (see the GLM case in #47175); the generic tool path in `vllm/tool_parsers/utils.py` already hoists `$defs` and is unaffected. **Root cause.** `collect_tool_schema` in `vllm/reasoning/cohere_command_reasoning_parser.py` builds the constrained-decoding grammar for tool calls by nesting each tool's `parameters` schema inside a `{tool_call_id, tool_name, parameters}` envelope. The tool schema was spliced in verbatim, so a definitions block originally at the schema root ends up under `properties.parameters`, while its `$ref` pointers (`#/$defs/X`) remain root-anchored. xgrammar then correctly rejects the dangling JSON pointer: ```text RuntimeError: Cannot find field $defs in #/$defs/X ``` The request fails during preprocessing, before generation. Agentic frameworks such as pydantic-ai emit `$defs` for any nested model, so this breaks effectively all tools defined via such frameworks (see user reports in #16467). **Minimal repro** against a Cohere model served with `--reasoning-parser cohere_command4 --tool-call-parser cohere_command4 --enable-auto-tool-choice` (any tool schema whose parameters carry `$defs` + `$ref`; a flat schema without `$defs` does not reproduce): ```bash curl -s http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"CohereLabs/North-Mini-Code-1.0\", \"messages\": [{\"role\": \"user\", \"content\": \"Weather in Prague?\"}], \"tool_choice\": \"auto\", \"tools\": [{ \"type\": \"function\", \"function\": { \"name\": \"get_weather\", \"parameters\": { \"$defs\": { \"Location\": { \"type\": \"object\", \"properties\": { \"city\": {\"type\": \"string\"}, \"country\": {\"type\": \"string\"} }, \"required\": [\"city\"] } }, \"type\": \"object\", \"properties\": {\"location\": {\"$ref\": \"#/$defs/Location\"}}, \"required\": [\"location\"] } } }] }' ``` Before this fix: HTTP 500 with `Cannot find field $defs in #/$defs/Location`. After: HTTP 200 with a valid `tool_calls` response. **Fix.** Build the envelope as a dict and hoist `$defs` (draft 2020-12) and `definitions` (draft-07) from the tool's parameters to the envelope root before calling `xgr.Grammar.from_json_schema`, so root-anchored refs resolve again. This mirrors what the shared tool path already does (`_get_tool_schema_defs` in `vllm/tool_parsers/utils.py`); the Cohere parser bypasses that path for its custom tag/envelope format and had reintroduced the bug. No cross-tool merge handling is needed here because each tool gets its own grammar. The parameters dict is copied so the incoming request object is not mutated. Grammars for schemas without definitions blocks are unchanged. Same bug class was recently fixed for the GLM tool parser path in #47175 (different code path, no overlap). **Not a duplicate.** No open PR references #16467 or touches the Cohere parser's tool-schema composition (checked `gh pr list` searches for \"cohere defs\", \"collect_tool_schema\", \"16467 in:body\"). The nearest related work, #47175, fixes the analogous issue in `glm47_moe_tool_parser.py`. ## Test Plan New regression tests in `tests/reasoning/test_cohere_command_reasoning_parser.py` (`TestCollectToolSchemaDefs`): nested-`$defs` tool through `adjust_request`, recursive `$defs` (can never be inlined away), draft-07 `definitions`, and a no-mutation check on the caller's parameters dict. ```bash python -m pytest tests/reasoning/test_cohere_command_reasoning_parser.py -v pre-commit run --files vllm/reasoning/cohere_command_reasoning_parser.py tests/reasoning/test_cohere_command_reasoning_parser.py ``` ## Test Result - Before the fix: all 4 new tests fail with `RuntimeError: Cannot find field $defs in #/$defs/Location` (the error). - After the fix: 44 passed (4 new + 40 pre-existing) in `tests/reasoning/test_cohere_command_reasoning_parser.py`. - pre-commit (ruff, mypy, etc.): all hooks pass. No model evaluation is included: the change only affects grammar composition for schemas that previously failed with a hard 500 before generation; grammars for all previously-working schemas are unchanged (no `$defs` block to hoist), so there is no before/after output quality to compare. --- AI assistance was used for this PR (Cursor agent): drafting the fix and the regression tests. All changes were reviewed, run, and are understood and defended by the submitter. Commit includes `Co-authored-by: Cursor Agent`. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49602", "createdAt": "2026-07-23T15:04:30Z", "updatedAt": "2026-08-13T16:51:37Z", "timestamp": "2026-08-13T16:51:37Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "tool-calling", "cohere" ], "author": "vaclavcadek", "state": "open", "assignees": [ "DarkLight1337" ] }, { "id": "github:vllm-project/vllm:pull_request:49639", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(kernel): restore scalar_t RMSNorm intermediate rounding boundary (#49616)", "text": "## Purpose Fixes #49616. ### Problem and root cause The RMSNorm CUDA kernels kept the normalized value in `float` through the weight multiplication. That changed the rounding boundary relative to the unfused `scalar_t` path and produced bit-level differences during greedy NGRAM/speculative verification. In our nightly comparison, the unpatched path differed in 1,208 of 4,096 elements; the patched path matched all 4,096. ### Solution The kernels now cast the normalized value to `scalar_t` before multiplying by the native-dtype weight in both the regular and FP8-quantized fused paths. This restores the execution-order boundary used by the unfused RMSNorm path. ## Test plan `tests/kernels/core/test_rmsnorm_precision.py` now calls `vllm._custom_ops.rms_norm` directly, compares the weighted kernel output with the scalar_t-rounded composite reference, covers both `float16` and `bfloat16`, and skips cleanly when CUDA/ROCm is unavailable. Local checks: ```text uv run --offline --no-sync ruff check tests/kernels/core/test_rmsnorm_precision.py uv run --offline --no-sync ruff format --check tests/kernels/core/test_rmsnorm_precision.py PYTHONPATH=. uv run --offline --no-sync pytest tests/kernels/core/test_rmsnorm_precision.py -v --noconftest ``` Ruff and formatting passed. The local CPU-only environment collected all 18 cases and skipped them because no CUDA/ROCm device is available; the CUDA validation is left to vLLM CI. The earlier nightly comparison above provides the GPU evidence for the regression. AI assistance disclosure: I used an AI coding assistant to help inspect the existing kernel/test paths and draft the focused regression-test correction. I reviewed and verified the final diff, test behavior, and commit contents myself.", "url": "https://github.com/vllm-project/vllm/pull/49639", "createdAt": "2026-07-23T19:55:06Z", "updatedAt": "2026-08-13T00:50:13Z", "timestamp": "2026-08-13T00:50:13Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [], "author": "Hasnaathussain", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49652", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Fix autoregressive draft decode capture with dynamic SD", "text": "## Purpose Fixes #48494. Dynamic speculative decoding (DSD) derives the decode query lengths for each scheduled speculative-token count from the manager's configured `decode_query_len`: ```text num_new_sampled_tokens = decode_query_len - max_speculative_tokens runtime_query_len = scheduled_speculative_tokens + num_new_sampled_tokens ``` This derivation is valid when `decode_query_len` includes the maximum number of draft tokens, as it does for target verification and autoregressive draft prefill. It is not valid for autoregressive draft decode: every draft step processes exactly one token per request and dispatches the CUDA graph with `num_tokens == num_reqs` and `uniform_token_count == 1`. For example, with `max_speculative_tokens=3`, applying the DSD derivation to the draft decode manager's `decode_query_len=1` produces query lengths `1`, `-1`, and `-2` for scheduled K values `3`, `1`, and `0`. The non-positive descriptors eventually reach `InputBatch.make_dummy` and fail: ```text assert 0 < num_reqs <= num_tokens ``` This PR: - adds `use_dynamic_decode_query_len` to `CudaGraphManager`, defaulting to `True` to preserve existing behavior; - disables dynamic query-length derivation only for the autoregressive draft decode manager, whose query length is fixed at one; - keeps dynamic query-length derivation enabled for target verification, autoregressive draft prefill, DFlash, and all other existing callers; and - adds a production-wiring regression test for a representative `K=3/1/0` DSD schedule. This keeps full CUDA graph execution enabled. It does not clamp invalid shapes, swallow the assertion, or fall back to eager execution. Duplicate-work checks found no open PR referencing #48494 and no existing autoregressive Dynamic-SD CUDA Graph fix. Related PR #47737 skips non-positive query lengths for DSpark's dynamic draft manager when a K=0 tier disables drafting. That is complementary: the autoregressive draft decode manager has a fixed query length of one for every draft step and must not derive its shape from the DSD schedule at all. In particular, filtering non-positive derived lengths can leave it with no FULL decode graph when the runtime schedule does not contain the configured upper-bound K. PR #47131 implements D-cut verification-depth pruning and is also unrelated to this capture-shape bug. ## Test Plan ### Unit and static checks ```bash .venv/bin/python -m pytest -q \\ tests/v1/spec_decode/test_dynamic_sd_cug.py \\ tests/v1/cudagraph/test_cudagraph_manager.py .venv/bin/ruff check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py .venv/bin/ruff format --check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py git --no-pager diff --check ``` The regression test covers: - a valid K=3/1/0 runtime schedule below the configured upper-bound K=4; - dynamic draft-prefill query lengths; - fixed draft-decode query length `1`; and - full-CUDA-graph dispatch for every batch size from 1 through 8. ### GPU red/green reproduction Environment: - 2x NVIDIA A800, tensor parallel size 2; - V2 model runner; - Qwen3.5 architecture with native `Qwen3_5MTP`; - GDN attention path; - full CUDA graphs; and - DSD schedule `K=3/1/0`. ### Real-weight end-to-end smoke test The patched branch was also tested with the public `Qwen/Qwen3.5-0.8B-Base` checkpoint, including its native MTP weights: ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-0.8B-Base \\ --tensor-parallel-size 2 \\ --max-model-len 1024 \\ --max-num-seqs 8 \\ --gpu-memory-utilization 0.50 \\ --dtype bfloat16 \\ --language-model-only \\ --speculative-config '{ \"method\": \"mtp\", \"model\": \"Qwen/Qwen3.5-0.8B-Base\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [ [1, 2, 3], [3, 4, 1], [5, 8, 0] ], \"draft_tensor_parallel_size\": 2 }' \\ --compilation-config '{ \"mode\": 3, \"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [8] }' ``` The same checkpoint and request set were then run with `--enforce-eager` as a numerical reference. ## Test Result ### Unit and static checks ```text Base: vllm-project/vllm@1479bd9e9 Environment: Python 3.11.2, torch 2.11.0+cu130, 2x NVIDIA A800-SXM4-40GB pytest: 6 passed ruff check: All checks passed ruff format --check: 3 files already formatted pre-commit: 10 applicable Python hooks passed git diff --check: passed ``` The pre-commit run covered Ruff check/format, typos, mypy 3.10, SPDX headers, root lazy imports, filenames, forbidden imports, new `torch.cuda` API usage, and boolean context-manager checks. ### GPU red/green result Before this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/3 failure: InputBatch.make_dummy -> AssertionError ``` After this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/1 server: Application startup complete ``` ### Real-weight result - 9/9 requests succeeded across concurrency 1, 3, and 5, exercising scheduled K values 3, 1, and 0. - Speculative-decoding metrics confirmed that the MTP path generated and accepted draft tokens. - For the patch-active K=3 and K=1 tiers, all four full-CUDA-graph results matched eager results field-for-field. Repeating the eager run produced the same four results. - K=0 does not execute autoregressive draft decode and was therefore excluded from the patch-path numerical comparison. - Neither the full-CUDA-graph nor eager log contained `AssertionError`, `InputBatch.make_dummy`, or an `ERROR` record. The end-to-end smoke test used Qwen3.5 native MTP rather than the Gemma4 checkpoint from #48494. Both use the shared `AutoRegressiveSpeculator` draft decode manager fixed by this PR. This test validates the control flow and small-model numerical behavior; it does not make a large-model performance claim. ## AI assistance OpenAI Codex assisted with root-cause analysis, code drafting, and test drafting. I reviewed the changes and validation results, and this PR is ready for maintainer review.", "url": "https://github.com/vllm-project/vllm/pull/49652", "createdAt": "2026-07-24T02:00:33Z", "updatedAt": "2026-08-13T17:35:51Z", "timestamp": "2026-08-13T17:35:51Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "bug", "speculative-decoding", "v1", "nvidia", "mrv2" ], "author": "CZT0", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49688", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs", "text": "## Purpose CPU GDN attention (Qwen3.5) selects its causal-conv1d implementation via `torch.cpu._is_amx_tile_supported()`: AMX CPUs (Intel GNR) use the C++ kernels (`causal_conv1d_fwd_cpu` / `causal_conv1d_update_cpu`), everything else falls back to `causal_conv1d_fn_cpu` / `causal_conv1d_update_torch`, a pure-PyTorch per-sequence loop. The C++ conv kernels (`csrc/cpu/sgl-kernels/conv.cpp`) do not use AMX tiles but `is_amx` gate blocks it from running on non-AMX platforms that support AVX-512BF16. The float32 SSM state(currently gated by `is_amx` is required for entire CPU GDN path, not just AMX This PR fixes this gating logic. ## Test plan Extended `tests/kernels/mamba/cpu/test_cpu_gdn_ops.py` (all gated on `_is_avx512_bf16_supported()`, so they run on non-AMX CPUs too). Expected Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. ```bash python -m pytest tests/kernels/mamba/cpu/test_cpu_gdn_ops.py -v ``` Qwen-3.5 workload run before and after the patch on both Intel GNR and AMD EPYC Turin ``` VLLM_CPU_OMP_THREADS_BIND=0-63 VLLM_CPU_KVCACHE_SPACE=40 vllm bench throughput \\ --model Qwen/Qwen3.5-9B \\ --trust-remote-code \\ --dataset-name random \\ --input-len 1024 \\ --output-len 1024 \\ --num-prompts 128 \\ --max-num-seqs 32 \\ --dtype bfloat16 ``` Lmeval ``` VLLM_CPU_OMP_THREADS_BIND=0-63 \\ VLLM_CPU_KVCACHE_SPACE=40 \\ lm_eval \\ --model vllm \\ --model_args \"pretrained=Qwen/Qwen3.5-9B,dtype=bfloat16,trust_remote_code=True,max_model_len=4096\" \\ --tasks gsm8k \\ --num_fewshot 5 \\ --batch_size auto ``` ## Test result Python GDN test Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. Workload Qwen3.5-9B, BF16, ISL/OSL 1024/1024, 64-core instance (median of 3 runs) Intel GNR is unchanged (it takes the identical path both ways, so no regression). AMD EPYC Turin gains about 9-10% on this memory-bound, decode-heavy shape, which is the hardest case for a conv speedup to show up. ## Correctness `lm_eval` gsm8k, 5-shot, full set (n=1319), Qwen3.5-9B BF16, before vs after: | Platform | Metric | Baseline | After | |----------|--------|---------:|------:| | Intel GNR (AMX) | exact_match | identical | identical | | AMD Turin (Zen5) | flexible-extract | 0.8696 +/- 0.0093 | 0.8749 +/- 0.0091 | | AMD Turin (Zen5) | strict-match | 0.8734 +/- 0.0092 | 0.8779 +/- 0.0090 | GNR is bit-identical (same code path both ways). On Turin, before/after differ by less than one standard error, so there is no accuracy regression. The C++ conv accumulates in fp32 and is numerically no less accurate than the torch fallback(both round to bf16 at store), which the unit tests also assert against an fp32 oracle. ## Duplicate Check Searched open PRs/issues for GDN / causal_conv1d / Zen / AVX-512BF16 CPU conv work; found none addressing the non-AMX conv fallback. This PR Fixes #49640", "url": "https://github.com/vllm-project/vllm/pull/49688", "createdAt": "2026-07-24T07:35:29Z", "updatedAt": "2026-08-13T15:10:57Z", "timestamp": "2026-08-13T15:10:57Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "bug", "cpu" ], "author": "dineshchitlangia", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49790", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][NVIDIA] Route DSA models to the SM100 implementation", "text": "## What this does Point the DSA architectures at the SM100-optimized implementation in `vllm/models/deepseek_v32/`, which nothing on `main` currently references. - `GlmMoeDsaForCausalLM` now resolves to `vllm.models.deepseek_v32` instead of the generic `deepseek_v2`, and `DeepseekV32MTPModel` is registered so speculative decoding uses the matching MTP module. - `deepseek_v32/__init__.py` gains a platform gate: SM100 gets the optimized module, everything else (ROCm, XPU, pre-SM100 CUDA, CPU) falls back to the generic `deepseek_v2`. Previously it imported the SM100 module on any CUDA device, which is what broke the H100 CI job on #48597. - DeepSeek V3.2 is unaffected: `DeepseekV32ForCausalLM` still maps to `deepseek_v2`, and the MTP rewrite is gated on `model_type == \"glm_moe_dsa\"`. This is the enabler for the rest of the series — until it lands, the other PRs in the split are runtime no-ops. Part of the #48597 re-split — see that PR for the tracker and merge order. ## Benchmark 8×B300 TP8, `nvidia/GLM-5.2-NVFP4`, MTP=5, `VLLM_USE_V2_MODEL_RUNNER=1`, fp8 KV cache, 8192 input / 1024 output tokens, concurrency 1, 64 prompts. ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve $MODEL --served-model-name glm-5.2 \\ -tp 8 --port 8300 \\ --kv-cache-dtype fp8_e4m3 --max-model-len 16384 \\ --max-num-seqs 256 --max-num-batched-tokens 16384 \\ --no-enable-prefix-caching --gpu-memory-utilization 0.85 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":5}' \\ --kernel-config '{\"ir_op_priority\":{\"rms_norm\":[\"vllm_c\",\"native\"],\"fused_add_rms_norm\":[\"vllm_c\",\"native\"]},\"enable_flashinfer_autotune\":false}' ``` ### Performance vs main `main` @ `48a077e4cf` versus the same commit with all six branches of the split applied, each built and measured back-to-back on one node, 32 prompts at concurrency 1, `--kv-cache-dtype fp8_e4m3` throughout. | weights | build | output tok/s | total tok/s | median TPOT | GSM8K | | --- | --- | --- | --- | --- | --- | | NVFP4 | `main` | 447.4 | 4026.5 | 1.93 ms | 0.940 | | NVFP4 | full series | **541.9** | 4876.7 | **1.56 ms** | 0.948 | | FP8 | `main` | 365.8 | 3291.8 | 2.39 ms | 0.956 | | FP8 | full series | **403.1** | 3627.7 | **2.15 ms** | 0.946 | That is **+21.1%** output throughput on NVFP4 (TPOT -19.2%) and **+10.2%** on FP8 (TPOT -10.0%). The GSM8K spread is roughly one standard error at 500 questions (SE ~= 0.01), so it shows no accuracy regression in either direction rather than a real difference. AI assistance (Claude) was used for the split and the benchmarks; every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49790", "createdAt": "2026-07-25T08:16:16Z", "updatedAt": "2026-08-12T23:40:19Z", "timestamp": "2026-08-12T23:40:19Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "new-model", "speculative-decoding", "needs-rebase", "v1", "deepseek", "nvidia" ], "author": "zhou9402", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode][Perf] Fuse the MTP trailing all-reduce; local-argmax draft tokens", "text": "Two optimizations on the DeepSeek-V3.2 / GLM-5.2 MTP draft path. - **Fuse the trailing all-reduce into the final RMSNorm** on the non-sequence-parallel path, as the main model already does at layer boundaries. The sequence-parallel path is unchanged. - **Greedy draft tokens via vocab-parallel local argmax** (`get_top_tokens`), skipping the full-vocab all-gather in `compute_logits`. The proposer already probes for this via `use_local_argmax_reduction`; this adds the model side. Not a duplicate: an earlier revision also carried the `index_share_for_mtp_iteration` gates and the V2 speculator lifecycle hooks, both of which have since landed on main; that part is dropped. Nothing open overlaps with what remains. ```bash gh pr list --repo vllm-project/vllm --state open --search \"MTP all-reduce RMSNorm fuse\" gh pr list --repo vllm-project/vllm --state open --search \"local argmax draft tokens\" ``` ## Benchmarks 2×GB300, TP8 across 2 nodes, `nvidia/GLM-5.2-NVFP4`, MTP=5, `VLLM_USE_V2_MODEL_RUNNER=1`, fp8 KV cache, no prefix caching. The baseline arm is the same tree with the fusion reverted to an explicit `tensor_model_parallel_all_reduce` + `RMSNorm`; local argmax is toggled with `--speculative-config.use_local_argmax_reduction`. **Batch 1 (8192 in / 1024 out, c=1) — no measurable change.** Three runs per arm, output tok/s and MTP acceptance length: | arm | fusion | local argmax | output tok/s | acceptance length | | --- | :-: | :-: | --- | --- | | baseline | ✗ | ✗ | 456.7 / 451.6 / 463.3 | 4.78 / 4.70 / 4.86 | | +fusion | ✓ | ✗ | 487.0 / 459.0 / 461.3 | 5.05 / 4.76 / 4.79 | | +both | ✓ | ✓ | 460.1 / 472.8 / 470.2 | 4.80 / 4.95 / 4.90 | The arms overlap completely. At c=1 the saved all-gather is one vocab row and the fused all-reduce is one small tensor, so this is the expected result; median ITL sat at 9.08–9.12 ms across all nine runs. **Concurrency 64 (1024 in / 1024 out, 256 prompts) — this is where it pays.** One A/B pair on the same node pair: | arm | output tok/s | acceptance length | median ITL | | --- | --- | --- | --- | | baseline | 2923.2 | 4.86 | 31.83 ms | | +both | 3321.1 (**+13.6%**) | 4.69 | 31.32 ms | The gain cannot be attributed to speculation luck: acceptance length is 3.5% *lower* in the faster arm. Normalizing throughput by acceptance length (steps/s) gives +17.8%. At c=64 the draft's per-step full-vocab all-gather is 64 × 151k × 2 B ≈ 19 MB, five times per accepted token — that is what `get_top_tokens` removes. Caveats, stated plainly: the c=64 result is a single A/B pair, and a repeat was lost to cluster contention. Cross-node variability on this cluster is large (the +fusion arm measured 4226 tok/s on a different node pair with the same acceptance length as the baseline), so only within-pair comparisons above are meaningful — do not compare numbers across the two tables. ## Accuracy gsm8k, 5-shot, full 1319 questions, via `lm_eval --model local-completions`: | arm | flexible-extract | strict-match | | --- | --- | --- | | baseline | 0.9439 ± 0.0063 | 0.9424 ± 0.0064 | | +both | 0.9431 ± 0.0064 | 0.9409 ± 0.0065 | Unchanged within stderr, which is the point: the fused kernel accumulates in fp32 and local argmax breaks vocab-boundary ties differently, so both are output-affecting in principle. AI assistance (Claude) was used; every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49793", "createdAt": "2026-07-25T08:16:24Z", "updatedAt": "2026-08-13T16:22:17Z", "timestamp": "2026-08-13T16:22:17Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "speculative-decoding", "ready", "v1", "mrv2" ], "author": "zhou9402", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49819", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add Cohere2MoE Eagle3 auxiliary hidden states", "text": "## Purpose Enable Cohere2MoE target models to serve with compatible Eagle3 and DFlash draft models by exposing the auxiliary hidden states required by the speculator. I ~am working on training~ have released a DFlash speculator for North-Mini-Code-1.0 and realized that this was missing! Hopefully it helps others, ~especially when I upload the checkpoints.~ :) ### Update: Public Checkpoints As promised, the checkpoints are now public and there’s a turnkey runtime so this change can be exercised without a custom build. **Weights (auxiliary draft/speculator checkpoints, Apache-2.0):** - DFlash: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dflash - DSpark: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dspark **Runtime:** a thin overlay on official vLLM v0.27.1 carrying exactly this PR (#49819) plus PR #50937 (skip empty expert bias — required because North-Mini-Code w4a16 has an all-zero per-expert bias, causing vllm `0.27.1` to throw `AttributeError: 'RoutedExperts' object has no attribute 'w2_bias'`). Nothing is compiled from source; it’s two small pure-Python module overlays. - Repo: https://github.com/sdougbrown/north-mini-code-draft-runtime - Published multi-arch image: `ghcr.io/sdougbrown/north-mini-code runtime:v0.27.1-49819-50937` **Validation:** both DFlash (K3) and DSpark (K4) serve and run speculative decoding on NVIDIA GB10. ### Running the Draft Models You can use the pre-built docker runtime I put togther if you like, or apply the patches from https://github.com/sdougbrown/north-mini-code-draft-runtime/tree/main/patches to a fresh `0.27.1` wheel and then serve: ``` vllm serve sdougbrown/North-Mini-Code-1.0-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' ``` The easy docker path is: ``` docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ sdougbrown/North-Mini-Code-1.0-dspark \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --max-model-len 32768 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` Note that by default this fetches the weights from huggingface. If that's not desired, you can get fancier. Here's an example I would use on my local machine assuming `~/Code` is the parent folder for the runtime repo, and `~/Models` is where I've downloaded the weights: ``` # 1) Point the verifier at a local copy of the target and render a bundle: cd ~/Code/north-mini-code-draft-runtime VERIFIER_MODEL=\"$HOME/Models/North-Mini-Code-1.0-w4a16\" \\ RUNTIME_WORK=\"$HOME/Models/north-deploy\" \\ ./scripts/prepare-deployment-model.sh dspark # -> $HOME/Models/north-deploy/deploy/dspark # (config.json now has verifier.name_or_path -> local path; weights are symlinks) # 2) Docker: mount Models at the SAME path and pass the bundle path: docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ -v \"$HOME/Models:$HOME/Models\" \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ \"$HOME/Models/north-deploy/deploy/dspark\" \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` ## Implementation - implement `SupportsEagle3` for `Cohere2MoeForCausalLM` - capture embedding and post-layer boundaries using global layer indices - preserve Cohere2's parallel-residual behavior without adding the residual twice - preserve baseline model outputs when auxiliary states are disabled - preserve DFlash's `target_layer_ids -> auxiliary boundary + 1` semantics Eagle3, DFlash, and DSpark with pipeline parallelism are already rejected by the V1 GPU model runner. This change does not add auxiliary-state propagation between pipeline stages. ## Duplicate-work check PR #42078 added Cohere Eagle support, but it did not expose auxiliary hidden states from the Cohere2MoE target. Searches of open PRs found no overlapping implementation. ## Tests ```bash .venv/bin/pre-commit run --files \\ vllm/model_executor/models/cohere2_moe.py \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py .venv/bin/python -m pytest \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py -q ``` All pre-commit hooks passed. Pytest result: `15 passed`. ## Model evaluation The original `a1cad4f67` implementation was successfully exercised with: - AMD ROCm 7.2.4, TP=2, INT4/AutoGPTQ - NVIDIA GB10 CUDA 13, TP=1, W4A16 with MARLIN NVFP4 MoE - NVIDIA GB10 CUDA 13, TP=1, FP8 with TRITON FP8 MoE All three configurations completed target-model requests while exposing five finite auxiliary hidden states. The ROCm TP=2 run also verified byte-identical per-layer values across both ranks. These runs validated the target-side auxiliary-state path rather than end-to-end draft-token acceptance. ## AI assistance AI assistance was used to port and test this change. I manually reviewed every changed line and the test results.", "url": "https://github.com/vllm-project/vllm/pull/49819", "createdAt": "2026-07-25T17:06:03Z", "updatedAt": "2026-08-13T16:52:23Z", "timestamp": "2026-08-13T16:52:23Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "speculative-decoding", "v1", "cohere" ], "author": "sdougbrown", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49852", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MRV2][Multimodal] Enable encoder cuda graph for model runner v2", "text": "## Purpose - Enable encoder cuda graph on model runner v2. ## Test Plan ``` pytest -s -v tests/models/multimodal/generation/test_vit_cudagraph.py ``` ## Test Result All tests should pass ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49852", "createdAt": "2026-07-26T04:08:30Z", "updatedAt": "2026-08-13T09:00:53Z", "timestamp": "2026-08-13T09:00:53Z", "metrics": { "reactions": 5, "comments": 3 }, "labels": [ "ready", "v1", "nvidia", "mrv2" ], "author": "Isotr0py", "state": "open", "assignees": [ "shen-shanshan" ] }, { "id": "github:vllm-project/vllm:pull_request:49879", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[FEAT] Support fast engine recovery through weight cache", "text": "## Purpose Author: Siyu Liu @liusy58 Michael Qiu ; --- Engine restarts are dominated by weight loading from disk. This PR proposes a persistent per-GPU daemon holds post-quantized, TP-sharded weights in GPU memory; restarting engines map them via CUDA IPC (zero-copy) instead of reloading from disk. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49879", "createdAt": "2026-07-26T13:26:41Z", "updatedAt": "2026-08-13T13:35:11Z", "timestamp": "2026-08-13T13:35:11Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "needs-rebase", "nvidia" ], "author": "liusy58", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49925", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Switch to the Rock, Keep Python 3.12 and Ubuntu 22.04", "text": "## Purpose This PR switches to using The Rock 7.14 with wheels provided by The Rock while keeping Python 3.12 and Ubuntu 22.04 as-is. Need to wait for @Rohan138 to bump to torch 2.12, then will need to rerun testing. We could possibly add` /ready` to this and retry as needed when it becomes convenient. ## Test Plan Full CI runs. ## Test Result Currently failing groups that need to be addressed are: ``` MI300: Basic Models Tests (Other) Transformers Nightly Models (Single) MI355: Entrypoints Integration (Pooling) Language Models Tests (Standard) Multi-Modal Models (Extended Generation 3) ``` Keeping track in this BK build: https://buildkite.com/vllm/amd-ci/builds/11580/list ---
Essential Elements of an Effective PR Description Checklist - [X ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X ] The test plan, such as providing test command. - [ X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49925", "createdAt": "2026-07-27T02:32:21Z", "updatedAt": "2026-08-12T23:33:54Z", "timestamp": "2026-08-12T23:33:54Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "rocm", "ci/build" ], "author": "rasmith", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:49953", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][AITER] Add GDN long-prefill split-QKV fast path", "text": "## Motivation ROCm/aiter#4299 adds a channel-last HIP path for `causal_conv1d_fwd_split_qkv`, which can consume Qwen3.5's contiguous `[T, D]` projection through its logical `[D, T]` transpose without materializing a copy. vLLM already fuses its packed post-conv split, normalization, and gating, so dispatching the AITER op unconditionally regresses short prompts. This change uses the AITER path only for long BF16 Qwen3.5 GDN prefills where it is faster, and retains the existing Triton/FLA path everywhere else. A search of open vLLM PRs/issues did not find another change implementing this Qwen3.5 GDN channel-last split-QKV fast path. ## Summary - Probe for AITER's optional GDN prefill split-QKV op and gracefully fall back with older AITER builds. - Route non-interleaved Qwen3.5 BF16 prefills with at least 8192 tokens and width-4 convolution through the AITER channel-last kernel. - Add a fused Triton postprocessor that normalizes AITER's split Q/K outputs in place and computes GDN gating in the same launch; V is already in its final contiguous layout. - Precompute block-64 causal-conv metadata on the CPU to avoid a device-to-host synchronization at dispatch time while preserving block-8 metadata for the existing path. - Cover variable-length inputs, initial states, the vLLM null cache slot, long production-shaped inputs, and split-output postprocessing. ## AITER dependency - Required feature: ROCm/aiter#4299, now merged (`a96ee7b6e3c2688b099d6e32ec662bce8886efa7`). - End-to-end validation pin: AITER main `d19f33251400bbc21a3a49cc8db421f918716b93`. - AITER at this revision requires FlyDSL >= 0.2.4; the benchmark runtime pins FlyDSL 0.2.4. - Older AITER builds remain supported through the importability probe and use the existing Triton/FLA fallback. ## Test plan - [x] `pytest -q tests/kernels/test_fused_gdn_post_conv.py` — 68 passed - [x] `pytest -q tests/kernels/mamba/test_aiter_gdn_prefill.py` — 2 passed - [x] `pytest -q tests/v1/attention/test_gdn_metadata_builder.py` — 9 passed - [x] AITER `test_causal_conv1d_prefill_split_qkv.py -k hip_channel_last` — 11 passed - [x] `torch.compile(..., fullgraph=True)` smoke test for split-QKV postprocessing - [x] Ruff format/check, forbidden-import, torch-cuda-call, typos, and diff checks - [x] Qwen3.5-35B-A3B TP4 end-to-end serving A/B on MI308X/gfx942 - [x] Full 1319-question GSM8K, 5-shot, concurrency 256 Unit tests used `vllm/vllm-openai-rocm:v0.24.0`. End-to-end tests used that image with FlyDSL 0.2.4, vLLM baseline `afc94523c` vs PR `758e13f15`, and the same AITER `d19f332514` on both sides. ## Serve command Hardware: 4x AMD Instinct MI308X (gfx942), ROCm driver 6.16.13. Model: local BF16 `Qwen3.5-35B-A3B`. ```bash HIP_VISIBLE_DEVICES=0,1,2,3 VLLM_ROCM_USE_AITER=1 vllm serve /models/Qwen3.5-35B-A3B --served-model-name Qwen3.5-35B-A3B --host 0.0.0.0 --port 8000 --tensor-parallel-size 4 --trust-remote-code --dtype bfloat16 --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 256 --max-num-batched-tokens 32768 ``` ## End-to-end performance Each random workload was warmed up and then run three times with `temperature=0`, `ignore_eos=true`, fixed seed 20260727, and the same four GPUs. Values below are medians across the three runs. `ISL/OSL` are tokens per request. | Workload | Baseline total tok/s | PR total tok/s | Baseline median TTFT | PR median TTFT | Baseline p99 TTFT | PR p99 TTFT | |---|---:|---:|---:|---:|---:|---:| | ISL 4096 / OSL 1 / C1 / 8 prompts (fallback control) | 25,197.1 | 25,259.8 (+0.25%) | 161.37 ms | 161.64 ms (-0.17%) | 167.80 ms | 170.33 ms | | ISL 8192 / OSL 1 / C1 / 8 prompts (dispatch threshold) | 29,045.6 | 29,186.3 (+0.48%) | 283.37 ms | 280.48 ms (+1.03%) | 293.99 ms | 291.89 ms | | ISL 16384 / OSL 1 / C1 / 4 prompts (long prefill) | 25,250.4 | 25,486.3 (+0.93%) | 648.12 ms | 643.43 ms (+0.73%) | 662.01 ms | 653.29 ms | | ISL 8192 / OSL 128 / C256 / 256 prompts (loaded serving) | 25,113.6 | 26,989.7 (+7.47%) | 44.034 s | 40.399 s (+8.25%) | 78.858 s | 72.458 s | For the loaded workload, request throughput improved from 3.018 to 3.244 req/s (+7.47%), and mean TPOT improved from 323.73 to 306.22 ms (-5.41%). All runs completed with zero failed requests. The PR server log confirms `module_causal_conv1d_fwd_split_qkv` was built and invoked by the 9020-token probe. The previously reported `1.163x` is **operator latency speedup**, not TTFT, TPOT, or model throughput: for the Qwen3.5-35B-A3B TP4 per-rank shape (`D=2048`, BF16, width 4, T=8192), existing Triton conv + fused postprocess measured 0.1375 ms and AITER channel-last split-QKV + fused in-place postprocess measured 0.1182 ms. ## GSM8K accuracy Command: ```bash python tests/evals/gsm8k/gsm8k_eval.py --port 8000 --num-questions 1319 --num-shots 5 --max-tokens 256 --temperature 0 --seed 42 --max-concurrency 256 ``` | Revision | Correct | Accuracy | Invalid | Questions/s | |---|---:|---:|---:|---:| | Baseline `afc94523c` | 1177 / 1319 | 89.234% | 0% | 15.739 | | PR `758e13f15` | 1173 / 1319 | 88.931% | 0% | 15.658 | The observed delta is -0.303 percentage points (4 questions), with no invalid responses. A separate deterministic 9020-token long-prefill probe produced the correct answer (`42`) on both revisions; wording diverged after a common prefix because the fused path is not bitwise identical. ## AI assistance disclosure AI assistance was used for implementation support, benchmark automation, and result summarization. The human submitter remains responsible for reviewing the diff and validating the reported results before merge.", "url": "https://github.com/vllm-project/vllm/pull/49953", "createdAt": "2026-07-27T08:12:24Z", "updatedAt": "2026-08-13T07:12:49Z", "timestamp": "2026-08-13T07:12:49Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "v1" ], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50017", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] [bugfix] Chunked prefill paged decode masked load perf", "text": "## Purpose Observed 10-15% latency performance regression for serving Qwen/Qwen3-30B-A3B-Thinking-2507; trace revealed that `kernel_paged_attention_2d` was the culprit, taking 1.33x as long on average on v0.25.0 vs v0.24.0. #47305 fixed correctness but introduced performance drop because of universally applied masking. Edited masking so that it is only enacted during last block, when token id can be greater than sequence length. ## Test Plan ### Latency Ensure that fixes solve benchmark latency issue. `VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 --dtype float16 -tp 1 --no-enable-prefix-caching --disable-uvicorn-access-log` `vllm bench serve --model Qwen/Qwen3-30B-A3B-Thinking-2507 --percentile-metrics tpot,ttft,itl,e2el --dataset-name random --ignore-eos --temperature 0 --max-concurrency 1 --num-prompts 10 --random-input-len 1024 --random-output-len 1024` ### Accuracy Ensure accuracy remains using gsm8k benchmark. lm_eval --model local-completions --model_args model=Qwen/Qwen3-30B-A3B-Thinking-2507,base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=256,max_retries=10,max_gen_toks=2048 --batch_size auto --tasks gsm8k --num_fewshot 5 --limit 1000 ## Test Result ### Latency | Metric | v0.24.0 baseline | v0.25.0 (before fix) | v0.25.0 + fix | |---|---|---|---| | Output tok/s | 91.33 | 79.76 | **97.08** | | Mean TPOT (ms) | 10.69 | 12.25 | **10.23** | | Median E2EL (ms) | 11006 | 12613 | **10546** | ### Accuracy gsm8k, 5-shot, `--limit 1000`, same serve config: | Filter | v0.24.0 baseline | v0.25.0 + fix | |---|---|---| | flexible-extract | 0.901 ± 0.0094 | 0.913 ± 0.0089 | | strict-match | 0.920 ± 0.0086 | 0.917 ± 0.0087 | Deltas (+1.2pt flexible, −0.3pt strict) are within stderr. --- No other PRs address this performance regression. --- This PR was prepared with AI assistance (Claude). All changes were reviewed and the test/eval commands were run by Andy.", "url": "https://github.com/vllm-project/vllm/pull/50017", "createdAt": "2026-07-27T16:49:53Z", "updatedAt": "2026-08-12T23:04:02Z", "timestamp": "2026-08-12T23:04:02Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "rocm", "ready", "v1" ], "author": "afriedri", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Bound accepted-token state lookups in GDN/KDA spec decode", "text": "## Summary On a hybrid GDN model (Qwen3.5 / Qwen3.6) with MTP speculative decoding and prefix caching (`--mamba-cache-mode align`), the engine dies with `CUDA error: unspecified launch failure` within 7-10 requests of agent-shaped traffic. The GPU faults, not the runtime: the signature is an SM address exception (`Xid 13`, `ESR 0x404000`) or an MMU fault (`Xid 31`), depending on whether the wild address happens to be mapped. This is the crash half of the problems reported around hybrid-Mamba + MTP. It is distinct from the prefix-cache corruption in #43559 (the coordinator-level EAGLE cache-peek gating for Mamba), which is already handled on current `main`. This PR does not touch that path and does not claim to fix #43559; it fixes a separate GPU fault in two kernels downstream. ## Root cause: unchecked accepted-count-derived indices Speculative decoding produces a per-request accepted-token count. Multiple GPU state consumers turn that count into an array index without bounding it, so a count that is stale, zero, or too large indexes outside its tensor and yields a wild address the GPU then dereferences. **Site 1** `fused_recurrent_gated_delta_rule_fwd_kernel` (`fused_recurrent.py`): ```python i_t = tl.load(num_accepted_tokens + i_n).to(tl.int64) - 1 # unbounded state_idx = tl.load(ssm_state_indices + i_n * stride_indices_seq + i_t) if state_idx <= 0: return # positive garbage passes p_h0 = h0 + state_idx * stride_init_state_token # dereferenced ``` `i_t` (= count - 1) is unbounded against a `stride_indices_seq`-column tensor. A zero accepted count gives `i_t == -1`, a read before this request's row (before the tensor for `i_n == 0`); a stale or too-large count reads past the row. The `state_idx <= 0` guard only rejects non-positive values, so an out-of-range read that returns a garbage positive int flows into the address math and faults the SM. **Site 2** `_copy_mamba_state_block` (`mamba_utils.py`): the block-table columns derive from the same count and index the per-request block-table row unbounded; the loaded block id becomes `state_base_addr + block_id * state_block_stride`, which is then read and written. The two sites fire under different loads. A light decode load exercises only Site 1; a heavy cache-transition load (long prefills, A->B->A prefix reuse) also drives Site 2. Both must be bounded. ## The fix Both loads are masked to the valid range, so an out-of-range index falls into the existing invalid-state path instead of producing an address. No stream-ordering change, no device sync, no measurable throughput cost. ## Test plan RTX 5090 (sm_120), Qwen3.6-27B NVFP4, TP=1, `--enforce-eager --max-model-len 16384`, MTP 3 (`qwen3_5_mtp`), prefix caching on, `align` mode. | build | crash (Xid) | corruption (A->B->A probe) | |---|---|---| | stock | dies at 7-10 requests, Xid every run | 0/N reproduced (already fixed on main) | | Site 1 fix alone | survives light load; still crashes on the first heavy probe (`Xid 31`) | 0/N reproduced | | **both fixes** | **68 heavy probes + a 39-minute soak, 0 new Xid** | **0/68 reproduced** | The A->B->A probe issues an agent-shaped sequence (long prefill, prefix reuse, 20 tool schemas) and reports whether a poisoned prefix reappears; 68/68 returned a clean verdict, which also confirms the out-of-range early-return does not drop a needed state copy. Warm throughput and MTP acceptance length (3.98-4.00 of 4) are unchanged. ## Note: a separate, still-open livelock (#49203) Independently of this crash, the same stack can occasionally hang: engine alive, `/v1/models` answering, but the in-flight request stuck at 0% GPU util with no Xid. It is rare and timing-variable (seen once, then not across the 68 probes here) and matches open issue #49203. This PR does not address it and does not claim to; the 68/68 clean-verdict result rules out this change as a cause. ## Follow-up bounds audit A follow-up audit expanded the same fail-closed rule to the remaining consumers in this path: - Both FLA wrappers now zero rejected output deterministically instead of returning with new_empty storage visible downstream. - _causal_conv1d_update_kernel now bounds the accepted-count offset before state address math; invalid active rows produce zero output and leave state unchanged. - GPU regressions cover too-small/too-large accepted counts, NULL block IDs, source/destination columns crossing the block-table row, and temporal-bias overflow. Validation after this follow-up: full pre-commit passed; on an RTX 5090, 186 FLA/causal-conv kernel tests and all 21 fused Mamba postprocess tests passed (207 total). ## Kimi K3 KDA expansion The same accepted-count-derived state selection pattern also existed in Kimi K3 KDA fused recurrent decode, in both the NVIDIA and AMD vendored kernels. This PR now masks that initial state-index load to the request row, zeroes all invalid-count output tokens, preserves state on invalid counts, and releases NVIDIA PDL dependents before the new empty/invalid early returns. Additional validation on mini-beast RTX 5090: - Patched KDA invalid-count test: 4/4 passed across NVIDIA and AMD implementations (`num_accepted` 0 and 4). - Existing KDA spec-decode correctness plus the new invalid-count target: 12/12 passed. - Negative control at previous PR head `e7f66b199` with only the new test added: 4/4 failed on old KDA for NVIDIA and AMD, proving the regression is non-vacuous. ## AI assistance disclosure OpenAI Codex assisted with the follow-up bounds audit, implementation, and regression-test generation. The submitter owns the conclusions and final review. ## Mamba2 selective-state expansion The Mamba2 selective-state-update kernel also derives its initial state-slot lookup from `num_accepted_tokens - 1`. It already clamps the lower side, but had no upper row bound. This update preserves that zero-count behavior and makes an oversized count fail closed: it writes zero output and returns before reading or writing state. RTX 5090 red/green regression: a count one past a three-column state row made the prior source select an adjacent row and emit nonzero output; the new kernel passes by emitting zero output and preserving the complete state tensor. Pre-commit passes for both changed files.", "url": "https://github.com/vllm-project/vllm/pull/50021", "createdAt": "2026-07-27T17:29:26Z", "updatedAt": "2026-08-13T03:58:49Z", "timestamp": "2026-08-13T03:58:49Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "bug", "v1", "nvidia", "kimi", "k3" ], "author": "amittell", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50062", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP", "text": "## Summary This PR adds the scheduler and KV-cache-manager support required for multi-module MTP (one MTP module per speculative step, e.g. Inkling's 8-depth checkpoint). It is the companion to #48892, which introduced the speculator itself to Model Runner V2. The core property this PR protects: the multi-module drafter reads **ahead** of the computed tokens during prefill. MTP module `m` computes position `p`'s KV from token `p + m + 1`, so at every chunked-prefill boundary the drafter consumes the next `num_speculative_tokens` (N) known prompt tokens — the \"prefill lookahead\" — to write exact KVs into all modules. That lookahead, plus the decode-time rejection re-prefill (which rewrites up to N−1 trailing positions), creates three hazards that the scheduler and KV cache layer must handle: chunk boundaries landing where lookahead tokens don't exist, caching/freeing KV that can still change, and prefix-cache hits serving KV that encodes another request's continuation. ## Changes ### Config (`vllm/config/speculative.py`) - Inkling MTP hf-config override now exposes **all** checkpoint MTP depths (`n_predict = num_nextn_predict_layers` instead of clamping to 1), and the \"exactly one speculative token\" restriction is removed. Module `i` drafts speculative token `i`. ### Speculator selection (`vllm/v1/worker/gpu/spec_decode/__init__.py`) - `init_speculator` routes to `MultiModuleMTPSpeculator` when `use_multi_module_mtp()` (method `\"mtp\"` with >1 usable MTP layer). ### Scheduler (`vllm/v1/core/sched/scheduler.py`) - New unified field `num_prefill_lookahead`: how many positions past the computed tokens the drafter reads during prefill (N for multi-module MTP, 1 for other eagle-family drafters, 0 without spec decode). All three consumers below are projections of it. - **`_reserve_prefill_lookahead`** (both scheduling loops, after encoder truncation): never end a prefill chunk with `0 < remaining < N` tokens before the prefill end — either finish the prefill or leave ≥ N for the next chunk. Without this, a boundary near the prompt end has no real lookahead tokens; the drafter falls back to sampled drafts and the trailing modules' KVs at those positions are polluted permanently (they fall outside every future query window, so nothing rewrites them). Vacuous no-op for eagle-family (lookahead 1). - **Encoder scheduling shift generalized**: `shift_computed_tokens` to `_try_schedule_encoder_inputs` is now `num_prefill_lookahead` (previously the hardcoded eagle `+1`). MM spans starting inside the lookahead window get encoded one chunk early, so the drafter's future-token embeddings are available; when the encoder budget can't cover a span, the existing rollback lands the boundary at `span_start − N`, keeping the lookahead window all-text. (The runner-side consumption of these early embeddings lands in a follow-up PR; until then MM placeholder lookahead tokens use text embeddings.) - Passes `num_prefill_lookahead` to `KVCacheManager`. ### KV cache manager / coordinator (`kv_cache_manager.py`, `kv_cache_coordinator.py`, `single_type_kv_cache_manager.py`) - `num_prefill_lookahead` is threaded into `KVCacheCoordinator`, which derives `num_reprefillable_tokens = max(0, lookahead − 1)`. - **Delayed caching**: `cache_blocks` only hash-registers tokens up to `num_computed − num_reprefillable_tokens` (mirrored in the hybrid coordinator's EAGLE lookahead-block eligibility). During decode, rejection re-prefill can rewrite the last N−1 tokens' draft KVs; registering them earlier would expose unverified KV to other requests and mutate blocks after they are shared. - **Prefix-hit soundness assert**: for every EAGLE-flagged group, `block_size >= num_prefill_lookahead`. The last N slots of *any* cached prefix hold draft KVs computed from tokens past the block hash (the writer's continuation — lookahead tokens at a chunk boundary, or sampled/draft tokens later). The pre-existing EAGLE last-block drop already recomputes the trailing block on every hit — with the new request's own lookahead tokens, making the rewrite exact — but a single dropped block only covers all N polluted slots if the block size is large enough. Rather than generalizing the drop to multiple blocks, we assert (real configs use block 16/64 with N ≤ 8). ### Sliding-window retention (`kv_cache_interface.py`, `kv_cache_utils.py`, `single_type_kv_cache_manager.py`) - New `SlidingWindowSpec.extra_retained_tokens` (tagged as N−1 for multi-module MTP in `get_kv_cache_configs`): the SWA free boundary lags by N−1 tokens, and `max_memory_usage_bytes` accounts for the extra retained blocks. Rejection re-prefill recomputes positions up to N−1 behind the tip, and each recomputed position needs its full attention window; without the lag, those windows reach into already-freed (null) blocks and the corrected KVs would be computed from garbage. Single source of truth on the spec keeps pool sizing, the admission cap, and block eviction consistent. Prior spec methods never needed this knob because no drafter re-processed positions behind the tip (eagle's ±1 is a token-id shift, not a position shift). ## Why this is not duplicating an existing PR Multi-module MTP is not supported by any open PR; the eagle/single-module paths this builds on (`drop_eagle_block`, encoder shift, `use_eagle` plumbing) are extended in place rather than duplicated. ## Testing - `tests/v1/core/test_scheduler.py` — 137 passed (includes the eagle encoder-shift regression test, chunked-prefill, and preemption paths). - `tests/v1/core/test_prefix_caching.py` — 89 passed. - `tests/config/test_speculative_draft_hf_overrides.py` — updated for the all-depths Inkling override. - Direct construction check of the new assert: accepts `block_size=16, N=8` and spec-decode-off; rejects `block_size=4, N=8` with a clear message. - End-to-end acceptance-rate evaluation with the Inkling multi-module MTP model (chunked prefill + prefix caching enabled): [results to be added]. AI assistance (Claude) was used for implementation and review of this PR; all changes were human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50062", "createdAt": "2026-07-28T01:07:12Z", "updatedAt": "2026-08-13T09:03:51Z", "timestamp": "2026-08-13T09:03:51Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "ready", "v1", "mrv2" ], "author": "TheEpicDolphin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add Kimi K3 MoE support to benchmark_moe.py", "text": "## Purpose `benchmarks/kernels/benchmark_moe.py` cannot tune the fused MoE Triton kernel for Kimi K3. `get_model_params()` does not recognize `KimiK3ForConditionalGeneration`, so it falls through to the Mixtral default branch and raises `AttributeError` (Kimi's config has no `num_local_experts`). Kimi K3 is multimodal and stores its MoE parameters in a nested `KimiLinearConfig` `text_config`, and it exposes the routing top-k as `num_experts_per_token` rather than the more common `num_experts_per_tok` used by the existing handlers. This adds a dedicated branch, following the same pattern as the Qwen3.5 handler added in #37068. It also covers the text-only `KimiLinearForCausalLM`, for which `get_text_config()` returns the config itself. ## Test Plan Parse the released Kimi K3 config and confirm the extracted MoE parameters: ```python from vllm.transformers_utils.config import get_config from benchmark_moe import get_model_params cfg = get_config(\"\", trust_remote_code=True) print(get_model_params(cfg)) ``` ## Test Result ``` architectures = ['KimiK3ForConditionalGeneration'] (text_config: KimiLinearConfig) get_model_params -> (E=896, topk=16, moe_intermediate_size=3072, hidden_size=7168) ``` The values match the model config. With `--tp-size 8 --enable-expert-parallel` this yields `E_local=112` and shard `N=6144`. Before this change the script raised `AttributeError` before tuning could start.", "url": "https://github.com/vllm-project/vllm/pull/50082", "createdAt": "2026-07-28T04:27:24Z", "updatedAt": "2026-08-13T16:28:14Z", "timestamp": "2026-08-13T16:28:14Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "performance", "ready", "kimi", "k3" ], "author": "vanshbhatia-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50156", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Cohere][Spec Decode] Add CohereEagleProposer to support multi layer eagle drafts", "text": "Follow up to #42078 with model released. Cohere Eagle draft models have multipe layers and HybridKVCacheManager split these layers across multiple KV cache groups, which the base EagleProposer does not support (it assumes a single group with one shared block table / slot mapping). This PR adds CohereEagleProposer, which tracks per-group block tables and slot mapping buffers so each group's draft layers read/write their own KV cache correctly. > This is only relevant for MRv1. MRv2 natively supports draft layers spread across different KV groups. This PR also includes a few misc fixes to cohere model definitions. Eagle draft model for `CohereLabs/North-Mini-Code-1.0` uploaded to HF here: `CohereLabs/North-Mini-Code-1.0-eagle`. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50156", "createdAt": "2026-07-28T15:33:31Z", "updatedAt": "2026-08-13T16:53:44Z", "timestamp": "2026-08-13T16:53:44Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "speculative-decoding", "v1", "verified", "cohere" ], "author": "kkt-cohere", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50169", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Fix KV cache allocation for sliding-window drafters and local-attention pool sizing", "text": "## Purpose Three fixes around KV cache allocation for local-attention models and spec-decode drafters. Found via Laguna-S-2.1 + DFlash, but only one commit is model-specific. On current main (GB10, Laguna-S-2.1-NVFP4, DFlash, fp8 KV, max_model_len 131072, max_num_seqs 16, 23.1 GB KV pool) the series takes the KV pool from 415,517 to 871,430 tokens (2.1x), with draft acceptance unchanged. 1. Drafter layers of EAGLE-family speculators get scattered across KV cache groups by the strided group assignment in `_get_kv_cache_groups_uniform_page_size`. Fix: identify drafter layers (index >= target layer count, gated on `use_eagle()`) and give dedicated groups to the specs whose drafter layers would otherwise share a group with target layers. Sharing corrupts the drafter's absolute-slot context writes, which resolve through the target's block table: on Laguna-S-2.1 draft acceptance falls to 0.29% versus 26.67% with a dedicated group, at an identical pool size. On the V1 model runner the same layout instead trips the assertion in `SpecDecodeBaseProposer.validate_same_kv_cache_group` at startup. Separation is conditional because regrouping changes the group-size heuristic's inputs: separating regardless cut Qwen3.5-9B's pool from 863k to 185k tokens. Works for any PP size; the strided layout still applies to target layers. 2. `laguna_dflash.py` cleared the drafter's `attn.sliding_window`, forcing FullAttentionSpec on all 6 drafter layers (measured 38% of the pool for KV outside the window the drafter never reads). The `DFlashQwen3Model` base class already supports windowed draft layers; the clearing appears to be a workaround for the group-sharing corruption fixed in commit 1. Qwen3.5 DFlash avoids that corruption because its per-layer specs already give one group per layer, so its drafter never shares. Fix: keep the checkpoint-declared window. 3. `SlidingWindowSpec` and `ChunkedLocalAttentionSpec` charge the global `max_in_flight_tokens` bound to every request when sizing the pool. Since `sum_r (window - 1 + in_flight_r) <= R*(window - 1) + max_in_flight_tokens`, per-request sizing only needs the request's share of the allowance. Fix: amortize over `max_num_seqs`; the runtime admission gate is unchanged, and sizing is identical to today at `max_num_seqs = 1`. Without this, fixes 1 and 2 barely show (347k to 352k tokens), because sliding layers are charged almost like full attention. ## Scope - Commit 1: any EAGLE-family drafter (eagle/eagle3/mtp/dflash/dspark) over a hybrid target. All in-tree EAGLE-family implementations register drafter layers with indices continuing after the target's (`start_layer_id=target_layer_num` / `mtp_start_layer_idx=num_hidden_layers`), so identification cannot misclassify a target layer; unparseable names are skipped. No speculative config means grouping is unchanged. - Commit 2: Laguna only. Removes a divergence from its own base class. - Commit 3: every model using SlidingWindowSpec (Gemma 2/3/4, gpt-oss, Command R, EXAONE 4, and others), SlidingWindowMLASpec by inheritance, and ChunkedLocalAttentionSpec (Llama 4 style local attention). Broadest behavior change of the three; can be split into its own PR if preferred. The chunked local part is unit-tested only (no such model available for E2E here). ## Not duplicating existing PRs - #35062: same root cause as commit 1 (referenced in the commit message), stalled on rebase since June. Differences: per-spec drafter groups (a single merged group would mix specs for hybrid drafters), identification gated to `use_eagle()` (independent `draft_model` drafters number from 0), rebased with tests. Happy to defer to a revived #35062; will comment there. - #46032: relaxes the same-group invariant for `DraftModelProposer` only. EAGLE-family proposers still assert a single group. Complementary. - Nothing open covers commits 2 or 3. ## Test plan and results - `pytest tests/v1/core/test_kv_cache_utils.py tests/v1/core/test_swa_inflight_window_free.py tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/test_kv_cache_spec_registry.py`: 131 passed, 0 failed (needs a GPU; some device-inference tests fail in a GPU-less container with or without this change). New tests: `test_drafter_layers_get_dedicated_kv_cache_group`, `test_drafter_layer_identification_requires_eagle_family`, `test_sliding_window_pool_sizing_amortizes_in_flight_tokens`, `test_chunked_local_pool_sizing_amortizes_in_flight_tokens`. Two existing tests that hardcoded the old sizing constants now pin `max_num_seqs=1`, where old and new sizing coincide. - Validated on a second real model, not simulated: `Intel/Qwen3.5-9B-int4-AutoRound` with `z-lab/Qwen3.5-9B-DFlash` at 30b0d54 (hybrid drafter, 5 sliding + 1 full; auto-selects the V2 runner). Pool 863,023 stock vs 864,749 with this PR, and identical spec-decode counters, i.e. exactly neutral. Synthetic `KVCacheSpec` fixtures were not predictive here, indicating no change for a configuration that in fact lost 79% of the pool, so grouping claims are traced inside the live engine. - `ruff check` and `ruff format --check` clean on all changed files. - Test env: `vllm/vllm-openai:nightly` (0.26.1rc1.dev18+gd223c900d), branch rebased onto that exact commit, changed files mounted over the installed package (aarch64 box, no source install). Should be re-run in a source install before merge. E2E on GB10, Laguna-S-2.1-NVFP4 + DFlash, fp8 KV, `max_model_len` 131072, `max_num_seqs` 16, 23.1 GB KV pool, V2 runner: | | pool tokens | max conc @128k | |---|---|---| | stock main | 415,517 | 3.17x | | + commits 1-2 | 466,211 | 3.56x | | + commit 3 | 871,430 | 6.65x | Acceptance, paired (identical seeded prompts in both arms, 15 speculative tokens): | context | stock | this PR | paired diff | |---|---:|---:|---:| | ~59 tokens (below the 512 window, no eviction) | 15.20% | 16.73% | +1.53pp (t=1.12) | | ~7,050 tokens (well above the window) | 17.82% | 23.48% | +5.65pp (t=0.84) | Both favour this PR and neither is significant, so I read this as no acceptance regression. Power caveat: the long-context paired sd is 13.5pp, so that row shows no evidence of a cost rather than proving neutrality. An earlier unpaired comparison suggested a ~2.8pp deficit; it did not survive pairing, because between-boot and between-prompt variance dominate on this checkpoint (greedy is non-deterministic from atomic-add reductions). Coherence (greedy known-answer) passes on every configuration. A controlled prefill/decode throughput A/B for commit 3 alone (on the prior 0.25 nightly, same box) showed no change and zero preemptions; throughput on this image was only sanity-checked. ## Disclosure AI assistance (Claude Opus 5 via Claude Code) was used to investigate, draft, and test this change. The submitter has reviewed every line and re-run the tests. _Edited: corrected the separation criterion to sharing-only after it was found to cost capacity, and replaced simulated and mismatched-length measurements with paired real-model ones._", "url": "https://github.com/vllm-project/vllm/pull/50169", "createdAt": "2026-07-28T18:02:11Z", "updatedAt": "2026-08-13T11:26:54Z", "timestamp": "2026-08-13T11:26:54Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "v1" ], "author": "iggerask", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50174", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup", "text": "## Description This PR extends the shared JIT warmup infrastructure with provider registration and centralized orchestration. It builds on #49315 and the contract described in #47456. For more details, see parent (draft) PR: https://github.com/vllm-project/vllm/pull/49627 and tracking list issue https://github.com/vllm-project/vllm/issues/49349 ``` JIT kernel warmup (5 compile keys): 100%|██████████████████████████████████| 1/1 [00:00<00:00, 259.93kernel/s, ComputeSlotMappingKernel (5 keys)] (Worker_TP0 pid=425015) INFO 08-11 15:43:58 [kernel_warmup.py:120] JIT kernel warmup finished in 0.01s. ``` ## Motivation - Discover warmup providers from the model and backend objects actually instantiated. - Avoid model-name-based lists and warming kernels for inactive backends. - Run compilation under `kernel_warmup()` logging, ordering, progress reporting, and exception handling. - Respect `enable_jit_warmup`. - Keep model construction and runtime execution separate from startup compilation. ## What Changed - Added a scoped registry used while model and KV-cache components are constructed. - Added centralized provider deduplication and warmup execution. - Added kernel and compile-key progress reporting. - Added shared Triton helpers and an initial block-table migration. - Added contributor documentation for the warmup contract. - Added registry, orchestration, and AST-expansion tests.", "url": "https://github.com/vllm-project/vllm/pull/50174", "createdAt": "2026-07-28T19:24:41Z", "updatedAt": "2026-08-13T15:10:36Z", "timestamp": "2026-08-13T15:10:36Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "v1", "cpu", "mrv2" ], "author": "LopezCastroRoberto", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50180", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Skip generic required/named tool grammar for parsers that set supports_required_and_named", "text": "# Issue #49981 — `tool_choice: \"required\"` xgrammar FSM crash / infinite hang (GLM-5.2) ## 1. Root cause `ToolParser.supports_required_and_named` is the flag a tool parser sets to say *\"my model does not emit the standard tool-call JSON; do not use the generic JSON-based `required`/named handling for me.\"* GLM parsers (`Glm47MoeModelToolParser`, used for `--tool-call-parser glm45/glm47`, i.e. the GLM-4.5/4.7/5.x family) set `supports_required_and_named = False` because those models emit XML-ish tool calls plus thinking special tokens. The **parsing** side honoured that flag — `DelegatingParser.parse` treats `required`/named as `auto` and routes through the parser's `extract_tool_calls` (`vllm/parser/abstract_parser.py:433-447`). The **constraint** side did not. `ToolParser.adjust_request` unconditionally called `get_json_schema_from_tools()` and installed the resulting `{\"type\": \"array\", \"minItems\": 1, \"items\": {\"anyOf\": [...]}}` schema as `request.structured_outputs.json`, i.e. as a hard xgrammar decoding constraint. So for GLM with `tool_choice: \"required\"` the two halves disagreed: decoding was forced into a JSON grammar the model never produces, while the output was then handed to the XML parser that could not have read that JSON anyway. The failure modes reported in the issue follow directly: - **500 / `Failed to advance FSM ... grammar rejected tokens [..., 154842, ...]`** — GLM's first tokens are its native thinking/tool special tokens (154842 is one of them). The JSON grammar has no transition for them, so the FSM cannot advance and the request is terminated. - **Infinite hang with no `max_tokens`** — the array schema only reaches an accepting state once a valid JSON array is closed. With several tools the model keeps being pushed into a grammar it cannot satisfy and never emits a legal stop, so generation runs until the length cap. With no `max_tokens` set there is no cap. - **`auto` works** — no JSON schema is installed for `auto`. - **A specific named function \"works\"** in the reporter's environment because the single-function schema is small enough that the model can sometimes be coerced into it; it is broken for the same reason. Note this is only reachable when the xgrammar structural-tag path is *not* taken. With the default `VLLM_ENFORCE_STRICT_TOOL_CALLING=1`, `_apply_structural_tag` installs the `glm_4_7` builtin structural tag first and `adjust_request` returns early. Setting `VLLM_ENFORCE_STRICT_TOOL_CALLING=0` (or using any `supports_required_and_named = False` parser with no structural tag wired up, e.g. `InklingEngineToolParser`) drops into the broken JSON-schema path. This is the same defect class already fixed one-parser-at-a-time for Gemma4 (`Gemma4EngineToolParser.adjust_request`, regression after #45588), Poolside (`PoolsideV1ToolParser.adjust_request`) and the Rust bridge (`RustToolParser.adjust_request`) — each carries its own copy of the skip. GLM and Inkling never got one. ## 2. The fix `vllm/tool_parsers/abstract_tool_parser.py` — in `ToolParser.adjust_request`, return the request unchanged when `supports_required_and_named` is `False` and `tool_choice` is `\"required\"` or a named function, before the JSON schema is built and installed. Why in the base class rather than adding a fourth per-parser override: the flag already *means* \"the generic JSON required/named path does not apply to me\", and the parsing side reads it that way. Enforcing it in one place makes the two halves consistent, fixes GLM and Inkling together, and stops the next `supports_required_and_named = False` parser from re-introducing the bug. The existing per-parser overrides are left alone — they also flip `skip_special_tokens`, and they have their own regression tests. Behaviour after the fix: `required`/named on these parsers is unconstrained and parsed as `auto`, which is exactly what the serving layer already assumed. GLM still gets `skip_special_tokens = False` from `ParserEngineToolAdapter.adjust_request` → `ParserEngine.adjust_request`, so its native delimiters still reach the extractor. ## 3. Files changed | File | Change | |---|---| | `vllm/tool_parsers/abstract_tool_parser.py` | Skip the forced tool JSON schema for `required`/named when `supports_required_and_named` is `False`; import `ChatCompletionNamedToolChoiceParam` and `ToolChoiceFunction`. | | `tests/tool_parsers/test_structural_tag_registry.py` | Added `test_parser_without_required_and_named_support_skips_schema_constraints` (parametrized `required` / named), next to the existing `test_non_structural_tag_parser_uses_schema_constraints` that pins the opposite case. | | `NOTES.md` | This file. | ## 4. Risk / uncertainty - **Behaviour change:** `required` on these parsers is no longer grammar-forced. It was never actually enforcing anything useful there — it was producing output the model's own parser rejects — but a caller who relied on the constraint existing will now see an unconstrained generation. This matches what Gemma4/Poolside/Rust already do, and the default strict-tool-calling path (structural tag) is untouched and still enforces `required` properly. - **Mistral:** `MistralParser` flips `supports_required_and_named` to `False` on the *instance*, but only inside `parse`/`parse_delta`, which run after `adjust_request`; the parser is constructed per request in the renderer, so the new branch never fires for Mistral. Verified by `tests/tool_parsers/test_mistral_tool_parser.py` passing unchanged. - **Unverified:** I could not run GLM-5.2-NVFP4 end-to-end — no GPU in this environment. The 500/hang reproduction and its disappearance are argued from the code path and the reporter's token IDs, not observed on hardware. Since the change only removes a decoding constraint (never adds one) and does not touch model code, no model-quality eval is implicated; the on-GPU check worth doing before merge is a GLM-5.2 request with `tool_choice: \"required\"`, 4+ tools, `enable_thinking: false` and no `max_tokens`, under `VLLM_ENFORCE_STRICT_TOOL_CALLING=0`. ## 5. How I verified it Environment: CPU-only venv (`uv venv --python 3.12`, CPU torch 2.13 + test deps); no GPU, so only the non-GPU unit suites are runnable. ``` .venv/bin/python -m pytest tests/tool_parsers/test_structural_tag_registry.py -q → 54 passed ``` Confirmed the new test actually catches the bug — with the `abstract_tool_parser.py` change stashed and the test kept: ``` FAILED ...::test_parser_without_required_and_named_support_skips_schema_constraints[required] FAILED ...::test_parser_without_required_and_named_support_skips_schema_constraints[named] 2 failed, 52 passed ``` Regression sweep over every suite that touches tool parsing / request adjustment: ``` .venv/bin/python -m pytest tests/tool_parsers/ tests/parser/ \\ tests/tool_use/test_gemma4_responses_adjust_request.py -q → 4710 passed, 3 skipped, 34 xfailed, 15 errors .venv/bin/python -m pytest \\ tests/entrypoints/openai/responses/test_parsable_context_unit.py \\ tests/tool_use/test_chat_completion_request_validations.py \\ tests/tool_use/test_responses_request_validations.py -q → 40 passed ``` The 15 errors are all in `tests/tool_parsers/test_llama3_json_tool_parser.py` and are environment-only: fixture setup gets HTTP 401 fetching the gated `meta-llama/Llama-3.2-1B-Instruct` config from Hugging Face (no HF token here). They fail identically before and after the change. Lint: ``` ruff check → All checks passed! ruff format --diff → 2 files already formatted ``` AI assistance was used for this investigation and patch.", "url": "https://github.com/vllm-project/vllm/pull/50180", "createdAt": "2026-07-28T20:37:49Z", "updatedAt": "2026-08-13T03:36:30Z", "timestamp": "2026-08-13T03:36:30Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "tool-calling" ], "author": "pjdurden", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50221", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(security): enforce audio decode duration limit in NanoNemotronVL", "text": "The _extract_audio_from_videos method called load_audio_pyav without max_duration_s, allowing a small compressed video to decompress into gigabytes of PCM and crash the server via OOM. Pass VLLM_MAX_AUDIO_DECODE_DURATION_S to match the safeguard already used by AudioMediaIO. This should be merged only when https://github.com/vllm-project/vllm/pull/49948 branch) is merged", "url": "https://github.com/vllm-project/vllm/pull/50221", "createdAt": "2026-07-29T07:37:28Z", "updatedAt": "2026-08-13T10:31:56Z", "timestamp": "2026-08-13T10:31:56Z", "metrics": { "reactions": 2, "comments": 16 }, "labels": [ "ready", "multi-modality" ], "author": "jperezdealgaba", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50268", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][AMD] Enable fused bf16→fp32 router GEMM on ROCm", "text": "## Purpose This pull request addresses issue #50267 by enabling fused bf16→fp32 GEMM operations on AMD ROCm hardware for MoE router gates. ## Problem Statement The MoE router gate requires `out_dtype=torch.float32` for the `grouped_topk` operation. However, on ROCm, the fused fp32-output GEMM implementation was gated exclusively to CUDA platforms. This caused a fallback to bf16 GEMM followed by a separate cast operation, creating a standalone \"bf16→fp32 copy kernel\" visible in performance traces between the router GEMM and `grouped_topk` for every MoE layer on each decode step. ## Solution The changes enable hipBLASLt's `out_dtype` epilogue support (analogous to CUDA's cuBLAS capability) by modifying `vllm/model_executor/layers/fused_moe/router/gate_linear.py`: - Adds ROCm as an alternative condition for `allow_cublas_router_gemm` when: no bias exists, weights are bf16, and output is fp32 - Updates both `__init__` and `set_out_dtype` methods to ensure consistency - The fused tier now executes on ROCm, folding the fp32 cast into the GEMM epilogue ## Impact Assessment **CUDA:** No changes; the condition uses `current_platform.is_rocm()`, which evaluates false on CUDA. **Correctness:** Verified on gfx942 hardware; results match fp32 reference within ~6e-5 tolerance. **Scope:** Router gate only; expert GEMMs and other fp32 operations remain unaffected. ## Performance Results Benchmarked on ROCm (gfx942) using the online serving benchmark: baseline vs. fused, 3 seeds each (123 / 456 / 789), max concurrency 4, 40 requests per run (40960 input + 40960 generated tokens). All runs completed 40/40 successful with 0 failures. Averaged across the 3 seeds: | Metric | Baseline | Fused (this PR) | Δ | |---------------------------|----------|-----------------|----------------| | Benchmark duration (s) | 343.18 | 337.69 | −5.5 s (−1.6%) | | Output throughput (tok/s) | 119.35 | 121.29 | +1.9 (+1.6%) | | Total throughput (tok/s) | 238.71 | 242.59 | +3.9 (+1.6%) | | Mean TPOT (ms) | 32.87 | 32.34 | −0.53 (−1.6%) | | Mean ITL (ms) | 32.87 | 32.34 | −0.53 (−1.6%) | | Mean TTFT (ms) | 689.65 | 688.97 | ≈flat (−0.1%) | **Interpretation:** Folding the fp32 cast into the router GEMM epilogue removes a per-MoE-layer, per-decode-step conversion kernel from the critical path. The decode-path metrics (TPOT, ITL, output throughput) all improve by a consistent ~1.6%, while TTFT is unchanged as expected since prefill is not dominated by that per-step kernel. Variance across seeds is small (throughput within ±0.2 tok/s, TPOT within ±0.07 ms), so the improvement is stable and above noise, if modest. ## Testing Plan - Confirm the standalone `bfloat16tofloat32_copy_kernel` is absent from ROCm traces - Validate router logits match previous behavior within tolerance - Unit tests added for the modification", "url": "https://github.com/vllm-project/vllm/pull/50268", "createdAt": "2026-07-29T13:26:44Z", "updatedAt": "2026-08-12T20:07:32Z", "timestamp": "2026-08-12T20:07:32Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "ready" ], "author": "mpashkovskii", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50280", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Make EAGLE weight sharing TP-consistent", "text": "## Purpose Addresses the rank-divergent EAGLE weight-sharing decision described in #50158. The issue stays open: real-hardware runs (see discussion below) show the acceptance collapse reported there persists after this fix and has a separate root cause that is still under investigation. EAGLE currently decides whether to share the target model's `embed_tokens` and `lm_head` by comparing each tensor-parallel rank's local shards independently. If equality differs across ranks, some ranks replace the draft module with the target module while others keep the draft weights. The ranks then build different drafters, which can severely reduce speculative acceptance. This change requires unanimous TP-wide agreement before sharing either module. Each rank contributes its local decision to a CPU-group `MIN` reduction, so any mismatch keeps the draft weights on every rank. TP size 1 preserves the existing fast path without a collective. The regression test covers both unanimous agreement and one-rank mismatch for `embed_tokens` and `lm_head`. The change is limited to the existing EAGLE weight-sharing decisions. The collectives occur in the same fixed model-load order on every TP rank and do not affect inference-time execution. ## Test Plan ```bash pytest -q tests/v1/spec_decode/test_eagle.py::test_eagle_weight_sharing_requires_all_tp_shards_to_match pre-commit run --files vllm/v1/spec_decode/llm_base_proposer.py tests/v1/spec_decode/test_eagle.py ``` ## Test Result - Targeted regression test: `2 passed`. On macOS, the platform backend list was patched to a non-empty value before module collection; the test itself mocks the TP CPU-group reduction. - All applicable pre-commit hooks passed on both changed files, including Ruff, formatting, typos, mypy, SPDX, import checks, and forbidden-import checks. - Real-hardware validation (community, 2x RTX 3090, TP=2, see discussion): the consensus mechanism works as designed — both ranks now log the same verdict where stock vLLM disagreed on every boot; no measurable perf change. The acceptance collapse from #50158 is unaffected by this change (separate root cause, tracked in the open issue). AI assistance: OpenAI Codex was used for analysis, implementation, testing, and PR preparation. All changes were reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/50280", "createdAt": "2026-07-29T14:27:13Z", "updatedAt": "2026-08-13T17:59:54Z", "timestamp": "2026-08-13T17:59:54Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "speculative-decoding", "v1" ], "author": "sdflkjssl", "state": "open", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:50288", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[SM120] Add NVFP4 KV cache support for consumer Blackwell (RTX 5090)", "text": "## Summary - Fix V scale swizzle mismatch: the NVFP4 KV store kernel unconditionally swizzled V block scales for the SM100 trtllm-gen MHA kernel, but FlashInfer FA2 (used on SM120) reads scales linearly. Add runtime SM detection so SM≥120 writes V scales without swizzle, resolving garbled output on RTX 5090. - Enable FlashInfer NVFP4 KV cache on SM120: `supports_kv_cache_dtype` accepts SM120, HND layout, BF16 query/output (SM120 FA2 native path does not use FP8 quantization or FP8 output buffers), and \"auto\" backend selection for prefill/decode wrappers. - Route Gemma4 heterogeneous-head models (head_dim_qk=512, head_dim_vo=256) to FLASHINFER on CC 12.x with NVFP4 KV via `VLLM_NVFP4_KV_VOSPLIT` env var (default on). - Add `FlashInferBackend.supports_mm_prefix()` gated on `VLLM_FLASHINFER_MM_PREFIX` (default on) so multimodal Gemma3/4 models are not rejected by backend validation. - Bump flashinfer-python/cubin pin to 0.6.16. ## Context SM120 (RTX 5090) does not have trtllm-gen cubins. The NVFP4 KV cache path previously only worked on SM100 (B200) via trtllm-gen. This PR enables the FlashInfer FA2 tensor-core native path for SM120, which dequantizes FP4 KV to BF16 internally during attention computation. Tested on RTX 5090 with: - Gemma-4-26B-A4B (heterogeneous heads: sliding head_dim=256, full head_dim=512) - Qwen3.6-27B, Qwen3.6-35B-A3B (uniform heads) All produce correct output with `--kv-cache-dtype nvfp4`. ## Test plan - [ ] `vllm serve --kv-cache-dtype nvfp4` on RTX 5090 produces coherent output - [ ] `vllm serve --kv-cache-dtype nvfp4` on RTX 5090 produces coherent output - [ ] SM100 trtllm-gen path unchanged (V scales still swizzled for CC < 120) - [ ] FP8 KV cache unaffected on both SM100 and SM120", "url": "https://github.com/vllm-project/vllm/pull/50288", "createdAt": "2026-07-29T15:46:04Z", "updatedAt": "2026-08-13T14:09:32Z", "timestamp": "2026-08-13T14:09:32Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "needs-rebase", "ci/build", "v1", "nvidia", "quantization" ], "author": "ch2lab", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50303", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Stop the Granite reasoning marker leaking into streamed reasoning", "text": "## Summary `GraniteReasoningParser` leaks the tail of `Here is my thought process:` into the streamed `reasoning` field when a single delta closes that marker and opens the reasoning at the same time. `_get_delta_message_with_no_response_bounds` reconstructs \"the reasoning before this delta\" as: ```python previous_text = reasoning[: -len(delta_text)] ``` That assumes the whole delta lies inside the parsed reasoning. It does not when the delta straddles the end of the marker. BPE merges the marker's `:` with whatever punctuation follows it into one token, so for `Here is my thought process:(thinking)` the tokenizer emits `':('` as a single delta while `reasoning` is only `'('`. The slice underflows to `\"\"`, `prev_idx` and `delta_idx` both go to `-1`, and the function falls through to `return DeltaMessage(reasoning=delta_text)`, emitting the colon as reasoning. ## Reproduction Using the tokenizer the existing test file already uses (`facebook/opt-125m`): ``` text : Here is my thought process:(thinking)Here is my response:(answer) deltas : ['Here', ' is', ' my', ' thought', ' process', ':(', 'thinking', ')', 'Here', ...] expected : reasoning='(thinking)' content='(answer)' streaming: reasoning=':(thinking)' content='(answer)' ``` Non-streaming `extract_reasoning` is already correct on the same text, so the two paths disagree. This is not exotic. Any punctuation directly after the marker triggers it, because that is exactly where BPE merges. Over 70 marker/joiner combinations, `main` disagrees with non-streaming extraction on 16 of them, identically under three tokenizers: | tokenizer | before | after | |---|---|---| | facebook/opt-125m | 16/70 mismatched | 0/70 | | gpt2 | 16/70 | 0/70 | | bert-base-uncased | 16/70 | 0/70 | The joiners that trigger it are `,` `]` `\"` `'` `)` `(` `*` `-` `_` `/` `:` `;` `!` `?` `#` `0`. The ones that do not are whitespace, `.` and letters, which is why the existing fixtures (all of which use a letter or a newline after the marker) never caught it. ## Fix Clamp to what `reasoning` actually holds, and emit only that part of the delta: ```python delta_len = min(len(delta_text), len(reasoning)) previous_text = reasoning[: len(reasoning) - delta_len] delta_text = reasoning[len(reasoning) - delta_len :] ``` The explicit index math also removes the `-0` case: `reasoning[:-len(delta_text)]` returns `\"\"` rather than the whole reasoning when the delta is empty, which is reachable because the streaming loop only skips an empty delta while there are no token ids and no previous tokens. That case happens to produce the same `None` delta message either way, so it is a latent slice bug rather than an observable one, and this fix removes it in passing. ## Scope Only `_get_delta_message_with_no_response_bounds`. Two neighbours were checked and left alone: `_get_delta_message_with_no_reasoning_bounds` returns accumulated `current_text` by design, and `_get_delta_message_with_both_bounds` computes its offsets from `current_text`, so neither has this assumption. One limitation stays, deliberately, because I could not reach it with a real tokenizer: `delta_idx = delta_text.rfind(\"Here\")` needs the whole word `Here` inside one delta, so an arbitrary sub-word split of the *response* marker (`'H'`, `'er'`, `'e'`) can still leak a partial marker into reasoning. All three tokenizers above emit `Here` whole in every position tested, so this is not reachable today, and fixing it means reworking the buffering logic rather than a slice. Flagging it rather than quietly leaving it. ## Not a duplicate - `gh pr list --repo vllm-project/vllm --state open --search \"granite reasoning\"` returns nothing touching `vllm/reasoning/granite_reasoning_parser.py`. #44713 adds Granite parsers to the **Rust** frontend, a different implementation. - No open or closed issue describes this; the only Granite reasoning issue, #14202, is the closed request that added the parser. - Last change to this file was #45988 in June, an unrelated logger removal. ## Tests Added to the existing `tests/reasoning/test_granite_reasoning_parser.py` rather than a new file: - `PUNCTUATION_AFTER_START_SEQ`, wired into `TEST_CASES` for both streaming and non-streaming, so the two paths are pinned to agree. - `STREAMING_14`, the focused unit subcase for the straddling delta (`previous_text=\"Here is my thought process\"`, `delta_text=\":(\"`). Fail-before / pass-after, with the fix reverted and the tests present: ``` $ python -m pytest tests/reasoning/test_granite_reasoning_parser.py -q FAILED ...::test_reasoning[punctuation_after_start_seq_streaming] FAILED ...::test_streaming_subcases[Delta ends the start reasoning sequence and starts the reasoning] 2 failed, 28 passed $ # with the fix 30 passed ``` The non-streaming half of the new case passes both before and after, which is the point: it pins the expectation to what `extract_reasoning` already returns rather than to something invented. Whole directory, with a clean-tree control so the pre-existing failures are not mistaken for mine: ``` $ python -m pytest tests/reasoning/ -q # clean tree 158 failed, 249 passed $ python -m pytest tests/reasoning/ -q # this branch 158 failed, 252 passed ``` Same 158 either way. They are all `test_qwen3_reasoning_parser.py` cases that need model files my machine cannot fetch, and the delta is exactly the 3 tests added here. Five files (`cohere_command`, `kimi_k2`, `kimi_k3`, `mistral`, `nemotron_v3`) are excluded from both runs because they fail to import without `mistral_common` / `partial_json_parser` / `xgrammar` locally. Lint: `ruff 0.14.0` (the version pinned in `.pre-commit-config.yaml`) `check` and `format --diff` are both clean on the two changed files. ## Model evaluation Not applicable, and I want to be explicit rather than silently skip the requirement. This changes string slicing inside a reasoning parser that runs on already-decoded text in the API layer. It does not touch model weights, sampling, kernels or the scheduler, and cannot change which tokens are generated: identical output text produces identical results before and after, only the split between the `reasoning` and `content` fields of the streamed response changes. The behaviour is fully determined on CPU, which is what the tests above exercise. I do not have a GPU available to run `tests/evals/`, and an eval would not exercise this path anyway. AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/50303", "createdAt": "2026-07-29T18:01:52Z", "updatedAt": "2026-08-12T21:55:23Z", "timestamp": "2026-08-12T21:55:23Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "tool-calling" ], "author": "vineethsaivs", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50366", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "DCP: consume owner-sharded Top-K candidates through symmetric memory", "text": "# DCP: consume owner-sharded Top-K candidates through symmetric memory ## Summary This change replaces the sparse-indexer candidate AllGather with direct consumption from one symmetric-memory candidate shard per DCP rank. Each producer stores only its local `(rows, candidates, score_and_token)` shard. Symmetric-memory rendezvous gives every consumer a peer tensor view for each owner's allocation. The stable Top-K kernel reads those views and selects globally without first constructing a full gathered candidate tensor. This is the symmetric-memory counterpart to the generic CUDA VMM design in #50010. It preserves #50010 as the backend for asymmetric peer-addressable objects; this PR tests whether the existing PyTorch symmetric-memory dependency offers a narrower integration for the uniform DCP Top-K case. ## Data flow ```text rank 0 local candidates --\\ rank 1 local candidates ---+-> stable Top-K reads owner shards directly rank 2 local candidates ---+-> global token ids rank 3 local candidates --/ ``` There is no `world_size x local_candidates` inbox on each rank. At DCP4, the logical candidate payload remains one shard per rank instead of four shards per rank. ## Synchronization Each owner publishes a device-side write epoch after packing its shard. A consumer waits until all owner epochs are visible, reads all shards, and then publishes its read epoch. Before reusing an owner shard, its producer waits until every consumer has acknowledged the prior epoch. The protocol is CUDA-graph compatible and fail-closed. Once selected, it does not fall back to AllGather, the full symmetric-memory inbox, or VMM. ## Scope - Owner-local symmetric-memory allocation and peer-view discovery. - Direct peer-shard loads in the existing stable Top-K selection kernel. - Bounded decode dispatch behind an experimental environment flag. - Explicit phase/shape policy outside that bound; no error recovery fallback. - Distributed exactness, ties, empty-shard, prefill-indexing, and changing-input CUDA-graph tests. ## Validation Validated on 4x NVIDIA GB200: - Exact match with explicit AllGather: passed. - Changing-input CUDA-graph replay: passed. - Rows `1, 17, 64, 96`, ties, an empty owner shard, and prefill indexing: passed. - Repository checks including Ruff, formatting, mypy, and SPDX: passed. The full distributed test reports `1 passed`. ### Performance The matched DCP4 GLM-5.2 DSA sweep used 2048 candidates per rank, 4096 score columns, fixed 1800 MHz SM clocks, 100 CUDA-graph replays per sample, 30 samples, maximum-rank timing, and three fresh process runs. Against explicit candidate AllGather: | Rows | Explicit (us) | Symmetric peer shards (us) | Saved (us) | Improvement | |---:|---:|---:|---:|---:| | 1 | 44.429 | 41.042 | 3.387 | 7.624% | | 8 | 48.581 | 41.392 | 7.189 | 14.800% | | 32 | 56.431 | 43.247 | 13.184 | 23.363% | | 64 | 64.690 | 45.891 | 18.799 | 29.060% | The symmetric path's process CV was 0.021%-0.130%. Against the unchanged generic VMM implementation from #50010 at commit `5198c756b31f963c821a0c7752e50245d0da3a93`: | Rows | #50010 VMM (us) | Symmetric peer shards (us) | Symmetric improvement | |---:|---:|---:|---:| | 1 | 41.049 | 41.042 | 0.018% | | 8 | 41.467 | 41.392 | 0.183% | | 32 | 44.017 | 43.247 | 1.750% | | 64 | 46.014 | 45.891 | 0.267% | At 64 rows, explicit AllGather materializes 4,194,304 candidate bytes per rank. This path stores 1,048,576 candidate bytes and allocates 1,048,624 bytes including synchronization metadata. The VMM path allocates 2,097,152 bytes for the same candidate payload because of VMM allocation granularity. This candidate proceeds because it keeps the one-shard-per-rank property, wins against explicit AllGather at every measured row count, matches or slightly beats #50010, and halves the actual allocation relative to #50010 at this bound. It remains a separate DSA contribution; Kimi-K3 does not exercise this Top-K path. This is an operator and workspace result; the PR does not claim an end-to-end serving gain. Head commit: `a3db29b465cf540197bd8d2b20dc75bd5c71a435`.", "url": "https://github.com/vllm-project/vllm/pull/50366", "createdAt": "2026-07-30T02:48:12Z", "updatedAt": "2026-08-13T01:13:02Z", "timestamp": "2026-08-13T01:13:02Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "needs-rebase", "v1" ], "author": "foraxe", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50382", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DCP] Default query replication for GLM sparse attention", "text": "## Summary - add `--dcp-q-replicate` and `--no-dcp-q-replicate` as explicit CLI options; - preserve `VLLM_DCP_Q_REPLICATE` as the highest-precedence override; - enable query replication automatically for `glm_moe_dsa` models when decode DCP is enabled and PCP is not enabled; - keep query replication opt-in for other MLA model families. The automatic default removes GLM's decode query all-gather at the cost of replicating the query projection weights across DCP ranks. Explicit CLI or environment selections still override the model default. ## Performance and memory GLM-5.2-NVFP4, TP=4, DCP=4, 32K input tokens, 128 output tokens, batch 8, two warmups, five measured iterations: - QREP off: 20.1777 s - QREP on: 19.7426 s - delta: -0.4351 s (-2.16%) With QREP enabled, a fresh DCP backend comparison measured: | DCP backend | Average latency | | --- | ---: | | `ag_rs` | 19.7636 s | | `a2a` | 18.0213 s | For the matched 32K benchmark configuration, model-loading memory increased from 110.37 GiB/rank to 114.94 GiB/rank with QREP, approximately +4.57 GiB/rank. ## Model evaluation GLM-5.2-NVFP4, TP=4, DCP=4, GSM8K 5-shot, 300 questions: - automatic GLM QREP default: 94.0% accuracy, 0% invalid outputs; - explicit QREP off: 95.0% accuracy, 0% invalid outputs. The QREP-off evaluation used `max_num_seqs=32` because FlashInfer's sparse-MLA workspace overflowed when all prompts were scheduled concurrently. Both runs evaluated the same 300 questions with the same decoding settings. The one-point difference is not significant at this sample size. ## AI assistance AI assistance was used for implementation, tests, benchmarking, profiling analysis, and drafting this description. The human submitter has reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50382", "createdAt": "2026-07-30T04:48:07Z", "updatedAt": "2026-08-12T20:35:00Z", "timestamp": "2026-08-12T20:35:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "deepseek" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50388", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Fix ValueError on KV load failure with a hybrid KV cache", "text": "## Purpose `_update_requests_with_invalid_blocks` unpacks `get_block_ids()` -- which returns one block-id list per KV cache group -- into a single name: # TODO (davidb): add support for hybrid memory allocator (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id) so the first reported invalid block on a hybrid (Mamba/SSM plus attention) model raises `ValueError: too many values to unpack (expected 1)` inside the scheduler. Nothing catches it, so EngineCore calls `_send_engine_dead()` and exits: one recoverable read failure takes down every request on the instance, not just the one that failed. A hybrid model's groups sit on different block grids, so the same block index means a different token position in each and there is no single longest valid prefix to truncate at. Rather than build a block-level answer for that -- per- block error handling is being removed in https://github.com/vllm-project/vllm/pull/42461 -- affected hybrid requests recompute in full, which is the granularity that replacement moves to anyway. Single-group models keep the existing block-level truncation untouched. Blocks are still added to `marked_invalid_block_ids` so a request sharing one of them is rescheduled as before. The null block needs excluding on this path. In Mamba `align` mode a group's block list is mostly the shared null block -- a 9-chunk request looks like `[0, 0, 0, 0, 0, 0, 0, 0, 10]`, only the aligned snapshot is backed -- so without the guard a reported null block would restart unrelated requests, and `blocks_to_evict` would collect a block that belongs to no request. ## Test Plan Use Kimi-Linear-48B-A3B (TP=2, 4 KV cache groups). And also: `pytest -v tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py` ## Test Result Verified end to end on Kimi-Linear-48B-A3B (TP=2, 4 KV cache groups). ``` (vllm-dev) root@kebe-gpu-dev-0:~/vllm# pytest -v tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option \"asyncio_default_fixture_loop_scope\" is unset. The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: \"function\", \"class\", \"module\", \"package\", \"session\" warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) ========================================================================= test session starts ========================================================================= platform linux -- Python 3.12.12, pytest-8.3.5, pluggy-1.5.0 -- /home/jovyan/conda/envs/vllm-dev/bin/python3.12 cachedir: .pytest_cache rootdir: /home/jovyan/vllm configfile: pyproject.toml plugins: asyncio-1.0.0, anyio-4.9.0 asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 14 items tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs0] PASSED [ 7%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs1] PASSED [ 14%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs2] PASSED [ 21%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs0] PASSED [ 28%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs1] PASSED [ 35%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs2] PASSED [ 42%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs0] PASSED [ 50%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs1] PASSED [ 57%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs2] PASSED [ 64%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_progressive_load_failure[100-99-invalid_block_idxs0] PASSED [ 71%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_progressive_load_failure[100-99-invalid_block_idxs1] PASSED [ 78%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_recomputes_whole_request[0] PASSED [ 85%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_recomputes_whole_request[1] PASSED [ 92%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_ignores_null_block PASSED [100%] ========================================================================== warnings summary =========================================================================== ../conda/envs/vllm-dev/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs0] /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/transformers/models/gpt2/tokenization_gpt2.py:110: DeprecationWarning: Deprecated in 0.9.0: BPE.__init__ will not create from files anymore, try `BPE.from_file` instead BPE( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================== 14 passed, 15 warnings in 33.95s =================================================================== ``` ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50388", "createdAt": "2026-07-30T05:51:10Z", "updatedAt": "2026-08-12T18:45:35Z", "timestamp": "2026-08-12T18:45:35Z", "metrics": { "reactions": 3, "comments": 4 }, "labels": [ "v1", "kv-connector" ], "author": "kebe7jun", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50429", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] HunyuanOCR - handle duplicate image_start tokens in xdrope positions", "text": "## Purpose Fix IndexError crash in HunyuanOCR multimodal inference. `get_xdrope_input_positions` loops over `` token count but indexes actual image tensor array. HunyuanOCR tokenizer inserts duplicate tokens → out-of-bounds access → EngineCore crash on first multimodal request. ## Test Plan **Reproduce (before fix):** ```bash # Start vLLM server with HunyuanOCR python3 -m vllm.entrypoints.openai.api_server \\ --model tencent/HunyuanOCR \\ --dtype bfloat16 \\ --max-model-len 8192 \\ --gpu-memory-utilization 0.85 \\ --trust-remote-code # Send multimodal request curl -X POST http://localhost:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{ \"model\": \"tencent/HunyuanOCR\", \"messages\": [{ \"role\": \"user\", \"content\": [ {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mNk+M9Qz0AEYBxVSF+FABJADveWkH6oAAAAAElFTkSuQmCC\"}}, {\"type\": \"text\", \"text\": \"What color is this?\"} ] }], \"temperature\": 0, \"max_tokens\": 50 }' ``` Expected: IndexError in hunyuan_vision.py:851 → EngineDeadError Verify fix: Same commands → successful 200 response with generated text ## Test Result Before (IndexError crash): ```bash (EngineCore pid=440) ERROR [core.py:1332] File \".../hunyuan_vision.py\", line 851, in get_xdrope_input_positions (EngineCore pid=440) ERROR [core.py:1332] t, h, w = image_grid_thw[image_index] (EngineCore pid=440) ERROR [core.py:1332] ~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore pid=440) ERROR [core.py:1332] IndexError: list index out of range (EngineCore pid=440) INFO [xpu_worker.py:167] XPUWorker shutdown: cleaning up HTTP 500: {\"error\": {\"message\": \"EngineCore encountered an issue.\", \"type\": \"InternalServerError\", ...}} ``` After (successful generation): ```bash { \"id\": \"chatcmpl-a1177725ffaacc5a\", \"object\": \"chat.completion\", \"created\": 1785411625, \"model\": \"tencent/HunyuanOCR\", \"choices\": [{ \"index\": 0, \"message\": { \"role\": \"assistant\", \"content\": \"To determine the color of the image, observe the visual elements: the entire image is a solid, uniform green. There are no other colors present. \\n\\nSo, the color of the image is green.\" }, \"finish_reason\": \"stop\" }], \"usage\": {...} } ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50429", "createdAt": "2026-07-30T12:06:07Z", "updatedAt": "2026-08-13T12:05:45Z", "timestamp": "2026-08-13T12:05:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "mganczarenko", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50431", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[sleep functionality] code refactor about sleep/wake_up", "text": "## Purpose 1. For now, frontend multiprocessing the sleep and wake_up are utility commands and in `async_llm` the call will await the request future. So, it is a sync operation. The log should be deleted as v0 has been completely deleted. 2. remove hard code strings 3. more detailed logs about the scheduler state change ## Test Plan NA ## Test Result NA ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50431", "createdAt": "2026-07-30T13:06:06Z", "updatedAt": "2026-08-13T08:46:36Z", "timestamp": "2026-08-13T08:46:36Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "v1" ], "author": "andyxning", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50487", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Spec Decode] Tap the pre-norm AttnRes mixture as the Kimi K3 DFlash aux state", "text": "## Purpose The DFlash drafter consumes auxiliary hidden states captured at a fixed set of target layers. K3 captures the post-mixture stream, which is not what the drafter was trained against: the AttnRes residual mixture is applied before the layer norm, and the current capture site reads the value after it. Tapping the pre-norm mixture instead recovers the stream the drafter expects. This selects the correct weights to mix against depending on where the tapped layer sits — the next layer's own AttnRes weights when one follows, the model's output-side aggregation for the last layer on the final pipeline stage, and a fall back to the running prefix for the last layer of a non-final stage, where the consumer lives on the next rank and the output-side weights do not exist there. Gated behind `VLLM_KIMI_K3_AUX_ATTN_RES_STREAM` (default off) while it settles, since it changes what the model feeds the speculator. The one-shot log of which layers are tapped and which capture mode is active is emitted from `_set_aux_hidden_state_layers`, the setup-time hook the speculator calls once, rather than from inside `forward`. Setting an `nn.Module` attribute inside a compiled forward pass risks a graph break or recompile under `torch.compile`; doing it at setup time avoids that and is also the natural place to log the tapped layer tuple. I checked for duplicate/overlapping open PRs (`gh pr list --search` on \"kimi k3 aux hidden state\" and \"attn_res eagle3\"); no open PR touches the Kimi K3 auxiliary hidden state capture site. I used AI assistance (Cursor) to draft, test, and validate this change, and I reviewed every changed line before submitting. ## Test Plan Unit: ``` pytest tests/models/kimi_k3/test_aux_attn_res_stream.py \\ tests/models/kimi_k3/test_eagle3.py \\ tests/models/kimi_k3/test_attn_res.py -v ``` End to end: Kimi K3 on 8×B300 at TP8, DFlash drafter at `num_speculative_tokens 16`, one concurrent user, single variable (the capture mode) on an otherwise identical container. Acceptance is `1 + accepted/steps` from raw counter deltas. ## Test Result **Unit.** All pass: ``` tests/models/kimi_k3/test_aux_attn_res_stream.py ....... [7 passed] tests/models/kimi_k3/test_eagle3.py .... [4 passed] tests/models/kimi_k3/test_attn_res.py ......sss..... [12 passed, 3 skipped] ``` The 3 skips are the multi-GPU (`nvidia-1/4/8`) parametrizations of `test_attn_res`, not runnable on the single-GPU box used to verify this PR. `test_aux_attn_res_stream.py` is new and covers the selection logic added here: the consumer-layer branch, the last-layer/final-rank branch that uses the model's output-side aggregation, the last-layer/non-final-rank fallback to the running prefix, and that the feature is a no-op reproducing the original `prefix_sum + hidden_states` when disabled. One pre-existing test needed a fixture fix rather than a code change: `test_eagle3.py::test_kimi_k3_uses_shared_eagle3_layer_configuration`'s shared model stub predates this PR and didn't set `use_attn_res`, which the new `_set_aux_hidden_state_layers` override now reads unconditionally. Fixed the stub to set it, matching what the other two tests in that file already do at their own call sites. **End to end.** Mean accept length 4.7167 and 4.3948 with the capture on, against 2.2913 and 2.6579 with it off. Paired across 32 matched rows: +1.903 accept length, treatment winning 31 of 32, p = 1.5e-8. An independent implementation of the same idea reports +1.536 at a different operating point (2.165 prefix-only against 4.454), which is the same effect measured twice. **How to read these numbers.** Some rows come from different containers rather than uniformly from the same one, and the gain was measured with prefix caching off; a follow-up with caching on read +0.083, inside the within-arm spread. The flag is off by default for that reason — the mechanism is right and the off-cache effect is large and well established, but it should be re-measured on a cached lane before anyone depends on it there. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/50487", "createdAt": "2026-07-30T22:35:57Z", "updatedAt": "2026-08-12T22:00:51Z", "timestamp": "2026-08-12T22:00:51Z", "metrics": { "reactions": 3, "comments": 24 }, "labels": [ "ready", "verified", "kimi", "k3" ], "author": "rchalamala", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50496", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Apertus 1.5", "text": "## General Information This change enables support for serving Apertus v1.5 multimodal (hf tags apertus-ai/Apertus-v1.5-8B, apertus-ai/Apertus-v1.5-70B) Special thanks to @blancsw for creating a workable refactor from https://github.com/swiss-ai/vllm/tree/apertus_integration that optimizes the multimodal inference pipeline and provides a foundation for future integration. Also, special thanks to @tomas-polach for the bug-fix contribution, @AryanAhadinia for the reasoning parser, and @robmsmt for the double BOS-token fix. This PR depends on transformers [PR](https://github.com/huggingface/transformers/pull/47662). ## Test Plan ``` vllm serve apertus-ai/Apertus-v1.5-70B \\ --max-model-len 100000 \\ --tensor-parallel-size 4 \\ --enable-chunked-prefill \\ --async-scheduling \\ --enable-prefix-caching \\ --max-num-seqs 128 \\ --max-num-batched-tokens 65536 \\ --block-size 16 \\ --enable-auto-tool-choice \\ --tool-call-parser apertus \\ --reasoning-parser apertus \\ --chat-template-content-format string ``` ## Tests added Audio tests were not added because default sample rate supported is 16Khz and Apertus 1.5 works with 24Khz hence leading to test failures. and multiple audio case does not exist. ``` XDG_CACHE_HOME=/tmp/vllm-test-cache /opt/venv/bin/python -m pytest \\ tests/models/multimodal/processing/test_common.py \\ -k 'Apertus-v1.5-8B' -v XDG_CACHE_HOME=/tmp/vllm-test-cache /opt/venv/bin/python -m pytest \\ tests/models/multimodal/generation/test_common.py \\ -k apertus_1p5 -v ```", "url": "https://github.com/vllm-project/vllm/pull/50496", "createdAt": "2026-07-31T00:24:08Z", "updatedAt": "2026-08-13T08:54:15Z", "timestamp": "2026-08-13T08:54:15Z", "metrics": { "reactions": 8, "comments": 13 }, "labels": [ "documentation", "new-model", "ready", "multi-modality", "tool-calling" ], "author": "Anunay-Yadav", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50513", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] update UMD to 26.27", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50513", "createdAt": "2026-07-31T03:22:17Z", "updatedAt": "2026-08-13T05:15:26Z", "timestamp": "2026-08-13T05:15:26Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "intel-gpu", "ci/build" ], "author": "yma11", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50514", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Feat/spec decode under pipeline parallel", "text": "## Purpose EAGLE3-style speculative decoding (`eagle3` / `dflash` / `dspark`) is rejected outright when pipeline parallelism is enabled: ``` ValueError: with pipeline parallel is not supported. ``` The drafter runs on the last PP rank, but it consumes auxiliary hidden states tapped from target layers that may live on earlier stages, and those tensors were dropped at the stage boundary. This PR forwards them and lifts the guard. Lifting the guard alone is not sufficient. The code behind it had never executed, so it hid a stack of latent defects in the shared PP path — all of them reachable only once spec decode and PP run together, which means they were unobservable rather than latent-but-live: 1. **Draft tokens never reached the non-last ranks.** Only the last rank runs the drafter, so `req_states.draft_tokens` is written nowhere else — yet the *first* rank owns `embed_tokens` and builds the embeddings for the whole pipeline. It was embedding the scheduler's `PLACEHOLDER_TOKEN_ID` (`-1`) into the draft slots, so the last rank verified real proposals against logits computed from placeholders. Silent corruption. Fixed by broadcasting the proposals from the last rank on the existing deferred slot, which has identical production and consumption timing. 2. **`compute_need_sampled_mask` ended the broadcast early.** The scheduler advances `num_computed_tokens` by the full scheduled width up front and rolls the rejected part back in `update_from_output`, which under PP lands *after* the next batch is scheduled. Reading the inflated count marked requests as finishing up to `num_speculative_tokens` early, after which the last rank stopped broadcasting and the other ranks' `last_sampled_tokens` froze, repeating a stale token. 3. **The sampled-token broadcast width could mismatch.** `receive()` always allocates `max_sample_len` columns while `broadcast()` sent whatever width the sampler produced, and the non-spec path returns width 1. NCCL does not diagnose the count mismatch: the root completes and the peer hangs until the watchdog fires, so any PP + spec-decode run deadlocked on its first prefill. Fixed by padding the send side; `post_update` stops at `num_sampled`, so the pad columns are never read. ### Design Aux-state forwarding is added generically on `EagleModelMixin`. A stage packs its own taps plus those inherited from upstream into the `IntermediateTensors` payload, and sizes its recv buffers from the same rule — a tap `a` is upstream iff `a <= start_layer` — so send and recv counts agree with no negotiation. Models opt in with `supports_aux_hidden_states_over_pp`. The previous blanket error is retained for those that do not, now raised with the offending model named. Kimi-K3 opts in; no other model does, so **EAGLE3 and dflash under PP remain gated and untested.** Two smaller changes ride along, both no-ops for configurations that worked before (spec decode under PP > 1 was rejected outright): the DSpark drafter gets a real vocab embedding on the last stage, where the target's is a `PPMissingLayer`; and the draft parallel config stops inheriting the target's PP size, since the drafter is instantiated only on the last stage and is never itself pipelined. ### Scope: capped at `pipeline_parallel_size <= 2` `GPUModelRunner.load_model` raises `NotImplementedError` above 2. This reflects validation coverage, not a known defect. The forwarding rule is size-agnostic by construction and the accounting is unit-tested up to pp=8. But pp>2 is the first topology with a **middle** stage, which must both adopt upstream taps and contribute its own to the same payload, and no such run has happened on hardware. The conservative default is deliberate because this feature does not fail loudly: a drafter fed mis-ordered or missing taps still emits syntactically valid proposals that simply get rejected more often, so the only symptom is a depressed acceptance rate. Lifting the cap should require an acceptance-rate comparison at pp>2, not just a successful boot. Happy to drop the cap if maintainers would rather have the untested-but-general path available. ### Note on commit 1 The first commit is an independent bugfix — `MambaHybridModelState.postprocess_state` feeds an int32 `idx_mapping` to `index_fill_`, which accepts only int64: ``` IndexError: index_fill_(): Expected dtype int64 for index. ``` **This is reachable on main today with no speculative decoding involved.** Its only caller is the non-last-PP-rank path in `GPUModelRunner.execute_model`, so any Mamba-hybrid model (Jamba, Falcon-H1, Nemotron-H, Qwen3-Next, MiniMax, Kimi Linear) crashes under `pipeline_parallel_size >= 2` with a chunked-prefill batch. It is included here because Kimi-K3 is a hybrid model and hits it under PP. Glad to split it into its own PR if preferred. ## Test Plan **Unit.** A CPU test for the aux accounting invariant, no distributed init required: ```bash pytest tests/v1/worker/test_eagle3_aux_hidden_states_pp.py ``` 12 cases over pp ∈ {1,2,3,4,6,8}, exercising the real `EagleModelMixin` and real `get_pp_indices`. Covers tap ordering and the boundary-tap double-count hazard (a tap landing exactly on `start_layer` must be counted as upstream by exactly one stage). The parametrization intentionally runs past the pp<=2 cap, since the accounting is general and this is what a future enablement would build on. **End-to-end.** Kimi-K3 (93 layers, DSpark drafter, `num_speculative_tokens=7`) at TP8 × PP2 on 2× 8×B200 with EFA. Two checks: 1. *Correctness*: greedy decode compared token-for-token against the identical topology with speculative decoding disabled. Speculative decoding is supposed to be output-neutral, so any divergence is a bug. 2. *Acceptance rate*: the sensitive detector. Corruption of aux taps crossing the PP boundary degrades acceptance before it produces visibly wrong text. Throughput measured on 128 fixed GSM8K prompts, same seed, concurrency 1 → 32. ## Test Result Unit: 12 passed. `ruff check` and `ruff format --check` clean. **Correctness**: greedy output is token-for-token identical to the same topology without speculative decoding. **Acceptance**: 2.32 of 7 draft tokens per step. **Throughput** (output tok/s, GSM8K, 128 prompts): | Concurrency | No spec | Spec (PP2) | Speedup | |---:|---:|---:|---:| | 1 | 77.30 | 174.52 | 2.26× | | 4 | 255.77 | 461.53 | 1.80× | | 8 | 423.86 | 714.01 | 1.68× | | 32 | 1138.69 | 1584.07 | 1.39× |", "url": "https://github.com/vllm-project/vllm/pull/50514", "createdAt": "2026-07-31T03:29:39Z", "updatedAt": "2026-08-12T22:36:16Z", "timestamp": "2026-08-12T22:36:16Z", "metrics": { "reactions": 2, "comments": 23 }, "labels": [ "speculative-decoding", "ready", "needs-rebase", "ci/build", "llama", "qwen", "deepseek", "cpu", "nvidia", "mrv2", "kimi", "k3" ], "author": "yongqinwang-cmd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50525", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Fix NUMA node binding for multi-TP CPU workers", "text": "## Purpose On CPU, each tensor-parallel worker bound its memory to the NUMA node of the **first allowed CPU** (`get_allowed_cpu_list()[0]`), so every rank in a multi-TP run landed on the **same** NUMA node. This defeats NUMA locality: all workers compete for one node's memory bandwidth while other nodes sit idle, and the per-worker KV cache sizing is computed against a single node. This PR binds each rank round-robin across the visible memory nodes: ```python current_numa_node = allowed_memory_nodes[rank % len(allowed_memory_nodes)] ``` so worker `rank` maps to a distinct NUMA node when multiple nodes are available. Because the node now comes directly from `get_visible_memory_node()`, it is always a valid memory node by construction; the previous `cpu_core.numa_node not in allowed_memory_nodes` guard and the s390x synthetic-book-ID fallback are no longer reachable and have been removed. A `logger.debug` line records the node each rank binds to. ## Not a duplicate This is distinct from #47336, which distributes **data-parallel EngineCore processes** across NUMA nodes at the process level. This PR fixes **tensor-parallel workers** within a single engine (`CPUWorker.__init__`) that were all binding to the first CPU's node. Different layer, different mechanism, no overlap in changed logic. ## Notes AI assistance was used while preparing this change. The author has reviewed every changed line and understands the change end-to-end.", "url": "https://github.com/vllm-project/vllm/pull/50525", "createdAt": "2026-07-31T05:15:37Z", "updatedAt": "2026-08-13T08:39:35Z", "timestamp": "2026-08-13T08:39:35Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "cpu" ], "author": "Sundaresan-G", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50532", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch", "text": "# [Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch ## Purpose Split out of #50488 at a maintainer's request, so each fix is reviewable on its own. V2 classified a batch as uniform decode from its shape alone (`num_tokens == num_reqs * max_query_len`) and nothing about whether the requests were actually decoding. A prompt chunk of exactly `1 + num_speculative_tokens` tokens has that shape, so a prefill was dispatched with `cg_mode=FULL` and replayed the captured spec-verify decode graph over prompt tokens. Without spec decoding the same collision exists between a 1-token prompt chunk and the q=1 decode graph. Dense models survive it, because spec-verify attention over a fresh sequence is the same computation as prefill attention. Recurrent-state models do not: the mis-dispatched batch still builds prefill metadata, so the GDN builder's persistent-buffer refresh is skipped and the replayed graph reads capture-time state indices. `state_idx <= 0` makes the fused GDN kernels skip the recurrent state read and store, so the prompt's conv/SSM state is never created. The gate is per batch, so genuine decodes sharing a batch with such a chunk lose their refresh too. This is the mechanism behind #49918. The predicate now requires every request in the batch to be past its prefill, not merely for the batch to have the right shape. Dummy batches (DP padding, memory profiling, warmup) are uniform by construction and have no request state to consult, so they stay classified by shape, mirroring `InputBatch.make_dummy`. The second commit is a performance follow-on in the same path: the shape test is O(1) and rejects every mixed prefill/decode batch, which is most of them, so it is hoisted ahead of the `np.fromiter` gather of request state rather than run after it. A third commit covers `multi_module_mtp/speculator.py`, which landed on main after this branch was cut. It is a copy of the autoregressive speculator, comment included, and repeats the shape-only call, so it carries the same defect. The guard added alongside it is written over every speculator in the package rather than over that one file, because copying is how the call reappeared; it fails on the offending line without the change and passes with it. ## Test Plan - `tests/v1/worker/test_gpu_batch_ordering.py` — the classification itself, including the prompt-chunk shapes that previously passed and the dummy-run path that must keep passing, plus the dispatch consequences: an 8-token prefill dispatches PIECEWISE and answers the prompt, control outputs are unchanged byte for byte, and genuine decode steps still replay their FULL graphs. A further case pins that the O(1) shape test rejects a non-uniform batch on its own, which is the ordering the second commit relies on. - `tests/v1/spec_decode/test_dynamic_sd_cug.py` — the dynamic speculative schedule path over the same dispatch. ## Test Result `pytest tests/v1/spec_decode/test_dynamic_sd_cug.py tests/v1/worker/test_gpu_batch_ordering.py` — **14 passed**, run on an H100 sandbox against this branch overlaid on the pinned nightly wheel (`0.26.1rc1.dev77+g6f91edf96`). Revert check: with the decoding clause removed from the predicate, `test_prompt_chunks_shaped_like_spec_decode_miss_the_full_graph` and `test_prompt_chunk_of_decode_query_len_is_not_uniform_decode` fail while the uniform-decode and dummy-run cases still pass. `ruff check` and `ruff format --check` are clean on every touched file. ### Model evaluation `tests/evals/gsm8k/gsm8k_eval.py`, 400 questions, 5-shot, greedy, on `Qwen/Qwen3-4B` with the published `z-lab/Qwen3-4B-DFlash-b16` drafter at `num_speculative_tokens=16`, V2 model runner, single H100: **accuracy 0.880, 0.000 invalid, 400 questions**. This was measured on the combined set of fixes before the split, so it covers this change together with the others rather than in isolation. ## Related - #49918 — reproduces this independently of proposer (MTP and ngram), quantization and batch size, with PIECEWISE or enforce-eager as workarounds. Split from #50488, alongside the warmup lookahead reservation fix and the CUDA graph capture-size fix. I checked for duplicate and overlapping open PRs (`gh pr list --search` on \"uniform decode cudagraph\", \"uniform decode dispatch\", \"prefill full cudagraph\") and found none besides #49918, which is an issue rather than a fix. I used AI assistance (Cursor) to draft, test, and validate this change, and I reviewed every changed line before submitting.", "url": "https://github.com/vllm-project/vllm/pull/50532", "createdAt": "2026-07-31T05:58:03Z", "updatedAt": "2026-08-12T19:05:14Z", "timestamp": "2026-08-12T19:05:14Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "speculative-decoding", "needs-rebase", "nvidia", "mrv2" ], "author": "rchalamala", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50534", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B70", "text": "## Purpose Add tuned `selective_state_update` configs for **Intel Arc Pro B70 Graphics**, and fix `benchmarks/kernels/benchmark_selective_state_update.py` so it runs on an XPU-only build. The config directory holds AMD and NVIDIA devices only, so Mamba/hybrid models on Intel GPUs fall back to the heuristic. Four files, two shapes: | shape | cache_dtype | exercised by | | --- | --- | --- | | `headdim=128,dstate=256` | float32 + float16 | FalconH1 | | `headdim=64,dstate=128` | float32 + float16 | Mamba2 | `headdim=64,dstate={16,32,64,256}` was tuned and validated too, but no model on hand exercises those shapes, so per review they are dropped rather than shipped without E2E verification — they keep using the heuristic as they do today. - **`float16` is tuned, not copied from `float32`.** Without `--mamba-ssm-cache-dtype`, vLLM resolves `\"auto\"` → model dtype → the `float16` file. On a bfloat16 state the optimum differs at 9 of 12 grid points for `hd128 d256`; reusing `float32` values costs a median 79.6% at kernel level and is 1.10x *slower* than the heuristic at one point on `hd64 d128`. - **Grid starts at `effective_batch=8`.** Mamba2-2.7b at `concurrency=1` runs batch 1 × 80 heads; a grid starting at 128 was ~10% slower than the heuristic there. - **`hd64 d128` keeps some incumbent entries.** At `effective_batch=4096` the candidates were 0.1% apart in the microbenchmark and picking the winner cost 4.2% E2E — the tuner runs `ngroups=8`, models run `ngroups=1`. Re-tuned entries adopted only above a 10% margin. Script fixes, all needed on XPU: - `torch.cuda.get_device_capability()` → `current_platform.get_device_capability()` (`None` on XPU). Called unconditionally today, it raises `AssertionError: Torch not compiled with CUDA enabled` before tuning starts; also covers ROCm. - `torch.cuda.CUDAGraph()`/`torch.cuda.graph()` → `torch.accelerator.Graph()`. - `torch.cuda.Event` → `torch.Event`, as in `benchmark_moe.py`. The last two otherwise raise inside a `try/except` returning `None`, so every candidate is skipped and the sweep yields no timings. Note that `torch.accelerator.Graph()` is not used elsewhere in the repo — worth a check on CUDA. ## Test Plan Intel Arc Pro B70 (31.9 GiB), Triton 3.7.1, single card. Kernel sweep per shape and cache dtype; numerical validation against the CPU reference under each model's real `ngroups`/`nheads` with the heuristic as control; E2E `vllm bench serve` with a separate server launch and warm-up per measurement, on `tiiuae/Falcon-H1-7B-Instruct` and `AntonV/mamba2-1.3b-hf`. ## Test Result Kernel, speedup vs heuristic: | shape | 8 | 32 | 128 | 512 | 2048 | 16384 | peak | | --- | --- | --- | --- | --- | --- | --- | --- | | `hd128 d256` | 1.56x | 2.83x | 4.15x | 1.86x | 1.96x | 1.99x | **4.47x** @ 64 | | `hd64 d128` | 1.10x | 1.33x | 2.33x | 3.18x | 1.30x | 1.30x | **3.18x** @ 512 | E2E `in=128 out=1024`, tuned vs heuristic, explicit `--mamba-ssm-cache-dtype float32`: | model | conc | eff_batch | TPOT tuned | TPOT heur | TPOT gain | throughput gain | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | FalconH1-7B | 1 | 24 | 28.78 ms | 29.55 ms | 1.027x | 1.022x | | FalconH1-7B | 8 | 192 | 33.03 ms | 39.13 ms | 1.185x | 1.184x | | FalconH1-7B | 16 | 384 | 38.60 ms | 48.70 ms | 1.262x | 1.255x | | FalconH1-7B | 32 | 768 | 50.12 ms | 68.51 ms | 1.367x | **1.362x** | | FalconH1-7B | 64 | 1536 | 56.82 ms | 82.69 ms | **1.455x** | 1.213x | | Mamba2-1.3b | 1 | 64 | 15.00 ms | 15.36 ms | 1.023x | 1.021x | | Mamba2-1.3b | 8 | 512 | 15.26 ms | 15.70 ms | 1.029x | 1.029x | | Mamba2-1.3b | 16 | 1024 | 15.53 ms | 16.59 ms | 1.068x | 1.072x | | Mamba2-1.3b | 32 | 2048 | 19.00 ms | 25.01 ms | **1.316x** | **1.313x** | | Mamba2-1.3b | 64 | 4096 | 32.46 ms | 41.79 ms | 1.288x | 1.285x | FalconH1 throughput saturates by `concurrency=64` (631.5 tok/s at 32 vs 624.5 at 64), so it diverges from TPOT. On a `1024/256` shape: FalconH1 1.026x → 1.352x TPOT, Mamba2-1.3b 1.256x at 32 but 0.988x / 0.990x at 1 / 8 — ~1%, below the 1.5% threshold and single-shot. On the default path (no flag, so the `float16` file is loaded), tuning that file beats copying the `float32` values into it by 5.6–5.7% for FalconH1 at `concurrency` 16 and 64, and by 2.6% for Mamba2-1.3b at 128 — interleaved A/B with two warm-ups, run-to-run spread 0.02–0.23 ms. A null control at `concurrency=8`, where both variants resolve to the same entry, landed inside that spread. Validation passes 8/8 and 7/7, control clean. Mamba2-1.3b measured on two B70 machines with two vLLM builds; high-gain results agree to three decimals.", "url": "https://github.com/vllm-project/vllm/pull/50534", "createdAt": "2026-07-31T06:13:18Z", "updatedAt": "2026-08-13T12:36:56Z", "timestamp": "2026-08-13T12:36:56Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "performance", "intel-gpu", "ready", "verified" ], "author": "pmanczak", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50535", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Use AITER tuned GEMM for the MoE router gate", "text": "## Purpose The MoE gate linear layer was set to use a bf16×bf16→fp32 GEMM on tier 5 for faster performance, but it is conditioned on Hopper and Blackwell. This enables it for AITER when the underlying tuned GEMM shapes exist. ### Trace comparison \"pr_fig1_cpu_stack\" On the CPU side, the gate previously performed two operations: a conversion of its input to fp32, followed by the multiply. AITER was already invoked, but with fp32 operands it found no tuned kernel and fell through to `aten::linear` → `aten::matmul` → `aten::mm`. The input now remains in bf16, AITER dispatches a tuned kernel directly, and the cast is moved after the GEMM, where it applies only to the small output. \"pr_fig2_gpu_kernels\" The GPU stream shows the same change. The `add_rmsnorm_quant` kernel on the left is unaffected by this PR and serves as a scale reference between the two panels. ## Test Plan `tests/kernels/test_rocm_aiter_router_gemm.py`, skipped unless ROCm and AITER tuned GEMM are available. It verifies agreement with an fp64 reference, top-k expert selection, and equivalence with the `F.linear` fallback it replaces, across three router shapes and M from 1 to 128. End-to-end throughput, latency and accuracy were measured on MI355X against an otherwise identical build. ## Test Result 81/81 kernel tests pass on gfx950 with `VLLM_ROCM_USE_AITER=1`. MI355X, TP4, geomean over a 16-point concurrency sweep (ISL 1024/8192, OSL 1024): **TPOT -3.05%, output throughput +3.10%**. Accuracy is unchanged within one standard error: gsm8k strict-match 0.9515 against 0.9500 baseline, RULER niah_single_2 0.876 against 0.860. `ruff check` and `ruff format` are clean. ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 86.0 | 90.0 | **+4.6%** | 381 | 384 | 11.17 | 10.66 | -4.6% | | 1024 | 4 | 32 | 315.1 | 323.0 | **+2.5%** | 329 | 326 | 12.01 | 11.71 | -2.5% | | 1024 | 8 | 64 | 497.3 | 510.3 | **+2.6%** | 144 | 141 | 14.90 | 14.54 | -2.4% | | 1024 | 16 | 96 | 797.9 | 826.9 | **+3.6%** | 221 | 213 | 18.08 | 17.44 | -3.5% | | 1024 | 32 | 128 | 1180.4 | 1209.6 | **+2.5%** | 364 | 350 | 23.35 | 22.78 | -2.4% | | 1024 | 64 | 256 | 1786.7 | 1834.7 | **+2.7%** | 475 | 454 | 30.59 | 29.81 | -2.5% | | 1024 | 128 | 384 | 2577.1 | 2633.2 | **+2.2%** | 1079 | 1017 | 41.06 | 40.17 | -2.2% | | 1024 | 256 | 768 | 3850.1 | 3944.5 | **+2.5%** | 1819 | 1667 | 56.98 | 55.72 | -2.2% | | 8192 | 1 | 16 | 82.0 | 85.6 | **+4.3%** | 360 | 345 | 11.84 | 11.35 | -4.1% | | 8192 | 4 | 32 | 292.9 | 300.2 | **+2.5%** | 476 | 468 | 12.83 | 12.51 | -2.5% | | 8192 | 8 | 64 | 420.0 | 432.1 | **+2.9%** | 697 | 673 | 17.25 | 16.70 | -3.2% | | 8192 | 16 | 96 | 605.9 | 627.0 | **+3.5%** | 891 | 854 | 23.64 | 22.75 | -3.8% | | 8192 | 32 | 128 | 848.5 | 873.9 | **+3.0%** | 1911 | 1851 | 33.13 | 32.15 | -3.0% | | 8192 | 64 | 256 | 1136.1 | 1174.3 | **+3.4%** | 3427 | 3250 | 49.81 | 48.15 | -3.3% | | 8192 | 128 | 384 | 1452.9 | 1502.4 | **+3.4%** | 7128 | 6696 | 77.31 | 74.99 | -3.0% | | 8192 | 256 | 768 | 1731.1 | 1792.3 | **+3.5%** | 15375 | 14468 | 128.75 | 124.28 | -3.5% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+2.9%** | -3.6% | -2.8% | | ISL 8192 | **+3.3%** | -4.2% | -3.3% | | **all 16** | **+3.1%** | -3.9% | -3.0% | Per-case expansion of the geomean quoted above. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9500 | 0.9515 | | gsm8k flexible-extract | 0.9538 | 0.9553 | | RULER niah_single_2 @16k | 0.860 | 0.876 |", "url": "https://github.com/vllm-project/vllm/pull/50535", "createdAt": "2026-07-31T06:14:30Z", "updatedAt": "2026-08-13T07:53:19Z", "timestamp": "2026-08-13T07:53:19Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50539", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Gemma4] Keep image bidirectional attention within the sliding window on the V2 model runner", "text": "## Purpose #47217 fixed Gemma4 vision models silently falling back to **causal-only** image attention whenever a single image's soft-token span exceeded the text sliding window (e.g. ~1120 soft tokens at `max_soft_tokens=1120` vs `sliding_window=1024`). The runner was dropping the *entire* bidirectional `mm_prefix` range for such an image; the fix gated that skip on a model capability flag, `mm_prefix_clamp_sliding_window`, so models that bound `mm_prefix` **in-kernel** keep the full range and let the kernel clamp it per-query — matching HF's `(causal OR blockwise) AND sliding_window` on local layers. That fix only landed in the **V1** `GPUModelRunner`. The V2 model runner has its own copy of the range-building logic in `vllm/v1/worker/gpu/attn_utils.py::compute_mm_prefix_ranges`, which still drops oversized ranges unconditionally: ```python for r in mm_feature.mm_position.extract_embeds_range(): if sliding_window is not None and (r[1] - r[0] + 1) > sliding_window: continue # <- no opt-out, unlike V1 ``` So the original defect is still reachable today on `VLLM_USE_V2_MODEL_RUNNER=1`. This matters more than it did at the time of #47217, because V2 is no longer opt-in for these configs — `VllmConfig.use_v2_model_runner` force-enables V2 for DSpark speculative decoding and for hybrid DFlash drafters, both of which are used with Gemma4. A user who serves Gemma4 with images and DSpark drafting gets the degraded mask with no warning and no error. The kernel-side clamp (`flash_attn.py`, `triton_attn.py`, `triton_unified_attention.py`) is already shared by both runners and needs no change — V2 simply never delivers the range to it. ### Fix - `compute_mm_prefix_ranges` gains a `clamp_in_kernel: bool = False` parameter that gates the oversized-range skip, mirroring V1's `_clamps_in_kernel`. The default preserves current behavior for every other PrefixLM model. - `DefaultModelState.build_attn_metadata` reads `mm_prefix_clamp_sliding_window` off the model and forwards it, exactly as `GPUModelRunner` does. No behavior change for V1, for non-Gemma4 models, or for images that already fit within the sliding window. ### Not a duplicate Checked before opening, per `AGENTS.md`: ```bash gh search issues \"compute_mm_prefix_ranges\" --repo vllm-project/vllm --include-prs # 0 results gh search issues \"mm_prefix_clamp_sliding_window\" --repo vllm-project/vllm --include-prs # only #47217 gh search issues \"mm_req_doc_ranges\" --repo vllm-project/vllm --include-prs ``` No open issue or PR covers the V2 side. The nearest open PRs touch adjacent code but not this logic: #47646 (revert of an FA4 mm_prefix *kernel* mask change, `flash_attn.py` only), #46558 (FlashInfer mm-prefix, `flashinfer.py` only), and #48907 / #46963 (V2 `attn_utils.py`, but KV-cache reshape). #47217 is the V1 fix being ported here — its changed-file list contains no V2 file. ## Test Plan Added four unit tests to `tests/v1/worker/test_attn_utils.py` covering `compute_mm_prefix_ranges` (previously untested): - `test_compute_mm_prefix_ranges_skips_oversized_range_by_default` — default path still drops an oversized span (regression guard for other models). - `test_compute_mm_prefix_ranges_keeps_oversized_range_when_clamped_in_kernel` — the fix: an oversized span survives when the model clamps in-kernel. - `test_compute_mm_prefix_ranges_keeps_fitting_range_either_way` — spans within the window are unaffected by the flag. - `test_compute_mm_prefix_ranges_ignores_non_visual_modalities` — audio is still skipped. ```bash pytest tests/v1/worker/test_attn_utils.py -q ``` ## Test Result All tests pass with the fix: ``` ........ [100%] 8 passed, 14 warnings in 4.15s ``` Reverting only the two source files and re-running (tests kept) fails, confirming the tests exercise the fixed behavior rather than passing vacuously: ``` FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_keeps_oversized_range_when_clamped_in_kernel FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_keeps_fitting_range_either_way FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_ignores_non_visual_modalities 3 failed, 5 passed ``` `ruff format --check` and `ruff check` are clean on all three changed files (the one remaining `UP038` in `attn_utils.py` is pre-existing on `main`, in `build_attn_metadata`, and untouched here). ### Model evaluation This change affects generated output for one narrow case: Gemma4 vision requests under the V2 model runner where a single image's soft-token span exceeds `sliding_window` (i.e. `max_soft_tokens=1120` against `sliding_window=1024`; the default `num_soft_tokens=280` on the released checkpoints is unaffected, so most serving configurations see no change at all). The expected effect is the one measured in #47217 — restoring blockwise-bidirectional image attention instead of causal-only — with V1 as the reference implementation. I have not re-run the end-to-end vision eval from #47217 under V2; I don't have a Hopper box free for it right now. Flagging that gap explicitly rather than implying coverage I don't have. If a reviewer wants those numbers before merging, say so and I'll get them. ## AI assistance AI assistance (Claude) was used on this PR: tracing the V1/V2 divergence, drafting the patch and tests, and drafting this description. The commit carries a `Co-authored-by:` trailer. I have reviewed every changed line, run the tests and linters reported above, and can defend the change end-to-end. ## Related - #47217 — the V1 fix this ports (merged) - #42175 — introduced Gemma4 FA4 + FlashAttention `mm_prefix` support", "url": "https://github.com/vllm-project/vllm/pull/50539", "createdAt": "2026-07-31T06:51:39Z", "updatedAt": "2026-08-12T19:29:36Z", "timestamp": "2026-08-12T19:29:36Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "mrv2" ], "author": "dhruvil237", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] strict=false in response_format json_schema", "text": "Per OpenAI semantics, strict=false means best-effort: the schema may guide the model via the prompt, but the output is not grammar constrained. vLLM previously ignored the flag and always applied guided decoding. Now skip the json structured-output override when strict=false, on both Chat Completions and Responses APIs. Default (absent) and strict=true keep full guided decoding, so existing clients are unaffected. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50544", "createdAt": "2026-07-31T07:25:38Z", "updatedAt": "2026-08-13T16:55:48Z", "timestamp": "2026-08-13T16:55:48Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "frontend", "tool-calling", "cohere" ], "author": "wangln19", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50554", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3][ROCm] Fuse KDA upper-triangle zeroing", "text": "Avoid a separate full `Akk` buffer memset by writing the required zeros in the existing fused solve kernel. ## Purpose Kimi-K3's ROCm KDA prefill path allocates `Akk` with `torch.zeros`, even though the fused inter/solve Triton kernel already writes its lower triangle. This change makes the fused kernel explicitly write the required upper-triangle zeros, allowing `Akk` to use `torch.empty` and avoiding a separate full-buffer memset. It supports: - Fixed-length and variable-length inputs - Safe-gate and standard-gate paths - Partial chunks with boundary masking No documentation update is required because this is an internal performance optimization with no API or model-support changes. ## Test Plan Run the targeted ROCm tests: python3 -m pytest -q tests/models/kimi_k3/test_amd_kda.py Run Ruff checks: ruff check \\ vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py \\ tests/models/kimi_k3/test_amd_kda.py ruff format --check \\ vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py \\ tests/models/kimi_k3/test_amd_kda.py Validate performance using Kimi-K3 TP8 on 8 × MI355X: - Input length: 100,000 tokens - Output length: 20 tokens - Concurrency: 1 - Prefix caching: disabled - Five measured requests per run - Two independent baseline and patched runs ## Test Result ### Correctness Targeted ROCm tests: 4 passed The tests cover: - Fixed-length input with `safe_gate=False` - Fixed-length input with `safe_gate=True` - Variable-length input with `safe_gate=False` - Variable-length input with `safe_gate=True` Additional A/B validation confirmed: - `Akk` matched the original implementation bit-for-bit - End-to-end KDA output matched bit-for-bit - The upper triangle remained zero for every tested sequence and partial chunk - Ruff lint and formatting checks passed ### Accuracy / Numerical Parity The optimization does not change the KDA arithmetic. It only moves the required upper-triangle zero writes from a standalone `torch.zeros` initialization into the existing fused solve kernel. A direct baseline-versus-patched kernel comparison was performed with identical inputs. Results: - `Akk` matched the original zero-initialized implementation bit-for-bit. - The complete KDA output matched bit-for-bit. - Sequence lengths tested: `64`, `65`, `127`, `128`, `129`, and `1024`. - Partial chunks and chunk-boundary cases were included. - Fixed-length and variable-length layouts were tested. - Both `safe_gate=True` and `safe_gate=False` produced an exactly zero upper triangle. - No NaNs or numerical regressions were observed. - All TP8 serving requests completed successfully. The 100K serving environment was not deterministic at the generated-text level: repeated runs of the same patched server also produced different generated text. Therefore, generated-text equality was not used as numerical-parity evidence. Kernel-level tensor comparison was used instead and showed bit-for-bit output parity. ### Kernel Performance Same-GPU alternating A/B using Kimi-K3 TP8 per-rank shapes: Tokens Baseline median Patched median Improvement 8,192 0.1614 ms 0.1580 ms 2.08% 32,768 0.5193 ms 0.5157 ms 0.70% 100,000 1.5398 ms 1.5250 ms 0.96% ### End-to-End Performance Kimi-K3 TP8, 100K input, concurrency 1: Baseline mean TTFT, run 1: 7223.33 ms Baseline mean TTFT, run 2: 7221.91 ms Patched mean TTFT, run 1: 7206.20 ms Patched mean TTFT, run 2: 7210.31 ms Baseline average: 7222.62 ms Patched average: 7208.26 ms Improvement: 14.36 ms (0.20%) All serving requests succeeded. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR has been described. - [x] The test plan includes the relevant test commands. - [x] Correctness and performance results have been provided. - [x] Documentation impact was considered; no update is required.
", "url": "https://github.com/vllm-project/vllm/pull/50554", "createdAt": "2026-07-31T08:59:11Z", "updatedAt": "2026-08-13T04:49:52Z", "timestamp": "2026-08-13T04:49:52Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50592", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3][AMD] Return KDA and MLA projection outputs directly", "text": "## Purpose Kimi-K3's AMD attention paths copied each output projection into a caller-owned buffer even though the projection already returns a tensor with the required shape, dtype, layout, and lifetime. This PR removes that redundant post-projection allocation and copy from both attention families: - 69 KDA layers - 24 MLA layers - 93 post-attention-projection copies per model step in total The functionality from #50847 is now folded into this PR. #50654 is separate and complementary: it optimizes the earlier KDA recurrent/conv/RMSNorm path, while this PR removes the later copy after the output projection. ## Implementation - Return KDA output-projection storage directly from both the shared KDA implementation and the dedicated full-rank Kimi-K3 implementation selected by current `main`. - Return MLA output-projection storage directly. - Use one return-value contract in `KimiDecoderLayer` for KDA and MLA. - Preserve attention arithmetic, state updates, normalization, projection, tensor-parallel communication, and HIP-graph replay behavior. ## Validation ```text Targeted ownership tests: 3 passed Ruff 0.14.0 check: passed Ruff 0.14.0 format check: passed git diff --check: passed DCO: passed ``` The tests cover KDA dispatch ownership, the concrete full-rank Kimi-K3 KDA forward path, and MLA dispatch ownership. In each case, the returned tensor is the exact projection tensor and has the same `data_ptr()`. ## Fixed-cohort decode A/B ### Setup ```text GPU: 8 x AMD Instinct MI355X (gfx950) Parallelism: TP8 Model: Kimi-K3, BF16 activations / MXFP4 weights Decode: non-speculative Prefix caching: enabled Prompt: 63,911-token shared prefix + 4,089-token suffix Output: 1,024 tokens per request, ignore EOS Concurrency: C16 and C24 fixed cohorts HIP graph: FULL, capture sizes 1/16/24 Sampling: greedy, seed 42 Runs: 1 warmup + 3 measured runs per concurrency and variant Success rate: 240/240 measured requests ``` Both variants used image `localhost/kimi-pp8-upstream-clean` (vLLM `beca88e59`, AITER `99733dc00`). The baseline was the unmodified image. The candidate mounted only this PR's AMD runtime files from `43d81ba3e`. The relevant AMD Kimi-K3 files in the image are identical to the rebased `main` baseline before this PR. ### Results Means over three measured runs. TPOT and ITL are end-to-end serving metrics, not the reviewer's pure-GPU trace period. | Cohort | Metric | Baseline | Combined PR | Delta | |---|---|---:|---:|---:| | C16 | Mean TPOT | 73.141 ms | 72.428 ms | **-0.713 ms (-0.97%)** | | C16 | Median ITL | 70.488 ms | 70.186 ms | **-0.302 ms** | | C16 | Output throughput | 193.575 tok/s | 195.555 tok/s | **+1.02%** | | C24 | Mean TPOT | 74.958 ms | 74.457 ms | **-0.501 ms (-0.67%)** | | C24 | Median ITL | 71.311 ms | 70.916 ms | **-0.396 ms** | | C24 | Output throughput | 278.245 tok/s | 280.452 tok/s | **+0.79%** | Mean-TPOT standard deviation across the three runs was 0.184 ms / 0.348 ms for baseline C16/C24 and 0.019 ms / 0.122 ms for candidate C16/C24. ### Interpretation The reviewer's trace found that the complete 93-copy group costs approximately 0.495 ms at C16 and 0.563 ms at C24. The combined A/B removes all 93 copies and measures a 0.713 ms C16 and 0.501 ms C24 TPOT reduction. C24 closely matches the trace-derived removable cost. C16 is modestly larger, which is plausible because returning projection storage directly also changes allocator and lifetime behavior; the baseline's run-to-run variance is also higher. The absolute TPOT values should not be compared directly with the reviewer's 32.82/35.98 ms pure-GPU periods because TPOT includes serving and scheduling overhead. The A/B delta is the relevant same-harness comparison. ## Scope summary ```text #50592: remove 69 KDA + 24 MLA post-projection copies #50847: functionality absorbed by this PR #50654: separate earlier KDA recurrent-output/fusion optimization ```", "url": "https://github.com/vllm-project/vllm/pull/50592", "createdAt": "2026-07-31T16:26:50Z", "updatedAt": "2026-08-13T00:33:02Z", "timestamp": "2026-08-13T00:33:02Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "rocm", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50594", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix MiniMax M3 prompt reasoning initialization", "text": "## Purpose Fixes #46042. Fixes #50549. MiniMax-M3 adaptive reasoning can leak a leading `` into streaming `delta.content`. The existing initialization scans the entire rendered prompt with `is_reasoning_end()`. Because the prompt can contain paired `` markers as instructions or examples, the parser may incorrectly conclude that reasoning has already ended before generation starts. This change: - adds a prompt-boundary-specific reasoning-state hook while preserving the existing behavior by default - initializes MiniMax-M3 from `thinking_mode`: `enabled` starts in reasoning, `disabled` starts after reasoning, and `adaptive` lets generated markers determine the state - uses the hook in Chat Completions, the Responses API, streaming parsing, and structured-output initialization - preserves the deferred `adaptive=None` state through the structured-output gate instead of falling back to a full-prompt scan - scopes generated-marker checks to output tokens so instructional prompt markers cannot activate the grammar early - constrains unresolved adaptive sampling to grammar-valid tokens plus every decoded reasoning-marker tokenization - replays adaptive direct-content tokens through the grammar, including ambiguous prefixes and speculative draft state - respects the final reasoning boundary when continuing an existing assistant message, including explicit open continuations under disabled mode, and scopes continuation marker scans to that final message only - preserves open and closed `thinking` content-part state during continuation - propagates incomplete Responses reasoning items to frontend and engine parsers with the same request-local state - clears continuation-only parser state and refreshes the frontend parser after Responses built-in tool turns rerender the prompt - forwards request-local reasoning mode and prompt state through offline `LLM.chat()` and `enqueue_chat()` into the synchronous engine - keeps batch reasoning parsers request-local so continuation state cannot leak between conversations - initializes each engine-side structured-output parser from its prompt exactly once - adds focused regressions for instructional markers, adaptive marker boundaries, structured output, and all MiniMax-M3 thinking modes ## Related work and scope - #45718 fixed split-token MiniMax-M3 marker parsing, but not prompt-boundary state initialization. - #50152 identified the same full-prompt scan failure and demonstrated the production symptom, but closed without merging. - #48550 also addresses prompt-mode initialization while adding broader speculative-decoding and tool-parser changes. - This PR limits the fix to prompt-boundary reasoning initialization across online and offline chat paths and Responses, plus the structured-output state transitions needed to preserve it. ## Test Plan - Run focused parser and structured-output regressions: `uv run pytest -q tests/reasoning/test_minimax_m3_reasoning_parser.py tests/v1/structured_output/test_reasoning_structured_output.py` - Run offline chat prompt-state regression: `uv run pytest -q tests/entrypoints/llm/test_offline_reasoning_state.py` - Run batch prompt-state regressions: `uv run pytest -q tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py -k batch_forwards_prompt_reasoning_state` - Run Responses continuation regressions: `uv run pytest -q tests/entrypoints/openai/responses/test_serving_responses.py` - Run static validation on the changed Python files. - Serve `MiniMaxAI/MiniMax-M3` with the patch and send adaptive and enabled streaming chat-completion requests, including a structured response format. - Confirm that reasoning markers never appear in `delta.content`, reasoning remains in the reasoning field, and the output grammar starts only after generated reasoning ends or advances the first direct adaptive content token. ## Test Result Passed: - MiniMax-M3 reasoning parser: 44 tests passed - reasoning-aware structured output: 21 tests passed - offline chat prompt-state forwarding: 1 test passed - batch prompt-state forwarding: 3 tests passed - Responses serving: 23 tests passed, 1 expected xfail - `ruff check`: all checks passed - `ruff format`: all changed files formatted - Python `compileall` - `git diff --check` The 92 focused tests were run in a CPU-only temporary environment with `VLLM_TARGET_DEVICE=empty`; the repository-wide GPU CI has not run yet. Live MiniMax-M3 evaluation is pending because this workspace has no GPU/model deployment. It should be completed before merge. ## AI assistance OpenAI Codex assisted with implementation and regression-test drafting. The commits include `Assisted-by` trailers. The submitter requested the changes after reviewing the proposed scope.", "url": "https://github.com/vllm-project/vllm/pull/50594", "createdAt": "2026-07-31T16:34:24Z", "updatedAt": "2026-08-13T17:34:44Z", "timestamp": "2026-08-13T17:34:44Z", "metrics": { "reactions": 2, "comments": 24 }, "labels": [ "bug", "structured-output", "frontend", "tool-calling" ], "author": "jinbagi", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50595", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Structured Output] Mask request stop tokens in xgrammar until grammar terminates", "text": "## Purpose Following #49227's merge, remove the patch used in HarmonyParser and update test to mirror production path. ## Test Plan `pytest tests/parser/test_harmony.py` ## Test Result `64 passed, 38 warnings in 26.74s` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50595", "createdAt": "2026-07-31T16:41:25Z", "updatedAt": "2026-08-13T16:02:27Z", "timestamp": "2026-08-13T16:02:27Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "structured-output", "tool-calling", "verified" ], "author": "yzong-rh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50597", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm]Remove special-case SiTU support model-specific gating", "text": "## Purpose Mxfp4MoEMethod previously contained a model-specific predicate (_use_k3_situ_aiter) that special-cased the Kimi-K3 SiTU activation, gating three separate code paths — backend selection in __init__, size round-up bypass in maybe_roundup_sizes, and an entirely separate weight-shuffle method _setup_kernel_k3_situ in process_weights_after_loading. This tied SiTU behavior to a specific model identity rather than to the activation type itself. This PR removes that special-casing and routes SiTU through the standard machinery: MoEActivation.SITU is added to AiterExperts._supports_activation, so the oracle's is_supported_config accepts SiTU models without any model-specific intervention in Mxfp4MoEMethod mxfp4_round_up_hidden_size_and_intermediate_size gains an activation parameter; the ROCm 256-alignment is skipped when activation == MoEActivation.SITU, since the SiTU FlyDSL kernel pads internally and the round-up would OOM native intermediate sizes convert_weight_to_mxfp4_moe_kernel_format gains an activation parameter; the AITER_MXFP4_BF16 branch dispatches to the shuffle_weight_a16w4/e8m0_shuffle shuffle for SITU and the existing _shuf_w/_shuf_s shuffle for all other models _setup_kernel passes activation=self.moe.activation through to both functions above, and its shape assertions are gated on activation != MoEActivation.SITU since SITU's native intermediate sizes are not 256-aligned process_weights_after_loading calls _setup_kernel unconditionally for all MXFP4 paths _use_k3_situ_aiter, self.is_k3_situ_aiter, and _setup_kernel_k3_situ are deleted. Any future model using SiTU activation with MXFP4 weights on ROCm gfx950 will route correctly without requiring new special-case code. This work was done with AI assistance and reviewed by a human. ## Test Plan Validate Kimi-K3: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 vllm serve moonshotai/Kimi-K3 --host localhost --port 8001 --quantization mxfp4 --tensor-parallel-size 8 --max-model-len 32768 --max-num-seqs 59 --trust-remote-code ## Test Result lm_eval --model local-completions --model_args model=moonshotai/Kimi-K3,base_url=http://localhost:8001/v1/completions,tokenizer_backend=huggingface,trust_remote_code=True --tasks gsm8k --num_fewshot 5 Original: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9666|± |0.0049| | | |strict-match | 5|exact_match|↑ |0.9666|± |0.0049| Updated: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9651|± |0.0051| | | |strict-match | 5|exact_match|↑ |0.9651|± |0.0051|", "url": "https://github.com/vllm-project/vllm/pull/50597", "createdAt": "2026-07-31T16:57:26Z", "updatedAt": "2026-08-13T06:43:05Z", "timestamp": "2026-08-13T06:43:05Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "rocm", "ready", "needs-rebase", "quantization" ], "author": "stacyroberts", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50599", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Fix dead class references in MoE design docs", "text": "## Purpose Six symbol references in the two MoE design docs no longer resolve against main. `docs/design/moe_kernel_features.md`: - The all2all backend table lists a `naive` backend whose Subclass column is an autoref to `vllm.model_executor.layers.fused_moe.layer.FusedMoE`. That class no longer exists, so the cross-reference cannot resolve. `naive` is also gone as a backend: `ParallelConfig.__post_init__` logs \"The '%s' all2all backend has been removed. Falling back to 'allgather_reducescatter'.\" for both `naive` and `pplx` (see #33728, #36294), and `allgather_reducescatter` is not in the table at all despite being the fallback. The row is renamed to `allgather_reducescatter` and pointed at `MoEPrepareAndFinalizeNaiveDPEPModular`, which is what that path constructs (`all2all_utils.py`, `use_ag_rs_all2all_kernels` branch). - The intro prose still calls the layer `FusedMoE`; it is `RoutedExperts`. - Two cross-reference labels are misspelled relative to their targets, which are already correct: `CutlasBatchedExpertsFp8` becomes `CutlassBatchedExpertsFp8`, and `TrtLlmNvfp4ExpertsModular` becomes `TrtLlmNvFp4ExpertsModular`. `docs/design/fused_moe_modular_kernel.md`: - `TopKWeightAndReduceNoOp` becomes `TopKWeightAndReduceNoOP`, since the class name ends in a capitalised `OP`. - `FusedMoEPremuteExpertsUnpermute` has never existed under that name; the surrounding text already uses `FusedMoEExpertsModular`. Only the cells I could verify against the code were changed. The quantization-type columns on the renamed row are left exactly as they were. Two further staleness items in these files are deliberately not touched, because fixing them correctly needs a judgment call from someone who owns this code: 1. The \"Modular Kernel families\" table has a `flashinfer` row naming `FlashInferCutlassMoEPrepareAndFinalize`. That class is gone and `flashinfer` is not in the `All2AllBackend` literal; the real classes are `FlashInferNVLinkOneSidedPrepareAndFinalize` and `FlashInferNVLinkTwoSidedPrepareAndFinalize`. Splitting the row needs a decision on which side pairs with `FlashInferExperts`, given one-sided also dispatches bf16 and mxfp8. 2. In `fused_moe_modular_kernel.md`, the `select_gemm_impl` and `maybe_make_prepare_finalize` sections list `Fp8MoEMethod`, `CompressedTensorsW8A8Fp8MoEMethod`, `CompressedTensorsW8A8Fp8MoECutlassMethod` (which does not exist) and `ModelOptNvFp4FusedMoE`. None of those define `select_gemm_impl` today, and `ModelOptNvFp4FusedMoE.maybe_make_prepare_finalize` now raises `ValueError(\"uses the new modular kernel initialization logic\")`, the opposite of what the doc describes. Happy to fold either in if a reviewer tells me the intended answer. Docs only, no behavior change. ## Test Plan - `pre-commit run --files docs/design/moe_kernel_features.md docs/design/fused_moe_modular_kernel.md` - Resolve all mkdocs `][vllm...]` autoref targets across `docs/` against an AST index of modules and class members, before and after. - Confirm each replacement name exists, and each removed name is absent, by grepping for the class definitions under `vllm/` and `tests/`. - Read the `naive` and `pplx` removal in `vllm/config/parallel.py` and the `use_ag_rs_all2all_kernels` branch in `vllm/model_executor/layers/fused_moe/all2all_utils.py`. ## Test Result - `pre-commit run`: `typos`, `markdownlint-cli2`, `Check for spaces in all filenames`, `Update Dockerfile dependency graph` and the config validation hook all Passed. - Autoref resolution: `vllm.model_executor.layers.fused_moe.layer.FusedMoE` was the only genuinely unresolvable target in these two files, and it resolves after the change. - All six replacement names found at their definition sites: `MoEPrepareAndFinalizeNaiveDPEPModular` in `prepare_finalize/naive_dp_ep.py`, `CutlassBatchedExpertsFp8` in `experts/cutlass_moe.py`, `TrtLlmNvFp4ExpertsModular` in `experts/trtllm_nvfp4_moe.py`, `TopKWeightAndReduceNoOP` in `topk_weight_and_reduce.py`, `FusedMoEExpertsModular` in `modular_kernel.py`, and `RoutedExperts` in `routed_experts.py`. - All removed names: no matches anywhere under `vllm/` or `tests/`, under either spelling. I do not have GPU access, so this was verified statically against the source rather than by running MoE kernels.", "url": "https://github.com/vllm-project/vllm/pull/50599", "createdAt": "2026-07-31T16:58:58Z", "updatedAt": "2026-08-13T05:00:56Z", "timestamp": "2026-08-13T05:00:56Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready" ], "author": "UgaTheDev", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50605", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm]: Bump torch 2.13, triton 3.8, torchaudio, torchvision", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50605", "createdAt": "2026-07-31T17:52:47Z", "updatedAt": "2026-08-12T22:23:49Z", "timestamp": "2026-08-12T22:23:49Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "ready", "needs-rebase", "ci/build" ], "author": "Rohan138", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50620", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][NIXL] Include transfer mode (push/pull) in the compatibility hash", "text": "#### Overview: Include the NIXL transfer mode (push vs pull) in the connector so a push (WRITE) connector and a pull (READ) connector can never be paired, and so an external router can distinguish them. Follow-up to #49230 (now merged), addressing review feedback from @iyastreb ([#49230 thread](https://github.com/vllm-project/vllm/pull/49230#discussion_r3686841923), [this PR's thread](https://github.com/vllm-project/vllm/pull/50620#discussion_r3702697050)). #### Details: The push (`NixlPushConnector`, WRITE) and pull (`NixlConnector`, READ) connectors use incompatible transfer protocols, but nothing prevented them from being paired across prefill/decode, and the transfer mode wasn't visible to an external router. **Compatibility hash (worker side):** - `NixlBaseConnectorWorker._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorWorker`, threaded into `compute_nixl_compatibility_hash`. - Add `transfer_mode` to the compatibility-hash factors so mismatched connectors are rejected early at handshake with a clear message. - Bump the NIXL connector version **6 → 7** and document the new factor. **Router support (scheduler side):** - `NixlBaseConnectorScheduler._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorScheduler`. - Emit `transfer_mode` in the `kv_transfer_params` returned by both the pull and push `request_finished` paths, so an external router can route pull vs push producers. Consumed by [vllm-project/router#187](https://github.com/vllm-project/router/pull/187). **Not a duplicate:** this transfer-mode work was split out of #49230 (which fixed the speculative-config factors) to keep that PR scoped; no other open PR addresses mixing push/pull connectors. **Tests run:** - `.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py -k \"transfer_mode_changes_compatibility_hash or scheduler_advertises_transfer_mode or speculative_config_compatibility_hash\" -q` — **passed**. - `pre-commit run --files ` — all applicable hooks **passed**. - GPU-dependent handshake tests were not run locally (macOS, no CUDA); relying on CI. AI assistance was used to investigate, implement, and test this change. Every changed line was reviewed by the human submitter before submission. #### Where should the reviewer start? - `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py` — `compute_nixl_compatibility_hash` (new `transfer_mode` factor + version bump). - `base_worker.py` / `push_worker.py` — worker `_TRANSFER_MODE`. - `base_scheduler.py` / `pull_scheduler.py` / `push_scheduler.py` — scheduler `_TRANSFER_MODE` emitted into `kv_transfer_params`. - `tests/v1/kv_connector/unit/test_nixl_connector.py` — hash test, scheduler-mode test, and the pull `kv_transfer_params` assertion.", "url": "https://github.com/vllm-project/vllm/pull/50620", "createdAt": "2026-07-31T19:41:03Z", "updatedAt": "2026-08-13T17:32:37Z", "timestamp": "2026-08-13T17:32:37Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "documentation", "ready", "kv-connector" ], "author": "tzulingk", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50622", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends", "text": "# [ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends ## Summary Split overloaded `AITER_MXFP4_BF16` into distinct backends: - **`AITER_MXFP4_BF16`** → CK only (`AiterExperts`), behavior unchanged (gfx950). - **`AITER_TRITON_MXFP4_BF16`** (new) → aiter Triton `moe_gemm_a16w4` (`AiterW4A16ExpertsMonolithic`), available on **gfx942 / gfx950 / gfx1250**, selectable via `--moe-backend aiter_triton_mxfp4_bf16` and auto-selected for gpt-oss where CK is unavailable. ## Not a duplicate No open PR splits the CK vs Triton MXFP4-BF16 backends. ## Test plan / results Hardware: AMD MI300X (gfx942). Model: `openai/gpt-oss-120b`. Validated on the committed branch. **Unit test** (`tests/kernels/moe/test_ocp_mx_moe.py`): ``` pytest -s -v tests/kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle ``` → 3 passed (TRITON, TRITON_UNFUSED, AITER_TRITON_MXFP4_BF16), 2 skipped (AITER_MXFP4_BF16, AITER_MXFP4_FP8 — gfx950-only). **Accuracy** (lm_eval gsm8k, flexible-extract, TP1): `aiter_triton_mxfp4_bf16` = **0.903** — matches the pre-split path and the `--moe-backend triton` baseline (~0.906), confirming the refactor is numerically neutral. ### Performance sweep `vllm bench serve`, random dataset, input=1024 / output=1024, num_prompts = concurrency × 10. aiter Triton W4A16 (`aiter_triton_mxfp4_bf16`) vs the OAI `triton_kernels` backend (`--moe-backend triton`), output-token throughput (tok/s), mean ITL (ms), mean TTFT (ms): | TP | conc | aiter tok/s | triton tok/s | speedup | aiter ITL | triton ITL | aiter TTFT | triton TTFT | |----|------|-------------|--------------|---------|-----------|------------|------------|-------------| | 1 | 1 | 216 | 82 | 2.64x | 4.55 | 12.13 | 84 | 94 | | 1 | 8 | 740 | 467 | 1.59x | 9.74 | 16.70 | 1103 | 454 | | 1 | 32 | 1656 | 1137 | 1.46x | 17.58 | 27.46 | 1804 | 728 | | 1 | 128 | 3124 | 2651 | 1.18x | 39.19 | 47.39 | 1848 | 927 | | 2 | 1 | 276 | 107 | 2.58x | 3.56 | 9.30 | 71 | 72 | | 2 | 8 | 1086 | 706 | 1.54x | 6.74 | 10.96 | 645 | 388 | | 2 | 32 | 2484 | 1792 | 1.39x | 12.10 | 17.27 | 812 | 620 | | 2 | 128 | 4803 | 4016 | 1.20x | 25.61 | 31.14 | 1076 | 762 | | 4 | 1 | 309 | 117 | 2.65x | 3.18 | 8.52 | 54 | 58 | | 4 | 8 | 1430 | 859 | 1.67x | 5.16 | 8.92 | 445 | 408 | | 4 | 32 | 3691 | 2520 | 1.46x | 8.13 | 12.29 | 562 | 427 | | 4 | 128 | 7589 | 5845 | 1.30x | 16.21 | 21.32 | 681 | 591 | | 8 | 1 | 290 | 143 | 2.03x | 3.40 | 6.96 | 54 | 55 | | 8 | 8 | 1667 | 873 | 1.91x | 4.35 | 8.69 | 460 | 489 | | 8 | 32 | 4937 | 3003 | 1.64x | 5.93 | 10.29 | 568 | 381 | | 8 | 128 | 10794 | 7304 | 1.48x | 11.05 | 16.91 | 832 | 634 | **Mean speedup 1.73× (range 1.18–2.65×)** across all 16 points; ITL consistently lower. aiter TTFT is higher than triton at TP1 mid-concurrency (prefill scheduling) and evens out at TP4/TP8. > Known issue, tracked separately: the aiter Triton W4A16 path shows a GPQA > accuracy regression at **TP4** (a partial-scale-group shard shape); TP1/2/8 > are at parity (GPQA aiter 0.65/0.67/0.63 vs triton 0.67/0.66/0.67). This is an > independent kernel bug, not caused by the backend split, and is not a blocker > for this structural refactor. ## AI assistance This change was developed with AI assistance. Andy has reviewed every changed line and run the tests/evals reported above.", "url": "https://github.com/vllm-project/vllm/pull/50622", "createdAt": "2026-07-31T20:14:49Z", "updatedAt": "2026-08-13T15:48:35Z", "timestamp": "2026-08-13T15:48:35Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "afriedri", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50665", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fuse Kimi-K3 FP8 pre-route projections", "text": "## Purpose Integrate the Kimi-K3 gfx950 pre-route primitives through one AMD-only, fail-closed ownership boundary: - prepack routed-down and shared-expert weights after checkpoint loading; - produce routed inputs, shared-expert output, and FP32 router logits in the AITER mixed-precision grid; and - pass the precomputed shared output through a scoped seam so the generic shared-expert lifecycle does not repeat the work. FP8 is opt-in with `VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8=1`; the exact BF16 path is separately opt-in, and FP8 takes precedence when both are enabled. Unsupported shapes, dtypes, devices, architectures, LoRA configurations, or AITER installations use the existing implementation. NVIDIA and other models are unchanged. Review map: - `amd/ops/moe_preroute.py`: checked AITER adapter and weight packing; - `amd/linear.py`: Kimi-K3 lifecycle and dispatch; - `shared_experts.py`: scoped precomputed-output ownership seam; - tests: operator contract, fallback, graph replay, and equal-logit behavior. Depends on ROCm/aiter#4498 and ROCm/aiter#4504. ## Test plan Tested on 8x MI355X (`gfx950`) with the public Kimi-K3 image and `moonshotai/Kimi-K3@9f62e4e9`. Kernel heads: AITER #4498 `0d755fe1` and #4504 `3d1208d4`; the command below pins the vLLM base and PR revisions. Fetch and verify the vLLM source: ```bash git clone https://github.com/vllm-project/vllm.git vllm git -C vllm fetch origin refs/pull/50665/head:pr-50665 test \"$(git -C vllm rev-parse pr-50665)\" = \\ 84a9b03c803421804eb3062de43022edce6a8992 git -C vllm checkout --detach 6c91de36897932ba9b5adb11992235a2a789e009 git -C vllm diff 6c91de36897932ba9b5adb11992235a2a789e009..pr-50665 \\ --binary | git -C vllm apply - git -C vllm diff --check ``` Construct both arms from the public image using the complete verified vLLM and AITER trees above, including AITER `csrc/` and `hsa/`; do not copy individual files. Verify `import vllm, vllm._C, aiter` in the GPU container before testing to prevent a stale or missing overlay. Focused tests and companion microbenchmarks: ```bash python -m pytest -q \\ ../aiter/op_tests/flydsl_tests/test_kimi_k3_moe_preroute_bf16.py \\ ../aiter/op_tests/flydsl_tests/test_kimi_k3_moe_preroute_fp8.py \\ tests/model_executor/layers/fused_moe/test_shared_experts.py \\ tests/models/kimi_k3/test_amd_moe_preroute.py ``` Serve each arm with the same exact model and flags; set both pre-route variables to `0` for control: ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FP4BMM=1 \\ AITER_SITUV2_A8W4=1 AITER_BF16_FP8_MOE_BOUND=0 \\ VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=1 \\ VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8=1 \\ vllm serve /model --served-model-name moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 --trust-remote-code --moe-backend auto \\ --gpu-memory-utilization 0.95 --max-num-seqs 128 \\ --max-num-batched-tokens 4096 --max-model-len 1048576 \\ --enable-prefix-caching --kv-cache-dtype fp8 --reasoning-parser kimi_k3 ``` After one 8K/128 warmup, run three 8K/1K batch-one trials with seeds 1--3, temperature zero, and `--ignore-eos`. Run full `lm-eval==0.4.12` GSM8K on both arms: 1,319 questions, 5-shot, greedy completion, 2,048 generated tokens, concurrency 128, and seed 42. ## Test results - Combined focused suite: **21/21 passed**. - Changed-file pre-commit, `git diff --check`, and DCO passed. | MI355X microbenchmark | Control | Candidate | Speedup | | --- | ---: | ---: | ---: | | BF16 pre-route | 21.9358 us | 17.0914 us | **1.2834x** | | FP8 routed/shared output | 17.6833 us | 13.8263 us | **1.2790x** | | FP8 tri-projection | 20.3035 us | 10.6829 us | **1.9006x** | | TP8 8K/1K median | Control | Candidate | Change | | --- | ---: | ---: | ---: | | Decode throughput | 36.0596 tok/s/GPU | 38.2324 tok/s/GPU | **+6.026%** | | TPOT | 27.7319 ms | 26.1558 ms | **-1.5760 ms** | Full GSM8K: control **1270/1319**, candidate **1272/1319**, zero invalid or transport failures; 14 wins/12 losses (`p=0.8450`). This paired run found no statistically detectable accuracy difference. The public suite also exposed two exactly tied router logits (experts 111 and 184 at 189.0). The test requires the exact selected top-16 set and a strict 16/17 boundary (174.0 versus 167.0), without imposing undefined tie order. ## Overlap and limits - Operator tests establish the stated BF16/FP8 numerical contracts; GSM8K is the model-quality gate. - ROCm/aiter#4498 and ROCm/aiter#4504 own the kernels; this PR only owns vLLM lifecycle and dispatch. The two paths share one weight lifecycle, so splitting them would duplicate the model integration and shared-output ownership seam. ## Tool assistance OpenAI Codex assisted with implementation, tests, benchmarking, and drafting. ## Long-prefix agentic serving gate (2026-08-02) The exact user-requested serving sweep was run with official `aiperf==0.11.0` on 8x MI355X (TP8), FP8 KV cache, prefix caching, and no speculative decoding: ```bash aiperf profile --model moonshotai/Kimi-K3 --tokenizer moonshotai/Kimi-K3 \\ --tokenizer-trust-remote-code --url http://127.0.0.1:8000 --api-key EMPTY \\ --endpoint-type chat --streaming --use-server-token-count \\ --num-prefix-prompts 8 --prompt-prefix-length 63240 \\ --synthetic-input-tokens-mean 4760 --synthetic-input-tokens-stddev 0 \\ --output-tokens-mean 350 --output-tokens-stddev 0 \\ --extra-inputs ignore_eos:true --extra-inputs min_tokens:350 \\ --extra-inputs max_tokens:350 --warmup-request-count 3 --sweep-type zip \\ --concurrency 1,8,16,24,48 --request-count 5,40,80,120,240 \\ --random-seed 42 --ui simple ``` The parent, BF16 intermediate, and FP8 candidate each completed **485/485** requests (**1,455/1,455 total**) with byte-identical generated inputs, exactly 350 output tokens per request, and zero errors, cancellations, output-size mismatches, or endpoint fatal signatures. | Concurrency | Requests/arm | Output tok/s, parent -> BF16 -> FP8 | FP8 vs parent | Mean ITL, ms, parent -> FP8 | Mean TTFT, ms, parent -> FP8 | Mean E2E, ms, parent -> FP8 | |---:|---:|---:|---:|---:|---:|---:| | 1 | 5 | 14.12 -> 14.27 -> 14.41 | **+2.07%** | 51.70 -> 50.13 (**-3.02%**) | 6,743 -> 6,785 (+0.62%) | 24,785 -> 24,281 (**-2.03%**) | | 8 | 40 | 84.02 -> 83.23 -> 84.38 | +0.43% | 78.43 -> 78.02 (-0.53%) | 5,701 -> 5,714 (+0.23%) | 33,074 -> 32,941 (-0.40%) | | 16 | 80 | 145.22 -> 145.03 -> 145.36 | +0.09% | 85.87 -> 87.01 (+1.33%) | 7,776 -> 7,632 (-1.85%) | 37,744 -> 37,999 (+0.68%) | | 24 | 120 | 191.53 -> 191.62 -> 191.44 | -0.05% | 97.16 -> 94.91 (-2.32%) | 9,078 -> 10,069 (+10.92%) | 42,988 -> 43,191 (+0.47%) | | 48 | 240 | 296.30 -> 296.46 -> 296.51 | +0.07% | 124.75 -> 125.15 (+0.32%) | 12,306 -> 12,154 (-1.24%) | 55,845 -> 55,830 (-0.03%) | The FP8 path provides a clear concurrency-one benefit; c8-c48 is neutral. Both optimized support predicates require activation shape `(1, 7168)`, so larger scheduler batches are expected to use the existing fallback. This dispatch explanation is inferred from the source contract; no dispatch counter was collected. TTFT is queueing-sensitive, so the mixed TTFT deltas—especially c24—are reported without an improvement claim. Each cell is one profile trial and therefore a point estimate. Provenance: public image `vllm/vllm-openai-rocm:kimi-k3@sha256:5aa7e626ff73672f5ca7aae46754570488c23d33ca1ac90756a1d2d1a3fe099b`, model `moonshotai/Kimi-K3@9f62e4e9`, and random seed 42. The four runtime files at current PR head `84a9b03c803421804eb3062de43022edce6a8992` are SHA-256-identical to the measured candidate. The seven runtime files at AITER #4498 head `0d755fe1` and #4504 head `3d1208d4` are likewise byte-identical to the measured AITER stack.", "url": "https://github.com/vllm-project/vllm/pull/50665", "createdAt": "2026-08-01T08:37:37Z", "updatedAt": "2026-08-13T10:52:25Z", "timestamp": "2026-08-13T10:52:25Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "kimi", "k3" ], "author": "JohnQinAMD", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50685", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Keep Qwen3Next layer boundaries sequence parallel", "text": " ## Purpose Fixes #50681. Qwen3.6-35B-A3B produces corrupted output during single-token decode when expert parallelism and MoE sequence parallelism are enabled with DP2 and TP2. The Qwen3Next model path infers the hidden-state layout from the first tensor dimension, but during TP2 single-token decode the full input and each padded local shard can all have one row. Shape alone therefore cannot determine whether attention must first all-gather the input. This PR gives supported all-MoE Qwen3Next/Qwen3.5 models a fixed model-wide sequence-parallel contract, following the same structure used by Kimi K3: - shard once before entering the decoder stack; - all-gather before attention and reduce-scatter its output in every layer; - keep every decoder-layer boundary sequence parallel; and - gather once after the final norm. There is no runtime layout state to infer or pass through decoder-layer arguments. Qwen3Next configurations that mix dense and MoE layers retain the existing layer-local MoE sequence-parallel path instead of switching layouts inside the model loop. Qwen3.5 MoE inherits the shared model forward and decoder forward implementations, so no duplicate change is needed in `qwen3_5.py`. The MTP implementations call decoder layers directly and bypass the shared model forward. They therefore shard before that direct call and gather after the MTP norm to honor the decoder's fixed contract. This is compatibility with the new contract, not a claim that MTP already exhibited #50681 on `main`. No open PR references #50681, and searches for Qwen3Next single-token sequence-parallel all-gather fixes found no duplicate work. I used AI assistance during the analysis and implementation of this change. I reviewed every changed line and verified the results reported below. ## Test Plan Use `Qwen/Qwen3.6-35B-A3B` in BF16 with DP2, TP2, EP4, eager execution, greedy decoding, and the default `allgather_reducescatter` all-to-all backend: ```bash vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 2 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --enforce-eager \\ --max-model-len 512 \\ --max-num-seqs 4 \\ --gpu-memory-utilization 0.80 ``` Send a deterministic 64-token chat-completions request for `Explain quantum computing in simple terms:` with thinking disabled. Run code checks only on the three changed files: ```bash pre-commit run --files \\ vllm/model_executor/models/qwen3_next.py \\ vllm/model_executor/models/qwen3_next_mtp.py \\ vllm/model_executor/models/qwen3_5_mtp.py git diff --check ``` ## Test Result The four-GPU run confirmed the intended topology and backend: ```text world_size=4 [EP Rank 0/4] Expert parallelism is enabled. Using AgRsAll2AllManager all2all manager. ``` On current `main` (`89c8401c8a`), the non-MTP request reproduced corrupted and repetitive output: ```text To **E****E****E******** **Quant2** **Quant2** Quant2020** ``` With this change, the same request returned coherent output: ```text To understand quantum computing, it helps to first look at how our current computers work, and then see how quantum computers break the rules. ``` For scope validation, current `main` was also run with real Qwen3.6 MTP using one and two speculative tokens. Both produced coherent output. With two speculative tokens, 34 of 58 draft tokens were accepted, confirming that the draft path was active rather than entirely masked by target rejection. The MTP file changes in this PR are therefore contract adaptation, not a separate pre-existing output-corruption fix. All pre-commit hooks for the three changed files passed, including Ruff and mypy, and `git diff --check` passed. No test files were added. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] Before-and-after E2E results. - [x] AI assistance disclosure.
", "url": "https://github.com/vllm-project/vllm/pull/50685", "createdAt": "2026-08-01T13:18:59Z", "updatedAt": "2026-08-13T15:16:15Z", "timestamp": "2026-08-13T15:16:15Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "bug", "ready", "qwen" ], "author": "kzwrime", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50691", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix DeepSeek MoE sequence-parallel layout tracking", "text": "## Purpose `deepseek_v2.py` decides whether `hidden_states` are sequence-parallel by comparing the first dimension against the full token count. That inference is ambiguous, and single-token decode is exactly where it breaks. With TP=2 and one decode token, `ceil(1 / 2) == 1`, so a full input and a padded local shard both have one row: | Value | First dimension | Data represented | | --- | ---: | --- | | Full decode input | 1 | Real token | | TP rank 0 local shard | 1 | Real token | | TP rank 1 local shard | 1 | Padding row | When the check misfires, the layer skips its entry all-gather. Each TP rank then runs attention on a different row — rank > 0 holds padding — and the following `reduce_scatter` sums misaligned partials, so decode output is corrupted. This is the same root cause as #50681 / #50685, which fix `qwen3_next.py`. `deepseek_v2.py` carries an independent copy of the pattern, introduced earlier by #46635, so #50685 does not cover it. This PR mirrors that fix so the two read the same way. ### Sites fixed All four shape comparisons on `main` (`63e78ce`): 1. Layer-entry inference — [`deepseek_v2.py#L1300-L1304`](https://github.com/vllm-project/vllm/blob/63e78ce3652f4f94e9f484f40db71ca4cf019f21/vllm/model_executor/models/deepseek_v2.py#L1300-L1304) 2. All-gather that restores the full layout before a non-SP layer — [`#L1476-L1479`](https://github.com/vllm-project/vllm/blob/63e78ce3652f4f94e9f484f40db71ca4cf019f21/vllm/model_executor/models/deepseek_v2.py#L1476-L1479) 3. Aux hidden state capture (EAGLE) — [`#L1488`](https://github.com/vllm-project/vllm/blob/63e78ce3652f4f94e9f484f40db71ca4cf019f21/vllm/model_executor/models/deepseek_v2.py#L1488) 4. Trailing all-gather before the final norm — [`#L1503`](https://github.com/vllm-project/vllm/blob/63e78ce3652f4f94e9f484f40db71ca4cf019f21/vllm/model_executor/models/deepseek_v2.py#L1503) ### Approach Track the layout explicitly instead of inferring it, following #50685: - `DeepseekV2Model.forward` carries a `hidden_states_are_sequence_parallel` flag. - It is passed to each layer as `input_is_sequence_parallel`, replacing the in-layer inference. - It is set from `layer.use_sequence_parallel_moe` after each layer, and cleared wherever an all-gather restores the full layout. The reduce-scatter optimization itself is unchanged; only the layout bookkeeping moves from shapes to explicit state. DeepSeek's dense/MoE interleave (`first_k_dense_replace`, `moe_layer_freq`) is what site 2 exists for, and it stays correct: a dense layer after a MoE layer still gathers first and hands the layer a full input. Sequence-parallel MoE already requires `pipeline_parallel_size == 1`, so the `IntermediateTensors` PP path always carries full states and needs no change. ### Affected models Everything routed through `DeepseekV2Model` / `DeepseekV2DecoderLayer`: - `DeepseekForCausalLM` - `DeepseekV2ForCausalLM` - `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` - `GlmMoeDsaForCausalLM` - `MistralLarge3ForCausalLM` (subclasses `DeepseekV3ForCausalLM`) - `EagleMistralLarge3Model` (inherits `DeepseekV2Model.forward`) `input_is_sequence_parallel` is a trailing keyword argument defaulting to `False`, so the other callers of `DeepseekV2DecoderLayer` are unaffected: - `deepseek_mtp.py` and `longcat_flash_mtp.py` run a single `mtp_block` with `residual=None`, which the old code also evaluated as not sequence-parallel. Both already handle their own trailing all-gather. - `deepseek_eagle.py` keeps its own layer loop. It has no sequence-parallel handling at all — no all-gather anywhere, including before its final norm — so MoE SP is not functional on that draft path before or after this change. For the usual single-layer EAGLE draft the behaviour is identical either way. Left alone deliberately to keep this PR to the bug; happy to fold in a follow-up if maintainers prefer. Models that only borrow submodules from `deepseek_v2.py` (`AXK1`, `glm4_moe_lite`, `longcat_flash`, `deepseek_eagle3`) have their own model loops and are untouched. ## Test Plan New CPU unit test, `tests/model_executor/test_deepseek_v2_sequence_parallel.py`, drives `DeepseekV2Model.forward` with recording stub layers and a fake all-gather, so the layout state machine is checked without GPUs: ```bash pytest tests/model_executor/test_deepseek_v2_sequence_parallel.py ``` It asserts, over `num_tokens` in `{1, 2, 7, 8}` and five dense/MoE layer arrangements, that every layer receives a row count consistent with the layout flag it was handed, that the model returns the full sequence, and that the all-gathers happen exactly where they should. The single-token cases assert on all-gather counts rather than shapes, since shapes are ambiguous there — which is the bug. End-to-end, on 4 GPUs (default `--all2all-backend` selects `allgather_reducescatter`, which enables `use_sequence_parallel_moe`): ```bash vllm serve deepseek-ai/DeepSeek-V2-Lite \\ --tensor-parallel-size 2 --data-parallel-size 2 --enable-expert-parallel \\ --enforce-eager --trust-remote-code --max-model-len 2048 --max-num-seqs 4 ``` then a single greedy request (`temperature: 0`, `max_tokens: 64`), one at a time so every decode step is a single-token batch. `tests/model_executor/test_mistral_large_3_eagle.py` needed two lines: its `DummyDecoderLayer` now accepts the new keyword argument and declares `use_sequence_parallel_moe`, which the model loop previously only read behind a short-circuit. ## Test Result The new unit test fails on `main` and passes with this change. On `main` the single-token cases show the trailing all-gather being skipped entirely, which is the corruption path. ```text tests/model_executor/test_deepseek_v2_sequence_parallel.py .................. 24 passed tests/model_executor/test_mistral_large_3_eagle.py ......................... 4 passed ``` GPU verification in progress.", "url": "https://github.com/vllm-project/vllm/pull/50691", "timestamp": "2026-08-12T17:12:09Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "deepseek", "mistral" ], "author": "thegoldenflow", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:50729", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Mamba] Fix overlapping state copy race", "text": "[PR #30877](https://github.com/vllm-project/vllm/pull/30877) introduced generic Mamba state copies, and [PR #40172](https://github.com/vllm-project/vllm/pull/40172) added the fused GPU copy. A speculative-decode convolution-state shift can copy within the same physical block with overlapping source and destination ranges. Parallel memcpy-style loads/stores do not provide memmove ordering, which explains the intermittent [AMD CI failure](https://buildkite.com/vllm/amd-ci/builds/11556/list?sid=019fbc8d-6c5e-4172-83cb-f45a49946645&tab=output). This version is rebased over [PR #49436](https://github.com/vllm-project/vllm/pull/49436): - Temporal state copies retain #49436's 3D grid and 16-way tiled copy. - DS convolution copies use stable row-to-lane ownership while iterating tokens low to high. - SD convolution copies retain `_memcpy_u64_tiled(..., NUM_TILES=1)` for distinct physical blocks and exact self-copies, and use an ordered token copy only for a true same-physical-block left overlap. - Conv copies remain single-CTA (`tile_idx > 0` returns), so no grid-wide barrier is assumed or required. - The generic fallback is unchanged. MRv1 and MRv2 already share this fused copy kernel for the relevant hybrid/speculative/aligned path; routing every MRv1 configuration through the MRv2 lifecycle would break unsupported/custom state layouts such as ReplaySSM and would not cover the CPU fallback. - Existing overlap assertions derive expected values from untouched snapshots. The broad Python-vs-fused golden uses disjoint pages so the generic memcpy helper is not treated as a memmove oracle. ## Validation - `HIP_VISIBLE_DEVICES=0 .venv/bin/python -m pytest -q -s tests/v1/worker/test_mamba_utils.py`: **37 passed** - `uvx pre-commit run --files vllm/v1/worker/mamba_utils.py tests/v1/worker/test_mamba_utils.py`: **passed** - The original failing revision reproduced locally on MI355; the rebased implementation passed the same snapshot checks. - Qwen3.5-0.8B MTP checks across 8×MI355, MRv1/MRv2, SD/DS, and batch 1/8 produced matching output hashes. A counterbalanced device-kernel benchmark on MI355 covered both runners, SD/DS, same-page and distinct-page copies, and batch 1/8. For a 24-layer Qwen3.5-shaped state copy: | Path | Batch | Result vs current main | |---|---:|---:| | SD same-page, MRv1 | 1 / 8 | +14.7 µs / +1.1% | | SD same-page, MRv2 | 1 / 8 | +18.0 µs / +1.2% | | SD distinct-page, both runners | 1 / 8 | -2.5% / +1.7–1.8% | | DS, both runners and mappings | 1 / 8 | 95.5–98.5% lower latency | The SD same-page cost is the overlap-safe ordering; the common distinct-page path retains the tiled memcpy. MRv1 and MRv2 copy-kernel latency is effectively equivalent because they already execute the same fused mechanism. This PR was developed with assistance from OpenAI Codex.", "url": "https://github.com/vllm-project/vllm/pull/50729", "createdAt": "2026-08-02T05:57:32Z", "updatedAt": "2026-08-13T17:22:35Z", "timestamp": "2026-08-13T17:22:35Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "bug", "ready" ], "author": "AndreasKaratzas", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50732", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[PD] Emit inactive KV blocks for decode affinity", "text": "Publish GPU block inactivity when the last request reference is released so external schedulers can maintain accurate decode load signals. ## Purpose Add a new KV cache event, `BlockInactive`, so external PD schedulers can track when a GPU prefix-cached block is no longer actively referenced. Today, KV event consumers can observe `BlockStored` and `BlockRemoved`, but cannot reliably detect that the last request released a block while it may still remain in the prefix cache. Without a refcount-zero signal, decode-affinity load accounting (`active_blocks`) tends to only increase or needs fragile heuristics. This PR: - Adds `BlockInactive` to `KVEventBatch`, emitted from `BlockPool.free_blocks()` when `ref_cnt` drops to 0 for a hashed GPU block (`medium=MEDIUM_GPU`). - Publishes it through the existing KV event path (`take_events()` / ZMQ). No new scheduler-loop wiring. - Gates emission for Prefill-only instances: `kv_role == \"kv_producer\"` still emits `BlockStored` / `BlockRemoved`, but does **not** emit `BlockInactive`, so Prefill frees do not pollute decode load signals. - Decode (`kv_consumer`), union (`kv_both`), and non-PD setups continue to emit Inactive via `KVTransferConfig.should_emit_block_inactive` (`kv_role != \"kv_producer\"`). Related issue: #50730 ## Test Plan pytest tests/v1/core/test_prefix_caching.py \\ -k \"free_blocks_emits_block_inactive or free_blocks_skips_block_inactive or kv_transfer_should_emit_block_inactive\" \\ -q Coverage: free_blocks emits BlockInactive when enable_block_inactive_events=True. Prefill-style gate (enable_block_inactive_events=False) still emits BlockStored, but not BlockInactive. KVTransferConfig.should_emit_block_inactive is False for kv_producer, True for kv_consumer / kv_both / default. ## Test Result Unit tests above pass locally. Existing BlockStored / BlockRemoved behavior is unchanged. Old KV event consumers can ignore the unknown BlockInactive type. Validated end-to-end in a 1P1D (one Prefill + one Decode) setup. Additional tests are still in progress. Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. ", "url": "https://github.com/vllm-project/vllm/pull/50732", "createdAt": "2026-08-02T08:21:18Z", "updatedAt": "2026-08-13T08:26:55Z", "timestamp": "2026-08-13T08:26:55Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "needs-rebase", "kv-connector" ], "author": "tianmengx12-max", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50779", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Extensible (growable) KV cache", "text": "> **Stacked on https://github.com/vllm-project/vllm/pull/51718. Only the commits above that PR's head belong to this one. > This is a **draft** — it cannot merge until https://github.com/vllm-project/vllm/pull/51718 lands, after which the base will be retargeted to `main`. ## Summary Opt-in growable KV cache (`--enable-extensible-kv-cache`) built on the standardized KV cache layout from https://github.com/vllm-project/vllm/pull/51718. This is the productization of the demonstration https://github.com/vllm-project/vllm/pull/47363 from @zhuohan123. vLLM sizes the KV cache from a profiling run, then allocates it up front. Any memory the engine consumes *after* profiling — CUDA graph pools, the spec-decode logits all-gather, workspace growth — has to be guessed at via headroom margins, and a wrong guess is either an OOM or wasted HBM. This change reserves virtual address space for the KV cache capacity but commits physical pages incrementally (CUDA `cuMem*` / ROCm `hipMem*`): 1. Allocate the VA reservation, commit one block per layout segment. 2. Run warmup and CUDA graph capture. 3. Measure *actual* post-warmup free memory and re-size. 4. `extend_kv_cache()` maps the remaining pages under the same base pointer. Because each block keeps a fixed offset within its layout segment and the base pointer never moves, captured graphs and layer views stay valid across the growth — no re-view, no re-capture. --- ### What https://github.com/vllm-project/vllm/pull/51718 buys this The layout standardization removes most of what made the earlier version of this work invasive. Buffer segmentation is now a pure function of `KVCacheLayout.stride_order` (new `num_outer_segments()` helper) instead of per-backend shape/stride/block-dim probing; connector registration views are a plain `narrow(0)` of the logical `[B, H, N, C]` view; and the packed / `block_stride` special cases disappear entirely. The integration is ~30% smaller than the equivalent against `main`. ### Views are authoritative, not `storage.nbytes()` The one contract change worth reviewer attention. Under an extensible cache the untyped storage spans the **reserved** capacity, of which only each view's per-segment block prefix is physically backed. Any code deriving block geometry or registration extents from `untyped_storage().nbytes()` will compute a size that includes unbacked pages. Updated accordingly, with the contract documented on `KVConnectorBase_V1.register_kv_caches`: - `NixlBaseConnectorWorker.register_kv_caches` — collects per-region committed spans and registers those; falls back to the single per-allocation registration when the storage is fully backed. Block-major packing is detected from view strides rather than `nbytes()`. - `get_kv_cache_block_regions` (`v1/worker/utils.py`) — same, which also covers the CPU-offload connector. KV-transfer init is deferred until after `extend_kv_cache`, so connectors only ever register committed memory. On the NIXL path, extend also defragments (release + single-chunk recommit per segment): UCX cannot transfer a VMM region spanning multiple `cuMemCreate` handles. ## Relationship to existing work - **#47363** (@zhuohan123, \"Demo implementation of extensible kv cache memory) — this is the productionization of that demo, not a competing implementation. It adds the V1 + V2 runner integration, the measured post-warmup sizing pass, VMM probe/fallback, sleep-mode support, KV connector support, and a ROCm backend. Happy to fold this into #47363 instead if that's preferred. - **#44458** — strictly a dependency; no overlap in changed behavior. - Searched open PRs for extensible/growable/VMM KV cache work; nothing else overlaps. Contributors: @zhuohan123 (original demo), @zjy0516 (packed-storage bounding, sleep gate). ## Testing Unless noted, run against this branch at its current head. **Unit** — `pytest tests/v1/worker/test_extensible_kv_cache.py tests/utils_/test_extensible_tensor.py tests/v1/engine/test_engine_args.py -v` → 29/29 pass (GB200, aarch64). **e2e, 1×GB200, CUDA graphs on** — outputs byte-identical to a non-extensible baseline for: standard generation, sleep(level=1)/wake, and `--kv-cache-memory-bytes`. **e2e, sizing under pressure** — Qwen3.5-4B TP4, MTP `num_speculative_tokens=100`, chunk 256, `gpu-memory-utilization=0.92` (a config that OOMs without this change): passes in both eager and cudagraph modes. Sizing 35,277→34,721 tokens (eager), 35,243→34,592 (cudagraph). Functional check 64/43 answers correct. **e2e, NIXL P/D disaggregation** — Qwen3-0.6B, HND, block 128, gsm8k via P/D proxy: | config | gsm8k | expected | |---|---|---| | 1P1D eager | 0.422 | ~0.41 | | 1P1D cudagraph | 0.418 | ~0.41 | | P(TP1)→D(TP2) | 0.415 | ~0.41 | | P(TP2)→D(TP2) | 0.415 | ~0.41 | **Model evaluation** — the feature is opt-in and does not change model outputs; the byte-identical-output comparison above and the gsm8k table are the evidence for that. ### Known gaps - The V2 encoder-cache profiling reservation (last commit) is validated on the equivalent branch against `main` (Qwen2-VL-2B: 16,384-token budget reserved, 238,148 blocks committed) but **has not been re-run on this branch** — the cluster battery above has no multimodal configuration. Will re-run before marking ready. - No unit coverage for `commit(defragment=True)` or the connector sleep gate; both are exercised only by the e2e NIXL runs. Adding these before ready. - Multi-node NIXL (NIC RDMA path with the GDR + POSIX-FD allocation flags) is unvalidated; the repro harness is localhost-only. - The ROCm `hipMem*` backend is written but untested on AMD hardware. - Encoder-decoder mixed layouts raise rather than falling back. ## AI assistance AI assistance (Claude Code) was used for this change: for the port onto the #44458 layout model, the connector geometry audit, and drafting. I have reviewed every changed line, and ran the tests and cluster validation above myself.", "url": "https://github.com/vllm-project/vllm/pull/50779", "createdAt": "2026-08-02T22:36:23Z", "updatedAt": "2026-08-13T06:26:03Z", "timestamp": "2026-08-13T06:26:03Z", "metrics": { "reactions": 5, "comments": 17 }, "labels": [ "documentation", "performance", "rocm", "structured-output", "frontend", "intel-gpu", "speculative-decoding", "ready", "needs-rebase", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "kimi", "k3" ], "author": "njhill", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50786", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "docs(serving): note OpenAI client base_url for multi-model gateways", "text": "## Summary The OpenAI-compatible server docs already show the OpenAI Python client via `base_url`. This PR adds a one-line tip that the same client pattern works with OpenAI-compatible multi-model gateways when not running vLLM locally, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior changes. ## Test plan - [ ] Serving docs render - [ ] Local vLLM client example unchanged - [ ] Tip is clearly optional Co-Authored-By: Claude Fable 5 ", "url": "https://github.com/vllm-project/vllm/pull/50786", "createdAt": "2026-08-03T00:42:47Z", "updatedAt": "2026-08-12T18:09:00Z", "timestamp": "2026-08-12T18:09:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "seven7763", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:50796", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Build] Restore DeepGEMM SM120 support while preserving SITU", "text": "## Purpose Addresses the current-main regression reported in #47436 for block-scaled FP8 checkpoints on SM120. The Kimi K3 integration changed the vendored DeepGEMM revision from the `nv-dev` commit used by vLLM v0.26.0 to `f5a76426`, which provides SITU support but does not contain the SM120 scale-factor layout dispatch. As a result, current nightly builds select `DeepGemmFp8BlockScaledMMKernel` and fail while post-processing weights: ```text RuntimeError: Assertion error (.../deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation ``` This updates both DeepGEMM pins to `2fd67329`. Its direct parent is the previously validated `5f33a180` `nv_dev+situ` merge commit, which combines the SM120 support from `nv-dev` with the SITU activation support required by Kimi K3. The additional commit carries the CUDA 12.9 `cuda_fp8.h` include fix from #51003. ### Duplicate-work check No open PR addressed this regression when this PR was opened. Draft #51382 was opened later as a materially different alternative: it minimizes the DeepGEMM dependency delta by cherry-picking SITU onto the earlier SM120 revision, while this PR uses the later `nv_dev+situ` merge and its CUDA 12.9 follow-up. The other related PRs are also not duplicates: - #47988 and #48588 improve CUTLASS/Triton fallback behavior; they do not restore the DeepGEMM SM120 path. - #41834 is a broader DeepSeek V4 SM12x enablement branch and does not currently modify either DeepGEMM pin file. - #47304 previously moved the pin to `nv-dev` for SM120 support. This change restores that support after the later SITU pin update, while preserving SITU. ## Test Plan Build a CUDA 13 production image for SM120 from vLLM main with the updated pin, then serve the real `DeepSeek-V4-Flash-0731` checkpoint on six RTX 5090 GPUs: ```text model: DeepSeek-V4-Flash-0731 vLLM: 0.26.1rc1.dev251+g0033211c0.d20260803 GPU: 6x NVIDIA RTX 5090 (SM120) parallelism: TP=2, PP=3 checkpoint size: 155.43 GiB, 48 safetensors shards ``` ```bash vllm serve /model \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --max-model-len 524288 \\ --tensor-parallel-size 2 \\ --pipeline-parallel-size 3 \\ --gpu-memory-utilization 0.97 \\ --max-num-seqs 1 \\ --enable-prefix-caching \\ --disable-log-stats \\ --enable-auto-tool-choice \\ --tool-call-parser deepseek_v4 ``` Additional checks: ```bash bash -n tools/install_deepgemm.sh git diff --check ``` After startup, send repeated OpenAI-compatible chat-completion requests, followed by a request with a `get_weather` function schema. Scan the serving log for `Unknown SF transformation`, `illegal memory access`, engine failures, and CUDA errors. ## Test Result The original pre-rebase candidate (`5f33a180`) built successfully with CUDA architecture `12.0f`; the build log checked out that merge revision: ```text 5f33a18 Merge branch 'situ-activation' into nv_dev+situ DeepGEMM CUDA architectures: 12.0f ``` The model loaded all real checkpoint shards and selected the intended kernels: ```text Loading safetensors checkpoint shards: 100% Completed | 48/48 DeepGEMM E8M0 enabled on current platform. Using 'DEEPGEMM_MXFP4' Mxfp4 MoE backend. Using DeepGemmFP4Experts Application startup complete. ``` Runtime results after one-time per-shape JIT warmup: | Check | Result | | --- | --- | | Weight loading | 48/48 shards loaded; no SF-layout assertion | | Chat completion | Correct Chinese responses returned | | Warmed decode | 84.1-100.6 output tok/s across repeated 83/90-token responses | | Tool calling | `finish_reason=tool_calls`, `get_weather({\"city\":\"北京\"})` | | Error scan | No `Unknown SF transformation`, illegal memory access, engine failure, or CUDA error | The first request for a new shape was slower because current main performs one-time TileLang/DeepGEMM JIT compilation; repeated requests used the cached kernels. The performance figures above are warmed single-request measurements, not aggregate throughput. After rebasing, static validation confirmed that `2fd67329` is a direct child of the hardware-tested `5f33a180` and differs only by the CUDA FP8 header include from #51003. `bash -n tools/install_deepgemm.sh`, pin-consistency validation, and `git diff --check` pass. An exact-pin SM120 rebuild and long-context run have not yet been completed. No model-quality change is expected from this pin update. As a runtime sanity check, generated Chinese responses were coherent, and the tool-call name and JSON arguments matched the supplied schema. --- AI assistance was used for root-cause analysis, patch preparation, validation scripting, and PR drafting with GPT-5.6-sol (low). I reviewed the complete two-file diff and validated the patched image end-to-end on the hardware described above. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, including the related issue. - [x] The test plan and commands are provided. - [x] Before/after behavior and end-to-end results are provided. - [x] Duplicate work was checked.
", "url": "https://github.com/vllm-project/vllm/pull/50796", "createdAt": "2026-08-03T03:11:01Z", "updatedAt": "2026-08-12T21:47:33Z", "timestamp": "2026-08-12T21:47:33Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "ZacharyZcR", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50817", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Enable AITER MXFP4 MoE on gfx942 and optimize tile configurations for MI325X Target Kimi K3 running on MI325X", "text": "## Summary Makes Kimi-K3 run on AMD MI325X (gfx942 / CDNA3). The model loaded but aborted in MLA decode; once past that it fell back to a dequantise-to-BF16 MoE emulation path. Both are fixed, and a third change addresses AITER MoE tile shapes being sized for CDNA4 on all architectures. Measured on 8x MI325X with the real 2.75T-parameter MXFP4 checkpoint: | | before | after | | --- | --- | --- | | Kimi-K3 decode (TP=8, CUDA graphs, warm) | 4.27 tok/s | **99.67 tok/s** (23x) | | Prefill throughput (36k prompt) | 3,522 tok/s | **6,537 tok/s** (1.9x) | | MLA decode at 12 heads/rank | `AssertionError` | works, bit-exact | ## Motivation Three independent blockers, in the order they surfaced. **1. MLA decode aborted on any model with <16 heads per rank.** ``` AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 ``` AITER's small-head MLA decode kernel asserts CDNA4, but the backend selected it on head count alone. Kimi-K3 has 96 attention heads, so TP=8 gives 12 per rank — and TP=8 is the *only* viable parallelism: 1.5 TB of weights against 2 TB of HBM rules out TP=6 (260 GB/GPU, and 7168 % 6 != 0) and TP=4 (390 GB/GPU). The existing pad-to-16 fallback only handled head counts dividing 16; `repeat_interleave(16 // num_heads)` is a silent no-op for 12, leaving q at 12 heads while the output was allocated for 16. **2. MXFP4 MoE fell through to BF16 emulation on gfx942.** AITER's CK MXFP4 kernel needs CDNA4 native FP4 MFMA, so gfx942 landed on `OCP_MXQuantizationEmulationTritonExperts`, which dequantises the *entire* weight tensor every forward: ```python w1_dequant = self._dequantize_weights(w1, ...) # all 896 experts w2_dequant = self._dequantize_weights(w2, ...) # 16 are routed to ``` That is 56x the required work. For K3: 896 experts x 33.0M params / 8 GPUs x 2 bytes ~= 7.4 GB per layer, x 93 layers ~= 688 GB materialised per token, written then read back ~= 1.4 TB of HBM traffic, / ~6 TB/s ~= 229 ms/token. Predicted ~4.4 tok/s; measured 4.27. That was the entire performance story. AITER *does* ship a Triton `moe_gemm_a16w4` kernel which dequantises in-register during the GEMM. It runs correctly on gfx942 — the arch gate simply never admitted it. The ROCm base image already builds AITER with `AITER_ROCM_ARCH=\"gfx942;gfx950\"`, so these kernels were compiled and present all along. **3. AITER MoE tile selection is architecture-blind.** `get_kernel_config` hardcodes `block_k=256`, `block_n=512`, `num_stages=1` for every AMD part. Those suit CDNA4's 160 KiB LDS; CDNA3 has 64 KiB. `arch_info._LDS_CAP_BYTES` already records the per-arch capacity and is consulted by `pa_decode_sparse`, `gemm_config_utils` and `sparse_attention_dsv4` — the MoE GEMM is the one kernel that skips it. ## Changes ### MLA small-head decode (`vllm/v1/attention/backends/mla/rocm_aiter_mla.py`) - Add `AiterMLAHelper.gluon_decode_available()` (gfx950) and require it in `use_gluon_decode()` and in the multi-token verify branch, so non-CDNA4 falls through to the padded path instead of asserting. - `get_mla_padded_q` / `get_mla_unpadded_o` keep whole-repeat expansion where the head count divides 16, and zero-pad the head dimension otherwise. Zero-padding is exact, not approximate: the decode kernel treats query heads independently, so filler heads cannot perturb retained ones. The new test pins that property — running 16 real heads, then the same tensor with heads >= N zeroed, gives **bit-identical** outputs for heads 0..N-1. ### MXFP4 MoE on gfx942 - `mxfp4_utils.py`: new `use_aiter_mxfp4_triton_moe()` — a single predicate for \"AITER MXFP4 runs the Triton kernel\", replacing scattered `on_gfx1250()` checks that independently decided weight-preparation format. It also requires `has_triton_kernels()`, so a build without that package falls through to previous behaviour instead of asserting at weight load. - `aiter_mxfp4_w4a8_moe.py`: admit gfx942 in `AiterW4A16ExpertsMonolithic._supports_current_device`. - `oracle/mxfp4.py`: split the `AITER_MXFP4_BF16` weight-prep branch — the Triton path swizzles and returns `PrecisionConfig`s, the CK path keeps the de-interleave plus aiter shuffle. - `quantization/mxfp4.py`: route parameter-vs-wrapped-tensor assignment, the `is_shuffled` marking, and the quant-config scale source through the same predicate. The **de-interleave of w13 gate/up pairs is CK-only.** A first attempt applied it to the Triton path and produced uncorrelated output (cosine 0.034 vs the TRITON backend). The Triton kernel consumes the checkpoint's interleaved layout like every other Triton backend. ### SiTU activation support K3's MoE uses SiTU — `beta*tanh(gate/beta)*sigmoid(gate)*up` — which cannot be expressed through the a16w4 kernel's `alpha`/`limit`/`add_residual` knobs. `aiter_triton_kernel_w4a16_moe_forward` now runs gemm1 unfused for SiTU and applies the existing compiled `torch.ops._C.situ_and_mul` between the two GEMMs. Only activations round-trip through HBM (a few hundred KB); weights never do. K3 already stores w13 as `[gate_all, up_all]`, exactly what `situ_and_mul` splits on, so no reshuffling is needed. `_setup_kernel_k3_situ` emits the Triton layout on gfx942 rather than the CK shuffle, and drops the registered Parameters before rebinding since the swizzled tensors are `triton_kernels` wrappers. ### Tooling and CI - `benchmarks/kernels/benchmark_moe_a16w4_tiles.py` — sweeps MoE tile shapes and reports the best against stock. Overrides only tiles at/above `--min-block-m` (default 64), leaving decode alone. - `tools/vllm-rocm/patch_aiter_cdna3_moe_tiles.py` — LDS-aware override for AITER's tile selector on gfx942, with `--revert`. - `tools/vllm-rocm/README_kimi_k3_mi325x.md` — serving recipe and two operational traps (below). - `.buildkite/test-amd.yaml` — add `vllm/models/kimi_k3/amd/ops/third_party/kda/` to the KDA job's source dependencies; edits to the AMD kernel copies did not trigger the AMD job. - `kimi_k3/amd/.../kda/__init__.py` — correct a provenance note claiming byte-identity with a NVIDIA copy that has since diverged. ## Results All on 8x MI325X (gfx942), ROCm 7.2.3 container, AITER v0.1.19, real `moonshotai/Kimi-K3` weights, TP=8, CUDA graphs, `max_num_seqs=512`, `gpu_memory_utilization=0.97`. **Decode** | backend | tok/s | | --- | --- | | EMULATION (before) | 4.27 | | Triton a16w4 + SiTU, cold | 65.66 | | Triton a16w4 + SiTU, warm | **99.67** | Cold carries ~18 Triton JIT compilations inside the timed window. **MoE GEMM tile shapes** (4096 tokens; stock = `block_n=512 block_k=256 warps=8 stages=1`) | GEMM | stock | best gfx942 | | | --- | --- | --- | --- | | gemm1 K=3584 N=768 | 8.208 ms | 1.303 ms (`bn=128 bk=128 s=1 w=4`) | 6.30x | | gemm2 K=384 N=7168 | 8.019 ms | 1.701 ms (`bn=512 bk=128 s=2 w=8`) | 4.72x | `block_k` is the dominant variable — halving it shrinks both LDS tiles at once. Triton names the constraint when it does not fit: `OutOfResources: shared memory, Required: 100352, Hardware limit: 65536`. **End-to-end prefill** | prompt tokens | before | after | | | --- | --- | --- | --- | | 3,697 | 2,268 tok/s | 4,459 | 1.97x | | 14,497 | 3,156 tok/s | 5,910 | 1.87x | | 36,097 | 3,522 tok/s | 6,537 | 1.86x | | 72,098 | 3,427 tok/s | 6,585 | 1.92x | Smaller than the kernel-level 5-6x because prefill also pays for MLA, 69 KDA layers, norms and TP collectives. ## Testing ```bash pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_small_head_pad.py # 11 passed VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \\ pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py -k test_rocm_mxfp4_moe_oracle # 3 passed, 1 skipped (AITER_MXFP4_FP8 still requires gfx950) VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \\ pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py \\ tests/kernels/moe/test_mxfp8_aiter_backend_selection.py # 22 passed, 117 skipped, 0 failed pytest -v -s tests/models/kimi_k3/test_amd_attn_res.py # 4 passed pytest -v -s tests/models/kimi_k3/test_kda.py tests/models/kimi_k3/test_kda_metadata.py # 45 passed, 6 skipped ``` Additional verification beyond the suites: - Isolated a16w4 kernel vs dequantised reference GEMM: max rel err **5.7e-4** - Full MoE vs the trusted TRITON backend, identical weights: cosine **0.999906** - MLA head padding: **bit-identical** outputs for retained heads ## Model evaluation Greedy decode (`temperature=0`) through the OpenAI-compatible server on real weights returns identical text to the EMULATION baseline, including `17 * 23 = 17 * 20 + 17 * 3 = 340 + 51 = 391`. Post-tile-patch spot checks: 4/4 correct, including `4839 * 271 = 1311369`. I did **not** run a formal lm-eval / gsm8k harness. Given the change is kernel-path only and matches the reference backend to cosine 0.999906, I consider that proportionate, but I will run one if a reviewer wants it. ## Not included / known limitations - **The AITER tile fix is a runtime patch**, not a vLLM change. It modifies a third-party package and does not survive an AITER upgrade. The durable fix is upstream in ROCm/aiter, deriving `block_k` from `_LDS_CAP_BYTES`. Its constants are tuned to K3's two GEMM shapes; the `n <= 1024` split is a coarse proxy that lands on both measured optima but is not a general autotuner. - **Speculative decoding on gfx942 with <16 heads is untested.** Gating the Gluon multi-token branch means that case now falls through to the padded path. It is dormant for K3 (`num_nextn_predict_layers=0`), but anyone enabling MTP/EAGLE3 would be on an unvalidated causality path. - **`AITER_SITUV2_A8W4` is ignored by the new Triton path.** The CK branch reads it to select a gate/up-interleaved layout; the Triton path always assumes separated. Default-off, but a silent trap if set. - **No tuned MoE configs for MI325X** at K3's shapes (`E=896,N=384,device_name=AMD_Instinct_MI325X.json` is absent). - `AITER_MXFP4_FP8` (W4A8) remains gfx950-only — it needs the CDNA4 swizzle. ## Operational notes for anyone serving Kimi-K3 on MI325X Two traps that look like model corruption and are not: - **`max_num_seqs` must stay below the Mamba/KDA state block count.** K3 has 69 KDA layers and needs one state block per concurrent decode sequence, so the default 1024 fails CUDA graph capture at 854 available blocks. - **The `kimi_k3` reasoning parser strips the think channel on its closing marker.** If `max_tokens` truncates before that marker, raw chain-of-thought is returned as `content` instead of an answer. Combined with the template's `thinking_effort=max` default this reads as garbage output. Measured tool-call emission, 6 trials per cell: | schema | default (max) | low | high | thinking off | | --- | --- | --- | --- | --- | | 1-param | 6/6 | 6/6 | 6/6 | 0/6 | | 2-param | 3/6 | 6/6 | 6/6 | 3/6 | | 3-param | 6/6 | 6/6 | 6/6 | 0/6 | `thinking_effort: low` is 18/18 and ~3.5x cheaper in tokens than the default. `thinking: false` must not be used with tools.", "url": "https://github.com/vllm-project/vllm/pull/50817", "createdAt": "2026-08-03T05:44:20Z", "updatedAt": "2026-08-13T06:42:53Z", "timestamp": "2026-08-13T06:42:53Z", "metrics": { "reactions": 7, "comments": 11 }, "labels": [ "performance", "rocm", "needs-rebase", "ci/build", "quantization", "kimi", "k3" ], "author": "seanfilimon", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50857", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Release] Add XPU wheel release build", "text": "## Summary This PR adds XPU wheel support to the release Buildkite pipeline. Changes included: - Add a new Buildkite release job to build and upload the x86_64 XPU wheel. - Add an `export_vllm_wheel_release` target in `docker/Dockerfile.xpu` for producing the XPU wheel artifact. - Update nightly wheel index parsing so `+xpu` local-version suffixes are recognized as an XPU wheel variant. ## Details The new release job builds the XPU image target from `docker/Dockerfile.xpu`, copies `dist/` into `artifacts/`, then reuses the existing nightly wheel upload flow: - `.buildkite/scripts/upload-nightly-wheels.sh` - `.buildkite/scripts/annotate-build-artifact.sh` The XPU wheel is built with: ```bash python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38", "url": "https://github.com/vllm-project/vllm/pull/50857", "createdAt": "2026-08-03T11:18:29Z", "updatedAt": "2026-08-13T02:23:46Z", "timestamp": "2026-08-13T02:23:46Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "intel-gpu", "ci/build" ], "author": "wendyliu235", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50874", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][R3] Size monolithic routing replay buffer for DP", "text": "## Purpose Fix routing-replay capture for the FlashInfer monolithic MoE kernel under naive data parallelism, including padded sequence-parallel shards when expert parallelism is enabled. Two related assumptions fail in a TP2/DP2 deployment: 1. **Replay buffer capacity.** `max_num_tokens` is a per-rank scheduler limit, while the naive dispatch path all-gathers rank-local batches before invoking the monolithic kernel. This produced an 8192-row replay buffer for a 16384-row kernel input and failed during warmup. The buffer is now sized for the larger of the DP and EP dispatch groups, since under EP the gathered batch is the flattened EP group rather than the DP group. 2. **Gathered shard layout.** Under DP+EP, `topk_ids` can contain gathered sequence-parallel shards whose `dp_metadata.local_sizes` include CUDA-graph/SP padding. Those rows do not match the unpadded per-DP token counts used by the existing capture paths, so the batch-dimension assertion fired. When the gathered tensor matches `sum(local_sizes)`, we locate the current DP rank's contiguous shard group using that exact all-gatherv layout and copy only its real token count, trimming trailing padding. Existing naive-DP, modular-local, and modular-SP layouts are unchanged, and no routing or model-output semantics change. ## Duplicate check I searched open upstream PRs for `routing replay buffer`, `routed experts capture`, `routing replay all-gather`, and `local_sizes routing`; none fixes this allocation or the padded-shard capture: - #48698 scales a separate FlashInfer B12x MoE workspace and does not touch the routing replay output added by #44214. - #50940 also edits `routed_experts_capturer.py`, but only the shape configuration; it leaves the `capture()` batch-layout branches alone. - #50759 skips MoE padding inside the router implementations, upstream of the gathered-batch layout handled here. ## Validation - `git diff --check`: passed - vLLM pre-commit on the changed files, run in the repo's Python 3.12 venv: Ruff check/format, `mypy-3.10`, typos, SPDX, root-lazy-import, forbidden-import, torch-CUDA-API, config-validation, and sign-off hooks all passed. (Under a host Python 3.9 two of these hooks crash on 3.10+ syntax; that is an environment artifact, not a finding.) - Runtime validation of the exact TP2/DP2+EP default-backend configuration was run on the corresponding branch of my fork on GB200: mixed and decode CUDA-graph capture completed, both API servers started, and a `/v1/completions` request was served with routed-experts capture enabled. The original routing batch-dimension assertion did not fire. - Model evaluation: not applicable; routing decisions and model outputs are unchanged by this fix. AI assistance was used. As the human submitter I have reviewed every changed line and the recorded runtime validation.", "url": "https://github.com/vllm-project/vllm/pull/50874", "createdAt": "2026-08-03T13:20:04Z", "updatedAt": "2026-08-13T07:18:09Z", "timestamp": "2026-08-13T07:18:09Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "bug", "ready" ], "author": "TomerBN-Nvidia", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50876", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual silu quant fusion", "text": "Adding more quant types to SiLU+quant activation. Made with the assistance of Claude, all generated code has been human-reviewed. ### Testing: #### Unit test ``` pytest tests/kernels/test_fused_quant_activation.py ``` #### E2E test example (fp8, static) - manually expect if outputs are sane ``` from vllm import LLM, SamplingParams # Sample prompts. prompts = [ \"Hello, my name is\", \"The president of the United States is\", \"The capital of France is\", \"The future of AI is\", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95, min_tokens=200, max_tokens=200) llm = LLM(model='neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8') outputs = llm.generate(prompts, sampling_params) # Print the outputs. for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f\"Prompt: {prompt!r}, Generated text: {generated_text!r}\") ``` #### Eval example (fp8, static) ``` lm_eval --model vllm \\ --model_args pretrained=neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \\ --tasks hellaswag,arc_easy \\ --batch_size auto ``` ### Eval results: (gsm8k with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8`) main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7710|± |0.0116| | | |strict-match | 5|exact_match|↑ |0.6922|± |0.0127| ``` this PR + https://github.com/vllm-project/vllm/pull/46864 ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7582|± |0.0118| | | |strict-match | 5|exact_match|↑ |0.6816|± |0.0128| ```", "url": "https://github.com/vllm-project/vllm/pull/50876", "createdAt": "2026-08-03T13:50:48Z", "updatedAt": "2026-08-13T13:12:54Z", "timestamp": "2026-08-13T13:12:54Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "rust", "kimi", "k3" ], "author": "ElizaWszola", "state": "closed", "assignees": [ "sangstar" ] }, { "id": "github:vllm-project/vllm:pull_request:50883", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KV Offload] Scale UniformTypeKVCacheSpecs groups by DCP", "text": "## Purpose #49964 gated the DCP scaling of a group's token span on `isinstance(group.kv_cache_spec, AttentionSpec)` so that Mamba groups stay unscaled. `UniformTypeKVCacheSpecs` is a container rather than an `AttentionSpec` subclass, so DSA models — whose indexer and MLA layers merge into a single uniform group — lost the DCP factor too: - `build_offloading_config` reports `tokens_per_block = block_size` (64), - `resolve_kv_cache_block_sizes` still returns `block_size * dcp` (128) for a single group, so enabling CPU offload under DCP dies at startup: ``` AssertionError: tokens_per_block=64 not divisible by tokens_per_hash=128. Hybrid models (e.g. Mamba+Attention) need --enable-prefix-caching to align block sizes. ``` Reported by @Leoyzen on GLM-5.2, TP8/DCP2, `fp8_ds_mla`, MTP, `OffloadingConnector`: https://github.com/vllm-project/vllm/pull/46514#issuecomment-5166406963 The scaling is now keyed off a helper that unwraps the container, so Mamba groups stay unscaled and every attention group keeps its DCP span. Same call site is fixed in `resolve_kv_cache_block_sizes` for multi-group configs. Related: #50823 fixes the other half of the same container-not-unwrapped family (block table width). ## Test Three tests, one per call site plus the hybrid case, next to the Mamba tests from #49964: - `offloading_connector/test_config.py::test_dcp_scales_uniform_type_attention_group_blocks` - `offloading_connector/test_config.py::test_dcp_scales_uniform_type_group_alongside_mamba` - `core/test_kv_cache_utils.py::test_resolve_block_sizes_scales_uniform_type_group_by_dcp` Verified fail→pass in a container built from the current nightly (`0.26.1rc1.dev255+g5e35a6f4f`); each test fails on unpatched `main` with the reported assert (`tokens_per_block=16 not divisible by tokens_per_hash=32`) or a wrong span, and the two suites pass with the fix: ``` python3 -m pytest tests/v1/core/test_kv_cache_utils.py \\ tests/v1/kv_connector/unit/offloading_connector/test_config.py -q # 115 passed ``` Config-level change, no effect on model output; no eval run. AI assistance (Claude) was used for this work; all changes reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50883", "createdAt": "2026-08-03T15:41:58Z", "updatedAt": "2026-08-13T17:13:23Z", "timestamp": "2026-08-13T17:13:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "needs-rebase", "kv-connector" ], "author": "drakosha", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50894", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Scale KV page size for hidden states extraction with TP", "text": "## Summary - When `extract_hidden_states` is combined with tensor parallelism, the target model's KV page size shrinks (`num_kv_heads / TP`) but the hidden-state per-token cost stays at full `hidden_size`. This causes an assertion failure in `KVCacheSpecBase.__post_init__`: `assert self.page_size_padded >= real_page_size` - Scale up all target group block sizes when the hidden-state per-token cost exceeds the common page, before aligning hidden-state specs Fixes #51016 ## Test plan - [x] Unit test: `test_hidden_states_with_tp_scales_page_size` — verifies block sizes are scaled when TP-sharded KV pages are smaller than hidden-state cost - [x] Verified end-to-end: `poolside/Laguna-S-2.1` (117.6B MoE) + `extract_hidden_states` + TP=4 launches successfully and produces hidden states for training 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/50894", "createdAt": "2026-08-03T16:38:16Z", "updatedAt": "2026-08-13T15:36:00Z", "timestamp": "2026-08-13T15:36:00Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "bug", "needs-rebase" ], "author": "orestis-z", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50897", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Add lookahead-aware prefix cache hashing for EAGLE-style draft models", "text": "## Purpose Implement https://github.com/vllm-project/vllm/issues/50438 EAGLE-style draft KV at a block boundary depends on both the target prefix through that block and the finalized token immediately following it. The existing cache key proves only the block prefix, so vLLM conservatively drops the last matched unit and recomputes it. That cost is especially visible on hybrid full-attention + Mamba/GDN models whose aligned cache unit can contain hundreds of tokens. The new key includes the successor token and its input identity. A successful match therefore proves that the boundary input consumed by the EAGLE-style drafter is equal, allowing the final matched draft-attention block to be reused without the unconditional drop. ## Design - Maintain the existing target block hashes and an EAGLE-style successor-aware hash list on each request. - Emit an EAGLE-style hash only after the following finalized token is available, so an exact-boundary tail remains pending. - Use successor-aware hashes for EAGLE-style draft-attention cache groups and keep normal hashes for target-only Mamba state groups. - Cap cache publication at the number of successor-aware hashes that are ready. - Skip the legacy EAGLE-style last-block drop only when successor-aware hashing is active. - Preserve the existing conservative drop as the fallback for all unsupported configurations. The mode is enabled through `SpeculativeConfig.use_eagle()`, covering EAGLE, EAGLE3, MTP, DFlash, and DSpark. Successor-aware keys are used for colocated caching and connectors that explicitly advertise compatible lookup and store semantics (currently Mooncake Store). Other connectors keep the conservative last-block fallback. KV-cache events and resumable requests follow the same materialization-aware lifecycle. ## Correctness For a block covering `[s, e)`, the EAGLE-style key proves the regular prefix through `e - 1` plus the finalized token at `e`. An identical successor retains the final safe block, while a different successor naturally falls back to the preceding boundary. Mamba state groups do not contain draft-model layers, so they continue to use their normal target-prefix hashes. This also preserves fine-grained aligned-Mamba partial-state hits when the hash unit is smaller than the physical Mamba page. A successor-aware entry is published only after the worker confirms that the corresponding draft KV has been materialized. Truncation and preemption invalidate readiness past the surviving prefix, including for resumable requests. External connectors opt in through an explicit capability; unsupported connectors preserve the legacy fallback rather than consuming keys they cannot reproduce. Mooncake Store carries the successor hashes through lookup, save, and delayed-finish publication. ## Duplicate-work check I checked the RFC and open PRs before submission. No open implementation PR references #50438 or implements successor-aware prefix-cache hashing. - #44986 uses a phase-based rule to skip the pop during prefill; this PR instead makes reuse conditional on a content-based cache-key proof. - #45614 and #48375 repair or preserve the conservative Mamba backoff; this PR strengthens the key so that backoff is unnecessary only when the boundary token is proven equal. - #39171 addresses hybrid coordinator drop behavior but does not add the successor dependency to the cache key. Commands used for the duplicate check: ```bash gh issue view 50438 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"50438 in:body\" gh pr list --repo vllm-project/vllm --state open --search \"lookahead aware prefix cache EAGLE\" gh pr list --repo vllm-project/vllm --state open --search \"EAGLE prefix cache hash successor\" gh pr list --repo vllm-project/vllm --state open --search \"eagle drop block\" ``` ## Model evaluation Model: `Qwen/Qwen3.5-2B`, `method=mtp`, `num_speculative_tokens=1`, `mamba_cache_mode=align`, greedy sampling, eager execution, prefix caching enabled. The same 1,249-token prompt was submitted three times. The resolved attention and Mamba cache unit was 544 tokens. ```text first_cached_tokens=0 second_cached_tokens=544 third_cached_tokens=1088 outputs_equal=True num_drafts=3 ``` The third request reaches the maximum reusable prefix below the logits token: `floor((1249 - 1) / 544) * 544 = 1088`. All three generated token sequences were identical. ### OffloadingConnector evaluation Model: `Qwen/Qwen3.5-2B`, MTP, real `OffloadingConnector`, GSM8K 200 questions, 10-shot, two passes with APC reset between passes. ```text completion_tokens=49085 first_accuracy=0.515 second_accuracy=0.515 output_mismatches=0 store_bytes=82673664 load_bytes=35356672 num_drafts=25294 num_accepted_tokens=23401 acceptance_length=1.925 ``` ### NixlConnector PD evaluation Model: `meta-llama/Llama-3.2-1B-Instruct` with `nm-testing/Llama3_2_1B_speculator.eagle3`, official vLLM `NixlConnector` P/D topology and toy proxy, HND layout, APC enabled, GSM8K 1,319 questions, 5-shot, two passes. ```text completion_tokens=203004 first_accuracy=0.321456 second_accuracy=0.320697 output_mismatches=385 prefill_second_pass_cache_hit_tokens=776992 decode_second_pass_cache_hit_tokens=776992 decode_second_pass_external_cache_hit_tokens=62810 num_drafts=97783 num_accepted_tokens=105318 acceptance_length=2.077 ``` The two passes are not required to produce byte-identical text; both exceed the repository GSM8K lower bound of 0.23 and their accuracy differs by less than 0.001. ## Tests ```bash HF_HOME=/tmp/codex-vllm-hf-cache \\ HUGGINGFACE_HUB_CACHE=/tmp/codex-vllm-hf-cache/hub \\ .venv/bin/python -m pytest -q \\ tests/v1/core/test_kv_cache_utils.py \\ tests/v1/core/test_prefix_caching.py \\ tests/v1/core/test_scheduler.py \\ tests/v1/worker/test_gpu_model_runner.py \\ tests/v1/kv_connector/unit/test_mooncake_store_connector.py \\ tests/v1/kv_connector/unit/test_mooncake_store_coordinator.py \\ tests/v1/kv_connector/unit/test_mooncake_store_scheduler.py \\ tests/v1/kv_connector/unit/test_mooncake_store_worker.py # 533 passed .venv/bin/pre-commit run --files $(git diff --name-only --diff-filter=ACMR c8602c7906...HEAD) # all applicable hooks passed ``` ## AI assistance AI assistance was used to investigate the issue, implement the change, prepare tests, and draft this PR description. The submitting human reviewed the change and is responsible for understanding and defending it end to end.", "url": "https://github.com/vllm-project/vllm/pull/50897", "createdAt": "2026-08-03T17:09:58Z", "updatedAt": "2026-08-13T05:52:24Z", "timestamp": "2026-08-13T05:52:24Z", "metrics": { "reactions": 4, "comments": 8 }, "labels": [ "ready", "needs-rebase", "kv-connector", "mrv2" ], "author": "ZJY0516", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50903", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kernel] Fix divergent warp collectives in partial NeoX QK-Norm+RoPE", "text": "## Summary ## Author: Drew Jin, SJTU & Qwen Team, Solving Issue: https://github.com/vllm-project/vllm/issues/51049 This PR fixes silent Q/K corruption in the partial-NeoX path of `fused_qk_norm_rope`. When `rotary_dim < head_dim`, the multi-head-per-warp kernel produces incorrect Q/K values for every head after the first in each warp chunk. On H200, `token_heads_per_warp=8` can additionally produce NaNs. Verified against `upstream/main` at commit `005fa017566177e1a66d75eba7908d326c5acd60`. ## Root cause In the NeoX branch, only the rotary lanes entered the RoPE section, while the branch invoked warp collectives using the full-warp mask: ```cpp if (laneId < rotary_lanes) { __syncwarp(); rotated = __shfl_xor_sync(FINAL_MASK, value, pairOffset); __syncwarp(); } ``` For partial RoPE, `rotary_lanes < warpSize`. Therefore, `FINAL_MASK` names lanes that do not execute the collective, which is undefined behavior. `warpReduceSum` itself is not the issue. In the packed kernel, the invalid RoPE collective occurs before the next head’s RMSNorm reduction, causing the observed corruption in subsequent heads. The single-head kernel contains the same latent UB, but it has no following packed-head iteration that exposes the failure. ## Fix The NeoX shuffle is now executed by every lane named by `FINAL_MASK`. Only rotary lanes consume the shuffled value, access the cos/sin cache, and update the rotated elements. The fix is applied to both the single-head and multi-head-per-warp kernels. The implementation retains the platform-specific `FINAL_MASK` rather than constructing a narrowed 32-bit mask. This preserves the existing CUDA and ROCm wave64 mask semantics and avoids undefined expressions such as `1u << 32`. ## Tests Added regression coverage for the previously untested packing paths: - Force `token_heads_per_warp` to `1`, `2`, `4`, and `8`, and compare each variant directly against the existing unfused PyTorch reference. - Cover `head_dim ∈ {64, 128, 256}`, `rotary_ratio ∈ {1.0, 0.5, 0.25}`, NeoX and interleaved layouts, and fp16/bf16. - Verify that the V segment remains bitwise unchanged. - Verify that packed and auto-selected outputs are bitwise identical to the single-head implementation. - Exercise the SM90 auto-selected pack4 and pack8 paths with large-token inputs. Verified on NVIDIA H200 (SM90). `compute-sanitizer` racecheck and synccheck are clean. Because these tools do not report this divergent-mask UB, the regression is enforced by numerical tests. ## Impact The QK-Norm+RoPE Inductor fusion is currently opt-in and disabled by default, so this issue does not affect default vLLM inference. The observed packed-head corruption requires all of the following: 1. per-head Q/K RMSNorm routed through this fused operator; 2. NeoX-style RoPE; 3. partial RoPE (`rotary_dim < head_dim`); and 4. dispatch to the multi-head-per-warp kernel. On SM90 with `head_dim <= 128`, the dispatcher at the target commit uses: ```text U = num_tokens × (num_q_local + num_kv_local) U < 10240 → token_heads_per_warp = 1 10240 ≤ U < 40960 → token_heads_per_warp = 4 U ≥ 40960 → token_heads_per_warp = 8 ``` Large prefills can therefore enter the affected packed paths. Confirmed partial-NeoX configurations include GLM-4.5/4.6/4.7, Ring-1T, and the partial-RoPE attention layers in Step-3.5-Flash and Laguna-XS.2. Full RoPE, interleaved RoPE, and models not routed through this fused operator do not trigger this issue. ## Performance The fix also removes the slowdown observed in the affected packed path. Benchmark configuration: ```text GPU: NVIDIA H200 (SM90) PyTorch: 2.11.0+cu130 CUDA: 13.0 dtype: bf16 head_dim: 128 Q heads: 16 KV heads: 4 Timing: CUDA events Warmup iterations: 30 Measured iterations: 300 Statistic: latency per call ``` | Tokens | Pre-fix auto | Post-fix auto | Latency reduction | |---:|---:|---:|---:| | 2,048 | 22.5 µs | 15.9 µs | 29% | | 8,192 | 68.5 µs | 43.9 µs | 36% | | 16,384 | 125.6 µs | 80.1 µs | 36% | The single-head path is essentially unchanged. ## ROCm ROCm was not tested. The patch preserves the existing platform-specific `FINAL_MASK` and wave64-compatible code path; it does not introduce a CUDA-specific 32-bit active mask. ## AI assistance This PR was developed with AI assistance. I reviewed all changed lines, validated the behavior on H200, and understand the implementation and test coverage.", "url": "https://github.com/vllm-project/vllm/pull/50903", "createdAt": "2026-08-03T18:27:39Z", "updatedAt": "2026-08-13T10:45:29Z", "timestamp": "2026-08-13T10:45:29Z", "metrics": { "reactions": 2, "comments": 17 }, "labels": [ "bug", "ready" ], "author": "drewjin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "buffer size insuffient Dspark sd for FlashInfer MNNVL allreduce", "text": "fixes [#50877](https://github.com/vllm-project/vllm/issues/50877) ## Purpose FlashInferAllReduce.should_use_fi_ar gates on: self.max_num_tokens = max_workspace_size // (hidden_dim * element_size) max_workspace_size is the size of the whole MNNVL allocation (2 MB for TP8). But the MNNVL backend is Lamport-based and rotates through NUM_LAMPORT_BUFFERS=3 buffers, so only ~1/3 of the budget backs any single all-reduce: budget 2,097,152 bytes → gate says 146 tokens per-buffer 698,368 bytes → real capacity is 48 tokens Any tensor between 49–146 tokens passes the gate and then aborts inside the kernel. The drafter produces B × (1 + num_speculative_tokens) tokens per step — e.g. 14 requests × 4 = 56 tokens — landing exactly in this window. Without a drafter, decode batches produce B tokens and stay below 48. The same defect exists in fused_allreduce_gemma_rms_norm._can_use_flashinfer, which its docstring notes \"mirrors FlashInferAllReduce\". ## Fix After the existing budget bound (kept as a cheap pre-filter), call the workspace's own is_buffer_size_sufficient(tp_size, num_tokens, hidden_dim, dtype) method rather than reimplementing the Lamport arithmetic. This is on the abstract base class (flashinfer-python==0.6.15.post1, the pinned version), correct for both mnnvl and trtllm, and version-proof. ## What this does NOT fix AllReduceFusionPass has the same too-permissive bound but self.max_token_num there also drives the workspace allocation request — fixing it requires separating those two concerns and verifying the one-shot/two-shot split, which needs GB200 hardware. Leaving that for a follow-up. ## Workaround (no code change needed) VLLM_ALLREDUCE_USE_FLASHINFER=0 Disables the eager FlashInferAllReduce path; the compiled fusion pass is unaffected. ## Test Plan I have added test file and you can see output below, every assertion in the capacity-accounting logic changed ran and succeeded. pytest tests/compile/passes/distributed/test_fusion_all_reduce.py -v , already existing test. On GB200 python -m vllm.entrypoints.openai.api_server \\ --model --speculative-config '{\"model\": \"\"}' --tensor-parallel-size 8 if reporter can test on GB200 that would be perfect. ## Test Result Cannot test as I do not have GB200, if @ilmarkov can test it, that would be best. ## AI assistance This fix was developed with Claude (claude-opus-4-6) assistance. I have reviewed every changed line and understand the root cause and the design. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/50932", "createdAt": "2026-08-03T23:46:19Z", "updatedAt": "2026-08-13T16:10:15Z", "timestamp": "2026-08-13T16:10:15Z", "metrics": { "reactions": 2, "comments": 21 }, "labels": [ "bug", "ready", "nvidia" ], "author": "khushali9", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50945", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[1/2][Model Runner V2] DBO support, eager mode", "text": "DBO for Model Runner V2 (RFC #50738) is two PRs: -> #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) ## Purpose Relate to #50738. This PR finishes stage P0, P1, P2 ## Test Plan Benchmark results are shown in the RFC smoke.sh ```bash #!/usr/bin/env bash # Baseline (no DBO) V2 eager server. Must stay config-identical to smoke_dbo.sh # apart from the DBO flags, otherwise the A/B is meaningless. MODEL_PATH=${MODEL_PATH:-/workspace/models/hub/models--deepseek-ai--Deepseek-V2-Lite/snapshots/604d5664dddd88a0433dbae533b7fe9472482de0/} export VLLM_USE_V2_MODEL_RUNNER=1 export NVSHMEM_REMOTE_TRANSPORT=none export NVSHMEM_DISABLE_CUDA_VMM=1 export TRITON_CACHE_DIR=/tmp/triton_cache_$USER export TRITON_HOME=/tmp/triton_home_$USER export TORCHINDUCTOR_CACHE_DIR=/tmp/inductor_cache_$USER export VLLM_CACHE_ROOT=/tmp/vllm_cache_$USER DP=${DP:-2} TP=${TP:-1} PORT=${PORT:-18305} GPUS=${GPUS:-0,1} MAX_NUM_SEQS=${MAX_NUM_SEQS:-256} MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-8192} GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.85} ALL2ALL=${ALL2ALL:-deepep_high_throughput} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p /tmp/{triton_cache,triton_home,inductor_cache,vllm_cache}_$USER # shellcheck disable=SC2086 CUDA_VISIBLE_DEVICES=$GPUS vllm serve \"$MODEL_PATH\" \\ --data-parallel-size \"$DP\" \\ --tensor-parallel-size \"$TP\" \\ --enable-expert-parallel \\ --enforce-eager \\ --max-num-seqs \"$MAX_NUM_SEQS\" \\ --max-num-batched-tokens \"$MAX_NUM_BATCHED_TOKENS\" \\ --gpu-memory-utilization \"$GPU_MEM_UTIL\" \\ --host 127.0.0.1 \\ --port \"$PORT\" \\ --all2all-backend=\"$ALL2ALL\" \\ --trust-remote-code $EXTRA_ARGS ```` smoke_dbo.sh ```bash #!/usr/bin/env bash # DBO-enabled V2 eager server. Config-identical to smoke.sh apart from the DBO # flags. MODEL_PATH=${MODEL_PATH:-/workspace/models/hub/models--deepseek-ai--Deepseek-V2-Lite/snapshots/604d5664dddd88a0433dbae533b7fe9472482de0/} export VLLM_USE_V2_MODEL_RUNNER=1 export NVSHMEM_REMOTE_TRANSPORT=none export NVSHMEM_DISABLE_CUDA_VMM=1 export TRITON_CACHE_DIR=/tmp/triton_cache_$USER export TRITON_HOME=/tmp/triton_home_$USER export TORCHINDUCTOR_CACHE_DIR=/tmp/inductor_cache_$USER export VLLM_CACHE_ROOT=/tmp/vllm_cache_$USER DP=${DP:-2} TP=${TP:-1} PORT=${PORT:-18306} GPUS=${GPUS:-2,3} MAX_NUM_SEQS=${MAX_NUM_SEQS:-256} MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-8192} GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.85} ALL2ALL=${ALL2ALL:-deepep_high_throughput} DBO_DECODE_THRESHOLD=${DBO_DECODE_THRESHOLD:-32} DBO_PREFILL_THRESHOLD=${DBO_PREFILL_THRESHOLD:-512} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p /tmp/{triton_cache,triton_home,inductor_cache,vllm_cache}_$USER # shellcheck disable=SC2086 CUDA_VISIBLE_DEVICES=$GPUS vllm serve \"$MODEL_PATH\" \\ --data-parallel-size \"$DP\" \\ --tensor-parallel-size \"$TP\" \\ --enable-expert-parallel \\ --enable-dbo \\ --dbo-decode-token-threshold \"$DBO_DECODE_THRESHOLD\" \\ --dbo-prefill-token-threshold \"$DBO_PREFILL_THRESHOLD\" \\ --enforce-eager \\ --max-num-seqs \"$MAX_NUM_SEQS\" \\ --max-num-batched-tokens \"$MAX_NUM_BATCHED_TOKENS\" \\ --gpu-memory-utilization \"$GPU_MEM_UTIL\" \\ --host 127.0.0.1 \\ --port \"$PORT\" \\ --all2all-backend=\"$ALL2ALL\" \\ --trust-remote-code $EXTRA_ARGS ``` bench.sh ``` #!/usr/bin/env bash # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project # # Drive load against a running AFD instance (e.g. the attention endpoint started # by decode_bench_server.sh). Thin wrapper around `vllm bench serve`; every knob # is overridable via an environment variable. # # Usage: # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh # HOST=127.0.0.1 PORT=18305 INPUT_LEN=8192 OUTPUT_LEN=256 MAX_CONCURRENCY=64 \\ # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh # # Variables (default): # MODEL_PATH model weights dir/name (required) # HOST server host (127.0.0.1) # PORT server port (18305) # DATASET_NAME vllm bench dataset (random) # INPUT_LEN random input length (ISL) (1024) # OUTPUT_LEN random output length (OSL) (128) # NUM_PROMPTS total prompts to send (1024) # REQUEST_RATE requests/sec (inf = as fast as can) (5) # MAX_CONCURRENCY max in-flight requests (32) # RESULT_DIR directory for saved results (/tmp/results) # RESULT_FILENAME result json filename (decode_bench.json) # EXTRA_ARGS extra args appended to the command (empty) set -euo pipefail MODEL_PATH=${MODEL_PATH:?set MODEL_PATH to the model weights dir/name} HOST=${HOST:-127.0.0.1} PORT=${PORT:-18305} DATASET_NAME=${DATASET_NAME:-random} INPUT_LEN=${INPUT_LEN:-1024} OUTPUT_LEN=${OUTPUT_LEN:-128} NUM_PROMPTS=${NUM_PROMPTS:-1024} REQUEST_RATE=${REQUEST_RATE:-inf} MAX_CONCURRENCY=${MAX_CONCURRENCY:-32} RESULT_DIR=${RESULT_DIR:-/tmp/results} RESULT_FILENAME=${RESULT_FILENAME:-decode_bench.json} SEED=${SEED:-0} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p \"$RESULT_DIR\" # shellcheck disable=SC2086 vllm bench serve \\ --host \"$HOST\" --port \"$PORT\" \\ --model \"$MODEL_PATH\" \\ --dataset-name \"$DATASET_NAME\" \\ --random-input-len \"$INPUT_LEN\" \\ --random-output-len \"$OUTPUT_LEN\" \\ --num-prompts \"$NUM_PROMPTS\" \\ --request-rate \"$REQUEST_RATE\" \\ --max-concurrency \"$MAX_CONCURRENCY\" \\ --result-dir \"$RESULT_DIR\" \\ --result-filename \"$RESULT_FILENAME\" \\ --seed \"$SEED\" \\ --ignore-eos \\ --percentile-metrics ttft,tpot,itl,e2el \\ --save-result \\ $EXTRA_ARGS ``` ab.sh ```bash #!/usr/bin/env bash # Rigorous baseline-vs-DBO A/B. This is what produced the headline numbers. # # Both servers must already be running (see README: \"Manual reproduction\"). # Does a warmup run on each, then alternates base/DBO across several seeds so # any drift hits both arms equally, then prints per-seed and mean throughput. # # Why the defaults matter: # - prefix caching MUST be off on the servers, or repeat runs reuse identical # prompts and DBO looks up to +91% faster than it is # - >=30 s runs; sub-10 s runs are dominated by ramp-up and are not stable # - multiple seeds; a single seed is not enough to call a few-percent delta # # Usage: ab.sh [seeds...] # e.g.: ab.sh 16384 16 32 192 1 2 3 set -uo pipefail source \"$(dirname \"${BASH_SOURCE[0]}\")/env.sh\" cd \"$VLLM_DIR\" ISL=$1; OSL=$2; CONC=$3; NP=$4; shift 4 SEEDS=(\"$@\"); [ ${#SEEDS[@]} -eq 0 ] && SEEDS=(1 2 3) mkdir -p \"$RESULT_DIR\" one() { # port num_prompts seed name -> prints total token throughput MODEL_PATH=\"$MODEL_PATH\" PORT=$1 INPUT_LEN=$ISL OUTPUT_LEN=$OSL \\ MAX_CONCURRENCY=$CONC NUM_PROMPTS=$2 SEED=$3 REQUEST_RATE=inf \\ RESULT_DIR=\"$RESULT_DIR\" RESULT_FILENAME=\"$4.json\" \\ bash bench.sh > \"$RESULT_DIR/$4.log\" 2>&1 .venv/bin/python -c \" import json,sys try: d=json.load(open('$RESULT_DIR/$4.json')); print(f\\\"{d['total_token_throughput']:.1f}\\\") except Exception: print('nan')\" } echo \"workload: ISL=$ISL OSL=$OSL conc=$CONC prompts=$NP seeds=${SEEDS[*]}\" echo \"warmup...\" one \"$BASE_PORT\" $((NP / 4)) 99 warm_base >/dev/null one \"$DBO_PORT\" $((NP / 4)) 99 warm_dbo >/dev/null printf \"\\n%-6s %12s %12s %9s\\n\" seed baseline DBO gain BSUM=0; DSUM=0; WINS=0; N=0 for s in \"${SEEDS[@]}\"; do b=$(one \"$BASE_PORT\" \"$NP\" \"$s\" \"ab_base_i${ISL}_s${s}\") d=$(one \"$DBO_PORT\" \"$NP\" \"$s\" \"ab_dbo_i${ISL}_s${s}\") g=$(.venv/bin/python -c \"print(f'{($d/$b-1)*100:+.2f}%')\" 2>/dev/null || echo n/a) printf \"%-6s %12s %12s %9s\\n\" \"$s\" \"$b\" \"$d\" \"$g\" BSUM=$(.venv/bin/python -c \"print($BSUM+$b)\"); DSUM=$(.venv/bin/python -c \"print($DSUM+$d)\") WINS=$(.venv/bin/python -c \"print($WINS + (1 if $d>$b else 0))\"); N=$((N+1)) done .venv/bin/python -c \" b=$BSUM/$N; d=$DSUM/$N print(f'\\nmean {b:12.1f} {d:12.1f} {(d/b-1)*100:+8.2f}% DBO wins $WINS/$N seeds')\" ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50945", "createdAt": "2026-08-04T02:08:31Z", "updatedAt": "2026-08-13T02:17:23Z", "timestamp": "2026-08-13T02:17:23Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "nvidia", "mrv2" ], "author": "specture724", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:50983", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve the audio duration-limit error through the PyAV fallback", "text": "## Purpose load_audio wraps every failure on its PyAV fallback path as \"Invalid or unsupported audio file.\" including the rejection raised by the VLLM_MAX_AUDIO_DECODE_DURATION_S guard itself. For containers soundfile cannot probe (like WebM), audio over the duration limit therefore reports a misleading invalid-file error, while soundfile-readable formats (mp3, ogg, flac) report the actionable duration message. This PR types the guard's rejection as AudioDurationExceededError and re-raises it ahead of the generic wrap, so only genuine decode failures are normalized. No behavior change for callers catching ValueError. ## Test Plan ``` pytest tests/multimodal/media/test_audio.py ``` New tests synthesize an opus WebM in memory to exercise the PyAV fallback while a garbage-bytes control verifies genuine decode failures keep the generic error. ## Test Result All 12 tests pass (Linux x86_64, CUDA torch). Without the fix, both new rejection tests fail: the duration error is masked as \"Invalid or unsupported audio file.\". With it, WebM over the limit reports the same actionable message as mp3.", "url": "https://github.com/vllm-project/vllm/pull/50983", "createdAt": "2026-08-04T06:33:26Z", "updatedAt": "2026-08-13T16:21:20Z", "timestamp": "2026-08-13T16:21:20Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "multi-modality" ], "author": "ieaves", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Gate Torch FP8 scaled-MM on architecture support", "text": "## Purpose Reject Torch FP8 scaled-MM kernels when the underlying PyTorch operator is unavailable. The existing implementation compares a CUDA-style numeric capability against the SM89 threshold. On ROCm, gfx1100 reports capability `110`, so it incorrectly passes this check even though RDNA3 has no native FP8 matrix support and `torch._scaled_mm` rejects the device at runtime. This change mirrors PyTorch’s current ROCm architecture gate for per-tensor, channel-wise, and row-wise Torch FP8 kernels while preserving existing CPU, CUDA, and XPU behavior. It does not change general FP8 support for Triton, upcast, or emulation paths. This does not duplicate #43615: that PR enables RDNA4 paths but does not fix the inherited per-tensor/channel-wise eligibility check on gfx1100. Once `torch.cuda.is_scaled_mm_supported()` is available in vLLM’s supported PyTorch baseline, the mirrored architecture gate can be replaced with that query. AI assistance was used; all changed lines were reviewed by the submitter. ## Test Plan ```bash ruff check vllm/model_executor/kernels/linear/scaled_mm/pytorch.py ruff format --check vllm/model_executor/kernels/linear/scaled_mm/pytorch.py git diff --check HIP_VISIBLE_DEVICES=0 python /tmp/validate_vllm_fp8_scaled_mm.py HIP_VISIBLE_DEVICES=1 python /tmp/validate_vllm_fp8_scaled_mm.py ``` The standalone script validates: - CPU, XPU, CUDA, and ROCm routing behavior. - gfx1100 rejects all three Torch FP8 scaled-MM variants. - gfx1201 accepts all three variants. - The underlying PyTorch operator rejects gfx1100. - Tensor-wise and row-wise execution succeeds on gfx1201 and matches an FP32 reference. ## Test Result - Ruff passed. - Formatting passed. - `git diff --check` passed. - Platform-routing matrix passed: - CPU/XPU supported. - CUDA FP8 capability preserved. - Unsupported ROCm rejected. - Supported ROCm accepted. gfx1100: ```text selected=[False, False, False], operator=rejected ``` gfx1201: ```text selected=[True, True, True], tensorwise=ok, rowwise=ok, relative_error=0.001601 ``` The local vLLM extensions were ABI-incompatible with the installed PyTorch, so validation covered vLLM’s Python selection logic and the underlying PyTorch operator directly rather than a full end-to-end model serving run. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] No documentation update is required for this kernel-selection fix.
", "url": "https://github.com/vllm-project/vllm/pull/51021", "createdAt": "2026-08-04T12:43:29Z", "updatedAt": "2026-08-13T08:46:58Z", "timestamp": "2026-08-13T08:46:58Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "ready" ], "author": "sstamenk", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51029", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Forward SwiGLU clamp/alpha/beta in compressed-tensors W4A4 MXFP4 MoE", "text": "## Purpose Fixes #51028 Models whose MoE uses the clamped SwiGLU-OAI activation (e.g. MiniMax-M3 in compressed-tensors MXFP4 W4A4 form, `olka-fi/MiniMax-M3-MXFP4`) crash at startup with: ``` RuntimeError: Worker failed with error 'SWIGLUOAI_UNINTERLEAVE requires clamp_limit' ``` `CompressedTensorsW4A4Mxfp4MoEMethod.get_fused_moe_quant_config` never forwards the layer's `swiglu_limit` / `swiglu_alpha` / `swiglu_beta` into the `FusedMoEQuantConfig`, so `gemm1_clamp_limit` / `gemm1_alpha` / `gemm1_beta` stay `None` and the clamp assert in `apply_moe_activation` (`vllm/model_executor/layers/fused_moe/activation.py`) trips. All four sibling compressed-tensors MoE methods (`w4a4_nvfp4`, `w8a8_fp8`, `w8a8_mxfp8`, `wna16`) already forward these via the `getattr(layer, \"swiglu_*\", None)` idiom; only the W4A4 MXFP4 method was missed. Branch-by-branch: 1. **Marlin / W4A16 branch**: `make_mxfp4_moe_quant_config` (`oracle/mxfp4.py`) already accepts `gemm1_alpha` / `gemm1_beta` / `swiglu_limit` and threads them into every backend config, and `MarlinExperts` already consumes them — the fix is to pass the three kwargs at the call site. 2. **CUTLASS W4A4 branch**: `mxfp4_moe_quant_config` (`fused_moe/config.py`) accepted only `w1_scale`/`w2_scale`. Extended its signature with `gemm1_alpha` / `gemm1_beta` / `gemm1_clamp_limit` (all defaulting to `None`, mirroring `nvfp4_w4a16_moe_quant_config` / `int4_w4a16_moe_quant_config` in the same file), passed through to `FusedMoEQuantConfig.make(...)`. Zero behavior change for the only other caller (`inc_mxfp4_moe.py`). Consumption needs no changes on current main: `ApplyMoEActivationConfig` already reads `gemm1_clamp_limit` / `gemm1_alpha` / `gemm1_beta` straight off the quant config in `from_configs`, it is built centrally in `FusedMoEModularKernel`, `run_cutlass_moe_mxfp4` already threads `activation_config` into `apply_moe_activation`, and `CutlassExpertsMxfp4._supports_activation` already delegates to `apply_moe_activation_supported`, which includes `SWIGLUOAI_UNINTERLEAVE`. So once the quant config carries the three values, the existing activation plumbing picks them up end to end. (An earlier revision of this PR wired those values through `cutlass_moe.py` by hand; the `ApplyMoEActivationConfig` refactor has since made that redundant, and this PR has been rebased down to construction-time plumbing only.) Notes: - Construction-time plumbing is the only reliable fix location: the config is consumed at kernel construction in `process_weights_after_loading`, so post-hoc mutation is not a viable workaround, and construction-time forwarding matches all sibling methods. - The fused `SILU` + `silu_and_mul_mxfp4_experts_quant` fast path in `run_cutlass_moe_mxfp4` is deliberately left untouched; a hypothetical `SILU`-with-clamp config on this path is out of scope here and is the class of problem the per-activation `supports_swiglu_clamp_limit` interface (#43589) is designed to surface. This PR does not overlap #43589's files. - Same fix family as #46845 (CT FP8, now merged — this PR's unit test sits directly alongside the one it added) and open PRs #47552 (int8), #49473 (ModelOpt FP8), #49854 (ModelOpt NVFP4), plus closed issues #48493 / #45624. ## Test Plan - New CPU unit test `tests/quantization/test_compressed_tensors.py::test_compressed_tensors_w4a4_mxfp4_moe_forwards_swiglu_params`, mirroring the pattern established by #46845: constructs the quant method directly and asserts both branches (CUTLASS W4A4 and Marlin W4A16) forward `gemm1_alpha` / `gemm1_beta` / `gemm1_clamp_limit` into the returned `FusedMoEQuantConfig`. - `ruff check` / `ruff format --check` (ruff 0.14.0, matching `.pre-commit-config.yaml`) pass on all touched files. - End-to-end: `vllm serve olka-fi/MiniMax-M3-MXFP4 --load-format instanttensor -tp 4 --distributed-executor-backend ray` on NVIDIA GB10 (`sm_121`, aarch64, DGX OS 7.5). Maintainers: which CI label should be applied for the CUTLASS MXFP4 MoE path (Blackwell runner)? Happy to adjust test placement if a kernels-level test is preferred. ## Test Result - Unit test asserts pass for both branches (CPU-only, no GPU required). - Startup crash is gone and generation quality is correct with the fix applied: a functionally equivalent patch (same three values populated on the quant config) has been serving MiniMax-M3-MXFP4 in production on a GB10 cluster via the CUTLASS W4A4 branch since 2026-08-04 against `0.26.1rc1.dev247+ge92dc7a9c`, with correct outputs. Without the fix, the server crashes at startup with `RuntimeError: Worker failed with error 'SWIGLUOAI_UNINTERLEAVE requires clamp_limit'`.", "url": "https://github.com/vllm-project/vllm/pull/51029", "createdAt": "2026-08-04T13:38:38Z", "updatedAt": "2026-08-13T10:44:07Z", "timestamp": "2026-08-13T10:44:07Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "nvidia", "quantization" ], "author": "Stupidoodle", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][K3] Extend FP8 asm MLA prefill to non-divisor small head counts", "text": "## Context The AITER FP8 MLA **prefill** path (`mla_prefill_ps_asm_fwd` + `mla_reduce_v1`) is gated on `num_heads % 16 == 0`. Kimi-K3 has 96 heads over `kv_lora_rank=512` → **12 heads/rank at TP8**, a non-divisor of 16, so its FP8 prefill falls back to the BF16 FMHA decompress path. That fallback builds a bf16 working set not covered by the FP8 KV-pool accounting and exhausts the activation arena at long context (K3 OOM'd at ~197k tokens, KV pool <4% used). This is the **prefill counterpart to #50578** (asm decode pad 12→16). MLA attention is independent per query head over the shared latent KV, so padding the query heads up to 16 and slicing the result back is exact. Rebased onto current `main` (includes merged #50578). Prefill-only diff: **+48 / −11**, one file (`rocm_aiter_mla.py`). ## Changes **FP8 prefill enablement (this PR):** - Relax the `_fp8_prefill_enabled` gate to also allow `0 < num_heads < 16`, and **gate it on FP8 KV** (`kv_cache_dtype_str == \"fp8\"` in the metadata builder; `is_quantized_kv_cache(kv_cache_dtype)` in the impl). Without this, a bf16-KV serve would reserve PS workspace unnecessarily. - In `_mla_fp8_prefill_attn`: replicate-pad Q/K/V to 16 via `AiterMLAHelper.get_mla_padded_q` (from #50578), run PS asm prefill + `mla_reduce_v1` at 16 heads, slice output back to `num_heads`. - Set `num_head_k = max(16, num_heads)` in `_init_fp8_prefill_ps_buffers` and `_build_fp8_prefill_ps_metadata` so work/reduce maps match padded tensors — also cuts partial tiles (`gcd(16,256)=16` vs `gcd(12,256)=4`), reclaiming **~6 GiB** PS workspace. Decode padding / Gluon routing is **not** in this PR — that is #50578 (merged). ## Dependencies / related - **#50578** (merged) — `AiterMLAHelper.get_mla_padded_q` / asm decode for non-divisor heads. - **ROCm/aiter#4452** (merged `a63ede724`) — 64-bit paged-KV byte offsets for >4 GB addressing. - Composes with **#48712** (gate PS workspace on fp8 KV) — this PR includes the same gate locally. ## vllm serve command **Hardware:** 8× MI355X (gfx950), TP8 **Model:** `moonshotai/Kimi-K3` (mxfp4) ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export AITER_SITUV2_A8W4=1 export SAFETENSORS_FAST_GPU=1 export GPU_ARCHS=gfx950 export HF_HUB_CACHE=/dev/shm/hf-cache HF_HOME=/dev/shm/hf-cache export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 export VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900 export HIPBLASLT_WORKSPACE_SIZE=32768 export CUBLASLT_WORKSPACE_SIZE=32768 MODEL=/path/to/Kimi-K3 # e.g. /shared_nfs/models/Kimi-K3 vllm serve \"$MODEL\" --served-model-name moonshotai/Kimi-K3 \\ --host 0.0.0.0 --port 8888 --tensor-parallel-size 8 --async-scheduling \\ --distributed-executor-backend mp \\ --gpu-memory-utilization 0.95 \\ --max-num-seqs 64 --max-model-len 1048576 --max-num-batched-tokens 4096 \\ --trust-remote-code --load-format auto --moe-backend aiter \\ --kv-cache-dtype fp8 --attention-backend ROCM_AITER_MLA --mm-encoder-tp-mode data \\ --compilation-config '{\"mode\":3,\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \\ --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 --enable-auto-tool-choice \\ --enable-prompt-tokens-details --disable-uvicorn-access-log ## Test results ### Long-context prefill (functional) | | Before PR-A | After PR-A | |---|-------------|------------| | Fresh prefill @ util 0.95 | OOM ~197k tokens (BF16 FMHA fallback, KV <4%) | **470k (68 s)**, **590k (28.6 s)** OK | | PS workspace | ~4032 partial tiles (~12 heads) | ~960 partial tiles (~6 GiB saved) | ### Accuracy — GSM8K (lm-eval, full 1319-test split, 5-shot greedy, conc=32, ctx=16384) Patches applied for this run: **#50578 + PR-A + #50618 only** (no cgmem/MoE/kv_b_proj). Server at `gpu-memory-utilization 0.88`. ```bash export HF_HUB_CACHE=/dev/shm/hf-cache HF_HOME=/dev/shm/hf-cache export OPENAI_API_KEY=EMPTY python3 -m lm_eval --model local-chat-completions --apply_chat_template \\ --include_path /path/to/InferenceX/utils/evals \\ --tasks /path/to/InferenceX/utils/evals/gsm8k.yaml \\ --output_path /tmp/eval_out --log_samples \\ --model_args \"model=moonshotai/Kimi-K3,base_url=http://localhost:8888/v1/chat/completions,api_key=EMPTY,eos_string=,max_retries=5,num_concurrent=32,timeout=1800,tokenized_requests=False,max_length=16384\" \\ --gen_kwargs \"max_tokens=12288,temperature=0,top_p=1\" ``` | Run | n | strict-match EM | flexible-extract EM | |-----|---|-----------------|---------------------| | **With PR-A** (MI355X node 193, 2026-08-09) | 1319 | **97.04%** ± 0.47% | **96.97%** ± 0.47% | | Prior stack (cgmem+MoE+PR-A, node 1319, util 0.95) | 1319 | 96.89% ± 0.48% | 96.89% ± 0.48% | | Baseline replicate (stock + ASM deps, node 1319) | 1319 | 96.66% ± 0.49% | 96.66% ± 0.49% | No accuracy regression vs baseline; delta within run noise.", "url": "https://github.com/vllm-project/vllm/pull/51040", "createdAt": "2026-08-04T14:48:33Z", "updatedAt": "2026-08-12T23:12:27Z", "timestamp": "2026-08-12T23:12:27Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "verified", "k3" ], "author": "xiaohuguo2023", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51075", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Rust] MiniMax-M3 tool parser: accept elided first-parameter opening tag", "text": "## Purpose Fixes #51073 Deployed MiniMax-M3 checkpoints (`MiniMaxAI/MiniMax-M3-MXFP8`; chat template byte-identical on `olka-fi/MiniMax-M3-MXFP4`) deterministically emit tool-call invokes whose **first parameter is missing its opening tag**: the value sits directly between the namespace sentinel and a closing tag carrying the parameter name. Real capture (no whitespace around values; line broken here for readability only): ```text ]<]minimax[>[ ]<]minimax[>[]<]minimax[>[offerta.docx]<]minimax[>[]<]minimax[>[Hello]<]minimax[>[]<]minimax[>[]<]minimax[>[ ``` The Rust grammar in `rust/src/parser/src/tool/minimax_m3.rs` hard-fails at that parameter boundary: `parse_invoke_params` treats a namespace marker not followed by `<` as malformed (`ErrMode::Cut` -> `ToolParserError::ParsingFailed`), and the same rejection exists in nested element bodies. The whole message then degrades to raw markup content instead of a native tool call. Full root-cause analysis is in #51073. This PR closes the emission-vs-grammar mismatch with a tolerant rule (`elided_parameter_element`): - At a parameter boundary — top level of an invoke body, and inside nested element bodies — a namespace marker **not** followed by `<` starts an elided-opening-tag parameter. - Its value is the text up to the next namespace marker, which must begin the closing tag that names the parameter. - The rule is positional: the invoke body is already delimited by `]<]minimax[>[`, so every closing tag inside it unambiguously ends one parameter. Any parameter may therefore be elided, not just the first. - In nested element bodies, recovery additionally requires the discovered closing tag to differ from the parent's, which keeps the parent element properly terminated. - Recovered values flow through the existing schema-coercion path, identical to tagged parameters. Every other previously-malformed shape stays malformed, and well-formed input is unaffected. Deliberate non-goal, left as follow-up as suggested in #51073: stripping namespace sentinels from the raw-content fallback when parsing still fails unrecoverably. ## Test Plan New unit fixtures in `rust/src/parser/src/tool/minimax_m3.rs`, built from the real captures: - `minimax_m3_parse_complete_accepts_elided_first_param_opening_tag` — capture 1 reconstruction (elided first parameter + fully tagged second) - `minimax_m3_parse_complete_accepts_elided_first_param_second_capture` — capture 2 (truncated in flight at `]<]minimax[>[ integer) - `minimax_m3_parse_complete_accepts_elided_first_child_in_nested_element` — first child of a nested object parameter elided - `minimax_m3_elided_close_tag_matching_parent_stays_malformed` — parent-close guard: preserved strictness for the ambiguous shape - `minimax_m3_parse_complete_extracts_mixed_elided_and_tagged_invokes` — multi-invoke message mixing an elided-parameter invoke with a fully tagged one, plus prefix text - `minimax_m3_streaming_handles_elided_param_split_across_chunks` — 3-char chunking so every marker is split across chunk boundaries Full crate suite: ```bash cd rust/src/parser && cargo test --lib ``` ## Test Result ```text test result: ok. 423 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` 416 pre-existing tests stay green, plus the 7 fixtures above. `cargo fmt --check` is clean on the touched file. An equivalent tolerant rule, ported as a Python `--tool-parser-plugin`, is serving `MiniMaxAI/MiniMax-M3-MXFP8` in production on a 6-node GB10 (sm_121) cluster. Verified end-to-end after deployment: OpenAI-compatible non-streaming (`finish_reason: \"tool_calls\"`, complete JSON arguments), OpenAI-compatible streaming (tool-call deltas with full name + arguments), and Anthropic-compatible `/v1/messages` (`stop_reason: \"tool_use\"` with populated `input`) — all with clean `content` (no leaked sentinels) and zero parser errors in the server log, including under live agent traffic (multi-turn tool loops from an AnythingLLM client).", "url": "https://github.com/vllm-project/vllm/pull/51075", "createdAt": "2026-08-04T20:31:43Z", "updatedAt": "2026-08-13T10:55:51Z", "timestamp": "2026-08-13T10:55:51Z", "metrics": { "reactions": 3, "comments": 4 }, "labels": [ "bug", "rust" ], "author": "Stupidoodle", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][TurboQuant] Resolve the cache preset when get_kv_cache_shape gets no dtype hint", "text": "# Issue #50709 — TurboQuant hybrid model crashes with `Unknown cache dtype: auto` Upstream issue: https://github.com/vllm-project/vllm/issues/50709 Reported symptom: serving a hybrid model (Qwen3.5-35B-A3B: full attention + GDN linear attention + MoE) with `--kv-cache-dtype turboquant_4bit_nc` aborts during startup (`determine_available_memory()`) with ``` Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc ``` ## 1. Root cause The only site in the tree that can emit that message is `TurboQuantConfig.from_cache_dtype()` (`vllm/model_executor/layers/quantization/turboquant/config.py:220`), and the only caller that can reach it with `\"auto\"` is `TurboQuantAttentionBackend.get_kv_cache_shape()`. `AttentionBackend.get_kv_cache_shape(num_blocks, block_size, num_kv_heads, head_size, cache_dtype_str=\"auto\")` treats `cache_dtype_str` as an **optional hint** — every other backend derives its shape from the four positional arguments alone. TurboQuant is the only backend that makes the hint *load-bearing*: it re-parses it into a `TurboQuantConfig` to obtain `slot_size_aligned` (the packed `[key|value]` slot width), and hard-fails on anything that is not a preset name. Several in-tree callers legitimately do not supply a per-layer preset: - `AttentionBackend.get_kv_cache_block_dim()` (`vllm/v1/attention/backend.py:100`) defaults the hint to `\"auto\"` and forwards it — so *any* call to `TurboQuantAttentionBackend.get_kv_cache_block_dim(...)` raises. - The KV-connector layout probes call `get_kv_cache_shape()` with no hint at all (`vllm/distributed/kv_transfer/kv_connector/utils.py:430`, `.../offloading/worker.py:250`). - Both model runners downgrade the hint to `\"auto\"` for every KV-cache group whose spec reports `KVQuantMode.NONE`: - v2: `vllm/v1/worker/gpu/attn_utils.py:323` (`_reshape_kv_cache`) - v1: `vllm/v1/worker/gpu_model_runner.py:7415` (`_reshape_kv_cache_tensors`) That downgrade exists for `--kv-cache-dtype-skip-layers`, where individual layers keep the native dtype while the engine-wide `cache_dtype` stays quantized. It reasons from the *spec's* quant mode but hands the result to the *group's backend*, and nothing ties those two together. The v1 runner's `_reshape_kv_cache_tensors` is reached from `determine_available_memory()` → `profile_cudagraph_memory()` → `_init_minimal_kv_cache_for_profiling()` → `initialize_kv_cache()`, which is exactly the stage named in the report. On hybrid models the extra spec types (`MambaSpec` for the GDN layers, plus the skip/boundary machinery that TurboQuant enables around them) are what make a group reach the reshape with a `KVQuantMode.NONE` spec while still being served by the TurboQuant backend. Reproduced directly (see §5): `get_kv_cache_shape(..., cache_dtype_str=\"auto\")` and `get_kv_cache_block_dim(16, 8, 128)` both raise the reported error verbatim. ## 2. The fix and why `vllm/v1/attention/backends/turboquant_attn.py`: added `_resolve_tq_preset()` and routed `get_kv_cache_shape()` through it. When the caller-supplied hint is not a TurboQuant preset, the preset is taken from the engine's configured `cache_config.cache_dtype`; if that is not a preset either, the original `ValueError` is raised unchanged. Why this is safe and correct rather than a papering-over: - A layer can only ever be served by this backend if its KV cache dtype is a `turboquant_*` preset. `TurboQuantAttentionBackend.supports_kv_cache_dtype()` requires `kv_cache_dtype.startswith(\"turboquant_\")`, and `AttentionBackend.validate_configuration()` enforces it for *both* auto-selected and explicitly selected (`--attention-config backend=...`) backends (`vllm/platforms/cuda.py:407-423`, `vllm/v1/attention/backend.py:344`). Since there is a single engine-wide `--kv-cache-dtype`, `cache_config.cache_dtype` is by construction the preset of every layer this backend serves. - The fix restores the documented contract of `get_kv_cache_shape`: the shape is a property of the backend and its configuration, not of a hint the caller may not have. It therefore covers all four caller classes above at once, instead of patching one runner and leaving `get_kv_cache_block_dim()` and the KV-connector probes still broken. - It is not a silent fallback: outside a TurboQuant run the clear diagnostic is preserved (covered by a test). Explicitly *not* changed, to keep the diff scoped to this issue: - The `KVQuantMode.NONE → \"auto\"` heuristic in the two runners. It is the proximate trigger, but it is shared with MLA/`fp8_ds_mla` and is already the subject of open upstream PRs (see §4); changing it here would collide. - `get_kv_cache_shape`'s hardcoded `cache_dtype_str=\"turboquant_4bit_nc\"` default. Worth revisiting separately: callers that omit the argument (the KV-connector probes) currently get a `turboquant_4bit_nc` slot width even on a `turboquant_k3v4_nc` run. Harmless today — those probes only inspect the shape's rank and the position of `num_blocks` — but latent. ## 3. Files changed | File | Change | | --- | --- | | `vllm/v1/attention/backends/turboquant_attn.py` | New `_resolve_tq_preset()`; `get_kv_cache_shape()` resolves the preset through it (+25/-1). | | `tests/quantization/test_turboquant.py` | New `TestTurboQuantKVCacheShape` regression class (5 tests) + one import (+50). | Test placement follows AGENTS.md (\"reuse before create\"): the existing TurboQuant suite already covers config/preset behaviour and runs on CPU. ## 4. Risk / uncertainty - **Duplicate check.** Done through the GitHub web UI (the task forbids running `gh`). No open PR references #50709. Adjacent open PRs touch the same `KVQuantMode.NONE → \"auto\"` heuristic — #47618 (\"Unify per-layer KV cache dtype selection across v1/v2\"), #48907 (\"Prefer per-spec `cache_dtype_str` when reshaping KV cache\"), #49798 (\"add TQFullAttentionSpec guard in v1 `_reshape_kv_cache_tensors`\") — but all three fix the *runner* side for MLA specs that carry a `cache_dtype_str` field. `TQFullAttentionSpec` has no such field, and none of them makes the TurboQuant backend tolerate a missing hint, so `get_kv_cache_block_dim()` and the KV-connector probes stay broken under all three. This change is complementary, not a duplicate — but if any of them lands first the interaction should be re-checked. - **Main uncertainty: I could not reproduce the exact hybrid configuration** that makes a TurboQuant-backed group's spec report `KVQuantMode.NONE`. There is no GPU and no model checkpoint in this environment, and the reporter attached no traceback. I verified the raise site, the callers, and the backend-selection invariant statically, and reproduced the error itself directly; I did not observe the upstream spec that loses the quant mode. The fix is therefore placed at the point where the invariant is *provable* (backend ⇒ TurboQuant dtype) rather than at whichever spec path produces the `NONE`. If the real trigger is a spec whose page size was also computed with the unquantized formula, this change turns the hard failure into a tensor shape/size mismatch at reshape time — a different, still-loud error, not silent corruption. - **Behaviour change:** `get_kv_cache_shape(\"auto\")` now succeeds inside a TurboQuant run instead of raising. Nothing in-tree depends on that raise. - `_resolve_tq_preset()` calls `get_current_vllm_config()`. This is already the established pattern in this file (`TurboQuantAttentionImpl.__init__`), and when no config is set the helper falls through to the original error rather than guessing. - **Not verified:** GPU kernels, end-to-end serving, and model quality. Per AGENTS.md a model eval would be required before submitting this upstream; it cannot be run here. This change does not alter any quantization math or kernel — only which preset name the shape helper resolves — so no accuracy impact is expected. ## 5. How I verified it Environment: `uv venv --python 3.12`, CPU-only torch 2.13.0 + `requirements/common.txt` (no GPU, no compiled `vllm._C`; the TurboQuant config/shape path is pure Python). **Reproduction of the reported error on unmodified `main`:** ``` $ .venv/bin/python -c \"...TurboQuantAttentionBackend.get_kv_cache_shape(4,16,8,128,cache_dtype_str='auto')\" RAISED: Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, ... $ .venv/bin/python -c \"...TurboQuantAttentionBackend.get_kv_cache_block_dim(16, 8, 128)\" RAISED block_dim: Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, ... ``` **Test fails before the fix / passes after.** With the source change stashed and the new tests in place, all 5 fail with the reported `ValueError`. With the fix applied: ``` $ .venv/bin/python -m pytest tests/quantization/test_turboquant.py -q -k KVCacheShape 5 passed, 127 deselected ``` **No regressions in the touched and adjacent suites:** ``` $ .venv/bin/python -m pytest tests/quantization/test_turboquant.py -q 115 passed, 17 skipped $ .venv/bin/python -m pytest tests/v1/worker/test_attn_utils.py tests/v1/test_kv_cache_spec_registry.py -q 42 passed ``` Note: every test in this environment also reports a teardown `ERROR` from `tests/conftest.py`'s autouse `cleanup_dist_env_and_memory()` fixture (`RuntimeError: Cannot access accelerator device when none is available`). This is the missing GPU, not the change — it reproduces identically on files I did not touch (`tests/v1/test_kv_cache_spec_registry.py`: 42 passed, 42 errors). **Behavioural spot-check** (`turboquant_k3v4_nc` engine, slot 118 vs `turboquant_4bit_nc` slot 134 — the resolver picks the configured preset, an explicit hint still wins, and a non-TurboQuant run still errors): ``` auto -> (4, 8, 16, 118) block_dim -> 0 explicit -> (4, 8, 16, 134) cache_dtype=\"auto\" engine -> ValueError: Unknown TurboQuant cache dtype: 'auto' ``` **Lint:** `ruff check` and `ruff format --diff` clean on both changed files (88-char limit respected).", "url": "https://github.com/vllm-project/vllm/pull/51082", "createdAt": "2026-08-04T21:34:13Z", "updatedAt": "2026-08-13T03:54:18Z", "timestamp": "2026-08-13T03:54:18Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51084", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Profiler] Add Proton CUDA graph attribution", "text": "## Purpose Extend the Proton backend merged in #48789 with opt-in CUDA graph replay attribution. Proton must observe graph creation using the same Data object that later receives replay events. Setting `proton_graph_attribution` to true therefore creates the user-configured Proton session immediately before vLLM enters CUDA graph capture, keeps that session active throughout capture, deactivates it with flushing after capture, and reactivates the same session when profiling starts. - Add `proton_graph_attribution`, disabled by default. - Require `proton_data=\"tree\"` when graph attribution is enabled. - Preserve the configured context, backend, mode, hook, and output format during capture. - Reuse one Proton session for graph capture and replay profiling. - Use Triton phases to discard graph-capture activity before request profiling. - Keep the session alive across repeated start/stop cycles; each stop advances the phase, flushes, deactivates, writes that phase, and clears it. - Avoid Proton initialization during CUDA graph capture unless graph attribution is explicitly enabled and graphs actually need capture. - Reject PC sampling with graph attribution because that mode synchronizes the CUDA context. - Require Triton 3.7 or newer for Proton profiling. No option is rewritten for CUDA graph capture, and no dedicated capture-only Proton session is created. Without `proton_graph_attribution`, Proton retains its normal endpoint behavior: each `/start_profile` creates an independent profiling session, and each `/stop_profile` finalizes it and writes the profile immediately. ## Scope This PR only adds opt-in CUDA graph capture and replay attribution for the V2 model runner. It does not add distributed rollback, HTTP error mapping, numeric trace metadata, or ROCm support. Those concerns remain separate follow-ups. ## CUDA graph attribution example Start vLLM with the V2 model runner and opt in to graph attribution. Tree data is required because Proton stores the capture/replay relationship in its tree representation. ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve facebook/opt-125m \\ --profiler-config '{ \"profiler\": \"proton\", \"proton_profiler_dir\": \"./proton_profile\", \"proton_data\": \"tree\", \"proton_output_format\": \"hatchet\", \"proton_hook\": \"triton\", \"proton_graph_attribution\": true }' ``` Profile a request through the existing endpoints: ```bash curl -X POST http://localhost:8000/start_profile curl http://localhost:8000/v1/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"facebook/opt-125m\", \"prompt\": \"CUDA graph replay\", \"max_tokens\": 32 }' curl -X POST http://localhost:8000/stop_profile ``` The stop call flushes the completed phase and immediately writes a rank-qualified Hatchet file under `./proton_profile`. In that file, kernels executed through CUDA graph replay are attributed to the scopes recorded when vLLM captured the graph. The same server can profile another interval without losing graph attribution: ```bash curl -X POST http://localhost:8000/start_profile # Send another inference request. curl -X POST http://localhost:8000/stop_profile ls ./proton_profile/*.hatchet proton-viewer -m time/ns \\ ./proton_profile/proton_dp0_pp0_tp0_dcp0_ep0_rank0_pid1234__run0.hatchet ``` `proton-viewer` displays the attributed tree, including the `` scopes that connect replayed kernels to CUDA graph capture. Each start/stop cycle writes a distinct profile. The long-lived Proton session retains only the graph metadata needed for replay attribution; activity from graph capture and completed profiling phases is cleared. ## Impact Model outputs and serving semantics are unchanged. Users who enable `proton_graph_attribution` can attribute kernels executed through vLLM CUDA graph replay while preserving repeated `/start_profile` and `/stop_profile` semantics. ## Test results - Focused worker profiler suite: 75 passed. - All applicable changed-file pre-commit hooks passed, including Ruff, markdownlint, repository checks, and mypy. - Real NVIDIA H20 FULL/FULL_AND_PIECEWISE CUDA graph smoke test with `facebook/opt-125m` and `proton_graph_attribution=true`: - Two consecutive `start_profile` / graph-replay generation / `stop_profile` cycles returned HTTP 200. - Each stop immediately wrote a distinct Hatchet profile. - Both profiles contained `` attribution entries. - Earlier configuration coverage confirmed that graph attribution works with both shadow and python context sources; graph-attributed output is intentionally restricted to TreeData because Triton's phase export API does not expose TraceData. AI assistance disclosure: OpenAI Codex assisted with implementation, code review, local test orchestration, splitting the original change, and PR preparation. The human submitter remains responsible for reviewing and understanding every changed line before merge.", "url": "https://github.com/vllm-project/vllm/pull/51084", "createdAt": "2026-08-04T22:09:08Z", "updatedAt": "2026-08-12T20:27:29Z", "timestamp": "2026-08-12T20:27:29Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "performance", "nvidia" ], "author": "Luosuu", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51119", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] Use saturated Gluon split bucket for CUDA graphs", "text": "## Summary This PR changes the default `AiterMLADecodeMetadata.min_kv_seq_len` from `1` to `16384`. The change is intentionally small, but the reason is specific to the interaction between vLLM FULL CUDA Graph capture and AITER's host-side Gluon split selection: - AITER uses `min_kv_seq_len` to choose the split-KV launch topology in its Python wrapper. - vLLM currently leaves this field at its default value. - FULL CUDA Graph capture runs the wrapper once with dummy decode metadata and then replays captured GPU work without running the wrapper again. - With the old default, the captured graph is permanently specialized to the one-split path, even when the real request has a long KV sequence. This PR only fixes that vLLM integration issue. Split-policy tuning and stage-1/stage-2 kernel optimization remain AITER responsibilities. ## How the issue was found I first found this from Kimi-K3 TP8 end-to-end serving performance on gfx950. The 100K-input / 1K-output, concurrency-1 workload measured roughly 87–90 ms TPOT, while an isolated 12-head Gluon configuration with sufficient split parallelism indicated that the decode path should be much faster. The initial investigation focused on AITER. [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) tuned the 12-head BF16 Gluon split policy and tested context buckets for the Kimi-K3 TP8 shape. Reducing the split count also reduced the cost of the existing serial stage-2 reduction, so the first apparent solution was an AITER split-policy change. During the review of [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450), [@Dewei-Wang-sh](https://github.com/Dewei-Wang-sh) and I spent an afternoon tracing this behavior from the AITER wrapper back into vLLM. His key review direction was to [pass the split-sizing information from the caller](https://github.com/ROCm/aiter/pull/4450#discussion_r3710954584). He explained that the existing AITER interface follows the Moonshot integration: Moonshot obtains a meaningful minimum KV length in its runtime and passes it to AITER. In that contract, `1` is a fallback default; it is not intended to describe every real long-context decode request. This exposed the integration mismatch. vLLM was leaving the fallback at `1`. More importantly, when FULL CUDA Graph was enabled, vLLM performed warmup/capture with a dummy sequence length of 1. Dewei helped me follow the execution path and confirm that real decode replay did not run the host wrapper again. Therefore, even an attempted runtime assignment could not reach the Python code that selects the launch topology. This discussion and the subsequent call-chain tracing moved the fix from AITER to the vLLM caller. Thank you to Dewei for the detailed review and for helping identify the actual boundary of the problem. The relevant AITER host calculation is in [`mla_gluon.py`](https://github.com/ROCm/aiter/blob/d1898b7af894bda10310a631963771782e5aebc4/aiter/ops/triton/gluon/mla_gluon.py#L1042-L1048): ```python NUM_KV_SPLITS = max( 1, min( 256 // (batch_size * qlen * NUM_M_BLOCKS), triton.cdiv(min_kv_seq_len, BLOCK_N), ), ) ``` For the `BLOCK_N=64` path, capture with `min_kv_seq_len=1` gives `cdiv(1, 64)=1`, so the host calculation collapses to one KV split. The captured graph then permanently records that one-split launch. With the vLLM capture hint set to `16384`, the same term becomes `cdiv(16384, 64)=256`; it no longer forces the policy to one split, and AITER can apply its workgroup budget and small-head policy to capture a saturated, non-one-split topology. The real sequence length is still read from device metadata during replay; `16384` is only the host-side capture hint. In vLLM, the field was declared with a default of `1`, and `_build_decode()` constructed `AiterMLADecodeMetadata` without assigning it. `forward_mqa()` then passed that unchanged value to `mla_gluon()`: - [`min_kv_seq_len` metadata default](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L129-L133) - [`_build_decode()` metadata construction](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L691-L703) - [`forward_mqa()` passing the host hint to AITER](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L1091-L1106) Therefore, even a real 10K or 100K request reached the AITER wrapper with `min_kv_seq_len=1` in normal execution. ## Why assigning the real runtime length did not fix FULL CUDA Graph I next tested assigning the real value while constructing decode metadata: ```python min_kv_seq_len=int(seq_lens_for_kernel.min()) ``` For an actual request this computes the expected value, for example approximately 10K or 100K. It works conceptually in eager execution because the AITER Python wrapper runs for every call and can select a new split count. It does not work for FULL CUDA Graph capture. For pure decode, vLLM's dummy run has query length 1. The capture path consequently builds dummy sequence lengths from `max_query_len`, which is also 1: - [pure-decode dummy batch construction](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/worker/gpu_model_runner.py#L5888-L5902) - [dummy attention sequence length selection](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/worker/gpu_model_runner.py#L5988-L6015) The resulting sequence is: ```text capture startup: max_query_len = 1 dummy seq_len = 1 min_kv_seq_len = 1 AITER wrapper selects NUM_KV_SPLITS = 1 CUDA Graph records the one-split grid and the no-stage-2 fast path real request: device metadata is updated to the real KV length FULL graph replay starts AITER Python wrapper is not called again captured split count, grid, workspace, and stage topology remain unchanged ``` ## Instrumented FULL CUDA Graph trace To verify this rather than infer it from performance, I instrumented an isolated vLLM/AITER copy and ran one Kimi-K3 TP8 request with 100K input tokens and 4 output tokens. Logging was restricted to TP rank 0. The complete capture call chain was: ```text worker_busy_loop -> compile_or_warm_up_model -> capture_model -> _capture_cudagraphs -> _warmup_and_capture -> _dummy_run -> self.model -> BreakableCUDAGraphWrapper._capture -> Kimi-K3 MLA layer -> unified_mla_attention_with_output -> forward_mqa -> mla_gluon ``` The following excerpts are sanitized to retain only the events relevant to MLA graph capture and replay.
Eager warmup ```text MLA_TRACE event=dummy_metadata mode=NONE seq_lens=1 MLA_TRACE event=aiter_metadata_build seq_lens=[1] static_seq_info_ptr=137855956567552 MLA_TRACE event=forward_mqa_call stream_capturing=False host_min_kv_seq_len=1 MLA_TRACE event=mla_wrapper_split stream_capturing=False block_n=64 num_kv_splits=1 MLA_TRACE event=mla_stage1_launch stream_capturing=False grid=(1, 1, 1) logits_shape=(1, 1, 12, 1, 512) mid_lse_shape=None MLA_TRACE event=mla_stage2_skipped stream_capturing=False reason=num_kv_splits_is_1 ```
FULL graph capture ```text MLA_TRACE event=dummy_metadata mode=FULL graph_capture_request=True seq_lens=1 MLA_TRACE event=model_capture_enter mode=FULL MLA_TRACE event=forward_mqa_call stream_capturing=True host_min_kv_seq_len=1 seq_info_ptr=137855956567552 MLA_TRACE event=mla_wrapper_split stream_capturing=True block_n=64 num_kv_splits=1 MLA_TRACE event=mla_stage1_launch stream_capturing=True grid=(1, 1, 1) logits_shape=(1, 1, 12, 1, 512) mid_lse_shape=None MLA_TRACE event=mla_stage2_skipped stream_capturing=True reason=num_kv_splits_is_1 MLA_TRACE event=model_capture_exit mode=FULL graph_segments=1 eager_breaks=0 ``` At this point capture has fixed all of the host-selected topology: - `NUM_KV_SPLITS=1` - stage-1 grid `(1, 1, 1)` - workspace/logits shape `(1, 1, 12, 1, 512)` - `mid_lse=None` - no stage-2 GPU node exists in the graph
Real 100K decode replay ```text MLA_TRACE event=runtime_dispatch mode=FULL computed_before_step=[100000] MLA_TRACE event=aiter_metadata_build seq_lens=[100001] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=1 segments=1 eager_breaks=0 MLA_TRACE event=aiter_metadata_build seq_lens=[100002] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=2 segments=1 eager_breaks=0 MLA_TRACE event=aiter_metadata_build seq_lens=[100003] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=3 segments=1 eager_breaks=0 ``` The capture-time `seq_info_ptr` and all three runtime `static_seq_info_ptr` values are the same: `137855956567552`. This proves that the real metadata was copied into the same static GPU buffer used by the graph. However, there were no `forward_mqa_call`, `mla_wrapper_split`, or Python stage-launch events between the three replay steps.
### Why the wrapper is not re-entered This is not a JIT-cache effect. During capture, `BreakableCUDAGraphWrapper` executes the model and records the graph. On later calls, its [capture-versus-replay dispatch](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L310-L333) selects `_replay()`, which [calls only `entry.capture.replay()`](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L406-L424). The capture artifact then [replays its saved segments](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L210-L214). This run reported `graph_segments=1, eager_breaks=0`, so the only saved segment was GPU graph replay; it was not a Python model, attention, or AITER-wrapper invocation. A PIECEWISE graph can re-enter Python when attention is registered as an eager break, but this FULL graph had no such break. JIT kernel caching also does not skip the Python wrapper and should not be confused with graph replay. The distinction is therefore: - **Dynamic during replay:** the real sequence length, because the captured kernel reads the latest values from the static `seq_info` GPU buffer. - **Frozen at capture:** `NUM_KV_SPLITS`, workspace shape, stage-1 grid, `mid_lse` allocation, and whether stage 2 exists, because those are host-side decisions made before the graph is recorded. This directly explains why a host-only split policy still regressed to approximately 87–88 ms: FULL capture recorded the `split=1` topology and replay could not change it. > **Diagnostic timing note:** this instrumented 100K-to-4-token run measured 131.33 ms TPOT, but the instrumentation added GPU-to-CPU synchronization, stack collection, and synchronous printing. That number is only call-chain evidence and must not be compared with the uninstrumented 23.49 ms result below. The raw logs are approximately 640 KB and contain unrelated runtime configuration, so only the focused, sanitized trace is included here. The runtime assignment also requires a device-to-host synchronization for `.min()`/`.item()`. It increased the measured TPOT from approximately 87–88 ms to approximately 89–90 ms while still replaying the one-split graph. It was therefore both insufficient and slower. ## Why use 16,384 for capture Changing vLLM to capture a separate graph for every context bucket would be a much larger architectural change. Moving split activation back to the device is an AITER policy decision and conflicts with the reviewer-requested host-side interface. Per-request device-to-host synchronization is not viable on the decode path. The smallest vLLM-side integration is therefore to provide one representative, saturated capture hint. `16384` was selected so capture does not fall into the one-split topology, while avoiding a context-length graph ladder or runtime synchronization. The value is a capture hint, not an assertion that every request has a 16K context. The kernel still reads the real per-request sequence length from `seq_info` during replay. The hint determines only the host-selected split topology that CUDA Graph records. The target workload matrix includes 8K/1K, 60K/1K, and 100K/1K cases. The fixed-hint end-to-end rerun completed so far is the 100K/1K case; the shorter fixed-hint cases have not yet been rerun end to end. ## Runtime-owner feedback and related AITER work The AITER runtime discussion clarified that two separate problems had been mixed together: 1. **vLLM integration correctness/performance:** the caller leaves the host hint at 1 and FULL graph capture freezes that topology. 2. **AITER kernel efficiency:** with a valid large split count, the original stage-2 reduction becomes serial and expensive. The related AITER PRs explore different kernel-side solutions: - [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) started the 12-head split-policy investigation and production-like scattered-page validation. - [ROCm/aiter#4507](https://github.com/ROCm/aiter/pull/4507) derives host launch sizing from page-table shape rather than the optional hint. - [ROCm/aiter#4509](https://github.com/ROCm/aiter/pull/4509) adds split-major stage-1 scheduling and a more parallel stage-2 reduction on top of [ROCm/aiter#4507](https://github.com/ROCm/aiter/pull/4507). - [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555), from the AITER runtime team, directly optimizes stage-2 by loading and reducing a block of partial outputs at once instead of walking one split at a time. The runtime-owner measurement makes the separation important: | Configuration | Splits | Stage 1 | Stage 2 | Total | |---|---:|---:|---:|---:| | Original serial reduction | 256 | 34.7 us | 74.3 us | 109.0 us | | Split-bucket workaround | 96 | 43.1 us | 26.1 us | 69.2 us | | [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) blocked reduction | 256 | 34.7 us | 10.7 us | 46.9 us | This shows that reducing splits helped partly because it avoided an inefficient stage-2 implementation. [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) is the better kernel optimization: it preserves enough stage-1 workgroups to fill the CUs and fixes stage-2 itself. The AITER runtime team will own that optimization. Accordingly, this PR does not claim that 48 or 96 splits is the final AITER policy. It only ensures that vLLM does not silently force every host-side policy to the pathological one-split capture. ## End-to-end validation Kimi-K3 TP8 on 8x MI355X/gfx950, BF16 KV, FULL CUDA Graph, 100K input / 1K output, concurrency 1: | Configuration | TPOT | |---|---:| | Previous host-side capture with default `1` | approximately 87.8 ms | | Runtime `min()` metadata experiment | approximately 89.2 ms | | This PR, fixed 16K capture hint | 23.49 ms | | Device-side runtime-policy reference | approximately 23.0 ms | The fixed capture hint removes the regression without changing graph dispatch or adding decode-path synchronization. ## Scope question for maintainers The remaining concern is short-context behavior. This change assumes that the small-head Gluon path is primarily valuable for the long-context workloads for which split parallelism is required. A fixed 16K hint may over-split very short requests. AITER has empty-split guards, and [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) validated explicit split counts larger than sequence lengths at boundaries including sequence length 1, so this is expected to be correct. However, the short-context performance trade-off still deserves explicit confirmation. Would maintainers prefer: 1. this single representative capture hint for the Gluon path; 2. a more narrowly scoped hint for gfx950/small-head deployments; 3. a configurable vLLM capture hint; or 4. relying on an AITER-side solution such as page-table sizing or device-side active split selection? Given the runtime-owner direction toward [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) for kernel performance, my preference is to keep this PR narrowly focused on preventing `min_kv_seq_len=1` from being frozen by vLLM FULL CUDA Graph capture. ## Change ```diff - min_kv_seq_len: int = 1 + min_kv_seq_len: int = 16384 ``` No graph-dispatch, replay-path, or AITER kernel code is changed. ## Test plan - [x] `python3 -m py_compile vllm/v1/attention/backends/mla/rocm_aiter_mla.py` - [x] Kimi-K3 TP8, gfx950, 100K input / 1K output / concurrency 1 - [x] Confirmed FULL capture enters `forward_mqa()`/`mla_gluon()` once and runtime decode uses graph replay without re-entering either wrapper - [x] Confirmed runtime sequence metadata updates the same static GPU `seq_info` buffer used by the captured graph - [ ] Re-run the fixed capture hint at 8K/1K and 60K/1K to quantify short/mid-context performance", "url": "https://github.com/vllm-project/vllm/pull/51119", "createdAt": "2026-08-05T05:55:47Z", "updatedAt": "2026-08-13T04:46:13Z", "timestamp": "2026-08-13T04:46:13Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "nvidia" ], "author": "LiuYinfeng01", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51157", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Pad SigLIP text prompts to the trained sequence length", "text": "## Purpose SigLIP is trained with `padding=\"max_length\"` and without an attention mask, so padding tokens are part of the input and the pooled embedding is taken from the last position. Text embeddings computed without padding are not aligned with the image embeddings, which makes image-text similarity unusable. The existing test only passes because it opts into padding explicitly: ```python # tests/models/multimodal/pooling/test_siglip.py tokenization_kwargs={ \"padding\": \"max_length\", \"max_length\": 64, }, # siglip2 was trained with this padding setting. ``` Callers that cannot pass tokenization kwargs get silently wrong results. The request still returns 200 with a correctly shaped vector; only the values are wrong, so the failure is easy to miss in production. Two entry points need covering, and they take different routes: | entry point | source of `TokenizeParams` | inherits model defaults | |---|---|---| | `LLM.embed` | `renderer.default_cmpl_tok_params` | yes | | `/v1/embeddings` (text and chat) | `request.build_tok_params()` | **no** | `entrypoints/pooling/base/protocol.py:97-116` builds `TokenizeParams` from scratch, so a model-level default alone does not reach the OpenAI-compatible server. Note that padding text-only prompts inside `SiglipMultiModalProcessor` does **not** work: `inputs/preprocess.py:169-181` routes text-only prompts straight to `_tokenize_prompt` and never reaches the multi-modal processor. I verified this by instrumenting `apply()` — during the text test it is only ever called with dummy profiling data. ## Fix Two small pieces. **1. The model declares its requirement** through the existing `BaseProcessingInfo.get_default_tok_params` extension point, which ten models already override (`paligemma`, `gemma4_mm`, `ovis`, `ovis2_5`, `whisper`, `ultravox`, `cohere_asr`, `lfm2_vl`, `nano_nemotron_vl`, `nemotron_parse`): ```python def get_default_tok_params(self) -> \"TokenizeParams\": tok_params = super().get_default_tok_params() return tok_params.with_kwargs(pad_prompt_tokens=self.get_text_max_length()) ``` **2. The renderer applies that default** when a frontend builds `TokenizeParams` without one, in `_tokenize_singleton_prompt` and its async twin: ```python if params.pad_prompt_tokens is not None or prompt.get(\"multi_modal_data\"): return params ``` Both guards matter. Explicitly requested padding wins. Multi-modal prompts are skipped because `renderers/base.py:786` tokenizes the prompt *before* multi-modal processing on the online path, so without the guard image requests would be padded too. The padding machinery itself already exists (`TokenizeParams.pad_prompt_tokens`, `_token_padding`); this only turns it on for a model that needs it. Models that declare nothing get `None` and the helper returns the params unchanged. ### Known trade-off A caller who deliberately passes `padding=False` for SigLIP will still get the model default reapplied, so there is no opt-out. Note this is not caused by the fallback alone. `TokenizeParams.with_kwargs` guards the padding mapping with a truthiness check: ```python if padding := tokenization_kwargs.pop(\"padding\", None): if padding == \"max_length\": pad_prompt_tokens = max_length elif padding in (False, \"do_not_pad\"): # unreachable pad_prompt_tokens = None ``` `padding=False` is falsy, so the key is popped and the `elif` never runs — `padding=False` is already a no-op on `main` for any model whose default sets `pad_prompt_tokens`. Consequently \"disabled\" and \"unspecified\" are indistinguishable to the fallback, and it reapplies the default. I judged that acceptable because unpadded SigLIP is simply incorrect, but I am happy to add an explicit sentinel if maintainers prefer to keep the opt-out. Fixing the dead `elif` branch itself looks like a separate change; happy to send one if that is wanted. ### Alternative considered Setting `pad_prompt_tokens` inside `_build_pooling_tok_params` would avoid touching the renderer, but it only fixes frontends that build `TokenizeParams` and puts a model-specific rule in the entrypoints layer. Happy to move the fix there if preferred. ## Test plan `test_models_text` is parametrized over `explicit_padding`. Both cases compare against the same HF reference computed with padding, so `explicit_padding=False` fails on `main`. `tests/renderers/test_default_padding.py` covers the server route by building `TokenizeParams` exactly the way the pooling endpoints do (no padding field). It runs on CPU in seconds. ```bash .venv/bin/python -m pytest tests/models/multimodal/pooling/test_siglip.py -v .venv/bin/python -m pytest tests/renderers/ -q ``` ## Test result All three models in `MODELS`, on an NVIDIA GB10 (sm_121, aarch64, CUDA 13.0): ``` $ .venv/bin/python -m pytest tests/models/multimodal/pooling/test_siglip.py \\ tests/renderers/test_default_padding.py -v test_models_text[True-float-google/siglip-base-patch16-224] PASSED test_models_text[True-float-google/siglip2-base-patch16-224] PASSED test_models_text[True-float-google/siglip2-giant-opt-patch16-384] PASSED test_models_text[False-float-google/siglip-base-patch16-224] PASSED test_models_text[False-float-google/siglip2-base-patch16-224] PASSED test_models_text[False-float-google/siglip2-giant-opt-patch16-384] PASSED test_models_image[float-google/siglip-base-patch16-224] PASSED test_models_image[float-google/siglip2-base-patch16-224] PASSED test_models_image[float-google/siglip2-giant-opt-patch16-384] PASSED test_models_text_image_no_crash[float-google/siglip-base-patch16-224] PASSED test_models_text_image_no_crash[float-google/siglip2-base-patch16-224] PASSED test_models_text_image_no_crash[float-google/siglip2-giant-opt-patch16-384] PASSED test_default_padding::test_model_default_is_applied PASSED test_default_padding::test_explicit_padding_is_not_overridden PASSED test_default_padding::test_multimodal_prompts_are_not_padded PASSED 15 passed in 2085.08s (0:34:45) ``` The same suite passes on an RTX 4070 SUPER (sm_89, x86_64, CUDA 12.8, vLLM built from source) for the two base models — 8 passed, giant skipped there for disk reasons. Full renderer suite on that machine, to check the shared `renderers/base.py` change against every other model: ``` $ .venv/bin/python -m pytest tests/renderers/ -q 1 failed, 428 passed in 469.25s ``` The single failure is `test_hf.py::test_resolve_content_format_fallbacks[facebook/chameleon-7b-string]`, which fails identically on unpatched `main` — unrelated to this change. ### Reverse check Reverting only the changed source files and rerunning: | test | unpatched `main` | with this change | |---|---|---| | `test_models_text[False-...siglip-base-patch16-224]` | **FAIL**, cosine 0.5006 vs HF | PASS | | `test_models_text[True-...siglip-base-patch16-224]` | PASS | PASS | | `test_default_padding::test_model_default_is_applied` | **FAIL**, `assert 9 == 64` | PASS | | `test_default_padding::test_explicit_padding_is_not_overridden` | PASS | PASS | | `test_default_padding::test_multimodal_prompts_are_not_padded` | PASS | PASS | ### Lint `pre-commit run --files ` passes, including `mypy` for Python 3.10 and `mypy-3.12 --hook-stage manual`. ## Model evaluation Zero-shot classification through `LLM.embed` on `google/siglip-base-patch16-224`, two images from `vllm.assets.image` against three labels. Cosine similarity, argmax picks the label. Before (unpatched `main`): | image | \"a photo of a stop sign\" | \"a photo of a cherry blossom\" | \"a scanned tax invoice\" | picked | |---|---|---|---|---| | stop sign | −0.0245 | −0.0166 | −0.0422 | cherry blossom (wrong) | | cherry blossom | −0.0684 | −0.0348 | −0.0488 | cherry blossom | After: | image | \"a photo of a stop sign\" | \"a photo of a cherry blossom\" | \"a scanned tax invoice\" | picked | |---|---|---|---|---| | stop sign | **0.1567** | 0.0114 | −0.0974 | stop sign | | cherry blossom | −0.0089 | **0.1033** | −0.0942 | cherry blossom | The accuracy change (1/2 to 2/2) understates it: before the change every similarity is negative and near zero, so there is no signal at all and the one correct pick is accidental. SigLIP's `logit_scale` (~118) and `logit_bias` (~−12.7) put the 0.5 probability threshold at a cosine of about 0.107, which only the patched matching pairs clear. Image-image cosine is 0.5879 in both runs, so the vision tower is untouched and only the cross-modal alignment changed. ## Duplicate check Checked 2026-08-05, no overlapping work found: ```bash gh pr list --repo vllm-project/vllm --state open --search \"siglip\" gh issue list --repo vllm-project/vllm --state open --search \"siglip padding\" ``` Open PRs mentioning siglip are unrelated (MiniCPM-RobotTrack, a Gemma 4 fp16 overflow fix, PaddleOCR-VL CUDA graph support, DeepSeek-VL2). Open issues mentioning siglip are unrelated. \\#29794 added `tokenization_kwargs` to the offline API and is already merged; this PR covers the case where the caller cannot pass them. ## AI assistance This change was prepared with AI assistance. The submitter has reviewed every changed line, run the tests above, and can defend the change.", "url": "https://github.com/vllm-project/vllm/pull/51157", "createdAt": "2026-08-05T12:12:34Z", "updatedAt": "2026-08-13T03:58:03Z", "timestamp": "2026-08-13T03:58:03Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "bug", "multi-modality" ], "author": "Hert4", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Defer `tilelang` import through its import `from vllm.tilelang_utils import tilelang` and relaxed `has_tilelang`", "text": "## Purpose Fixes https://github.com/vllm-project/vllm/issues/51151 This PR introduces `vllm.tilelang_utils` and prevents direct `tilelang` imports. Avoid importing TileLang during ROCm module import, because importing TileLang can load **wrongful/bugged** TVM and HIP stub symbols into the global process scope before AITER loads its JIT modules. This changes `_tilelang_jit` so ROCm applies `tilelang.jit` lazily on first kernel call, while non-ROCm keeps eager JIT decoration. This PR is AI-assisted. Proper fixes in followups need to be: * a PR in `tilelang` or TVM to fix the wrongful symbol load. * a PR in vLLM so that `_has_module` does not import modules https://github.com/vllm-project/vllm/issues/51162 ## Test Plan - `python tools/pre_commit/check_forbidden_imports.py --test-regex` - `pytest tests/kernels/test_mhc_tilelang_jit.py -v` Added new test to verify that CUDA applies `tilelang.jit` eagerly, ROCm defers `tilelang.jit` until first call, and the ROCm wrapper caches the compiled kernel. ```python VLLM_ROCM_USE_AITER=1 HIP_VISIBLE_DEVICES=1 pytest tests/quantization/test_online.py::test_online_quantization[True-fp8_per_tensor-None-Fp8PerTensorOnlineLinearMethod-Fp8PerTensorOnlineMoEMethod] -s -vvvvv ``` that is failing on `main` following https://github.com/vllm-project/vllm/pull/50879, and which surfaces the issue https://github.com/vllm-project/vllm/issues/51151 ## Test Result `test_mhc_tilelang_jit.py` pass on MI350. `tests/quantization/test_online.py` as well on MI300 (previous failing case) `python tools/pre_commit/check_forbidden_imports.py --test-regex` passes as well.", "url": "https://github.com/vllm-project/vllm/pull/51159", "createdAt": "2026-08-05T12:25:31Z", "updatedAt": "2026-08-13T16:45:26Z", "timestamp": "2026-08-13T16:45:26Z", "metrics": { "reactions": 2, "comments": 26 }, "labels": [ "documentation", "rocm", "ready" ], "author": "fxmarty-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Enable Speculative Decoding for NVIDIA-Nemotron-Parse-2.0", "text": "co autored with: @DenisOvchinnikov93 [NVIDIA-Nemotron-Parse-2.0](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0) ships: > Auxiliary Prediction Head: One training-time decoder prediction head is preserved separately in auxiliary_prediction_heads.safetensors.extra for future multi-token prediction research. Standard generation uses the tied decoder input/output embeddings; the default model.safetensors, Transformers examples, and vLLM examples do not load this auxiliary head. This PR leverages the auxiliary head to enable Speculative Decoding for Nemotron-Parse. Backward compatibility for older Nemotron-Parse versions is preserved (standard single-token decoding is only available for versions < 2.0). This PR also switches tests to track regressions on the latest model with speculative-decoding enabled. To measure throughput with vs without speculative decoding, I used ParseBench table and text_content (1,005 pages). That census has long target sequences, so MTP gains show up clearly. On H100, speculative-decoded vs baseline median output-throughput uplift was: Concurrency | speculative-decoding throughput gain vs single-token-prediction -- | -- 1 | 45.65% 8 | 40.85% 32 | 39.60% ## Purpose Enable spec-dec for nemotron-parse 2.0 ## Test Plan ``` .venv/bin/python -m pytest \\ tests/models/multimodal/generation/test_nemotron_parse.py \\ -vv --tb=short ``` ## Test Result ``` ================== 1 passed, 32 warnings in 74.73s (0:01:14) =================== ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51168", "createdAt": "2026-08-05T14:27:22Z", "updatedAt": "2026-08-13T12:32:26Z", "timestamp": "2026-08-13T12:32:26Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "multi-modality", "nvidia", "mrv2" ], "author": "mwawrzos", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51182", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] [Build] Fix DeepGEMM SM90 paged mqa `prefix_sum` out of range", "text": "## Purpose Fix #51181. DeepGEMM SM90 paged mqa logit contains an out-of-index error which is easily hit in common device setup (DeepSeek V4 DSpark on H100 and common `max-num-seqs`). This PR prevents the out-of-index error (`CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS` on CUDA coredump) by building DeepGEMM from bugfixed branch (vllm-project/DeepGEMM#2). **TODO**: rollback the DeepGEMM repo to vllm-project after vllm-project/DeepGEMM#2 merged. ## Test Plan Run the reproducer in #51181 and check that CUDA graph capture runs successfully. ## Test Result I have checked the validity of the referred DeepGEMM fork branch by installing the source-built DeepGEMM wheel atop of latest(beca88e) main CI image and run the reproducer in #51181. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51182", "createdAt": "2026-08-05T17:22:31Z", "updatedAt": "2026-08-13T13:28:00Z", "timestamp": "2026-08-13T13:28:00Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "cjackal", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51207", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bolins/shm tensor arena", "text": " ## Purpose `MessageQueue.enqueue` (`shm_broadcast.py`) already routes CPU tensors out-of-band via `_reduce_tensor` (protocol-5 `PickleBuffer`, #48442), which removed the dominant cost of the old in-band path — copying tensor bytes into the pickle stream. Two costs remain for a large multimodal `pixel_values` tensor on a TP=N worker: 1. the out-of-band buffer is published **once per node-local reader** (N transport copies for TP=N), and 2. each reader's H2D copies from a **pageable** ZMQ frame (pageable staging + first-touch faults). This PR adds an opt-in **slotted shared-memory tensor arena** that layers *on top of* `_reduce_tensor`: the writer does **one** memcpy into a free slot, every reader takes a **zero-copy view** of the *same* slot (no per-reader transport copy), and the mapping is `cudaHostRegister`-pinned so the H2D is a true DMA. Large contiguous CPU tensors are diverted by `_ArenaPickler.reducer_override`; anything it declines (too small, non-contiguous, or arena exhausted) **falls through to `_reduce_tensor` unchanged**. Net for a ~200 MB image at TP=8: **1 memcpy + 8 pinned DMAs** instead of **8 transport copies + 8 pageable stagings**. The gain scales with TP degree and tensor size, and is zero at TP=1 (no multiproc broadcast) — both now confirmed empirically below. Design doc: `docs/design/shm_tensor_arena.md`. **Safety / scope:** controlled by the `--enable-shm-tensor-arena` / `--no-enable-shm-tensor-arena` CLI flag (`ParallelConfig.enable_shm_tensor_arena`, default on — happy to flip to default-off if preferred); activates only when all queue readers are node-local; the writer **never blocks** (no free/oversized slot → fall back to `_reduce_tensor`, deadlock structurally impossible); slot reuse is gated on an **H2D-completion CUDA event**, so the writer can't overwrite a slot whose async DMA is still in flight — correct under `--async-scheduling`. ## Test Plan A/B against **current `main`**: the baseline arm is the unmodified `vllm/vllm-openai:nightly` image (`nightly-65b7662d3f…`, includes #48442); the arena arm is the same image plus this PR's four files. A third arm runs this PR's code with `--no-enable-shm-tensor-arena` as a disable-control (must be behaviorally identical to base; also bounds run-to-run noise). Two setups, so the TP axis and the model-realism axis are both covered: 1. **TP-scaling sweep** — Qwen2.5-VL-3B, TP ∈ {1, 2, 4} (ViT MLP dim caps this family at TP≤4), image sizes 512²–3072², 150 prompts @ 2 req/s. 2. **Production-scale VLM** — Qwen3-VL-235B-A22B (ModelOpt NVFP4) on 8×B300, TP ∈ {4, 8}, image sizes 1024²/2048²/3072², 120 prompts @ 1 req/s. Common method: `vllm bench serve --dataset-name random-mm` (openai-chat backend), one image per request at a fixed size per cell, poisson arrivals at a **non-saturated** rate, 20 warmups (excludes the one-time arena pinning), seed 42, **2 repeats per cell**. Built-in negative controls: TP=1 (arena structurally inactive) and 512² images (~3 MB `pixel_values`, below the 8 MB divert threshold).
Repro commands Server (one per arm × TP; arena arms = same image + this PR's files): vllm serve $MODEL --tensor-parallel-size $TP --max-model-len 32768 \\ --mm-processor-cache-gb 0 [--enable-shm-tensor-arena | --no-enable-shm-tensor-arena] Load (per image size HW, per repeat): vllm bench serve --model $MODEL --backend openai-chat --endpoint /v1/chat/completions \\ --dataset-name random-mm --random-mm-bucket-config \"{($HW, $HW, 1): 1.0}\" \\ --random-mm-base-items-per-request 1 --random-mm-limit-mm-per-prompt '{\"image\": 1}' \\ --random-input-len 128 --random-output-len 64 --ignore-eos \\ --num-warmups 20 --num-prompts 150 --request-rate 2 --seed 42 \\ --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 --save-result
Still outstanding: port the standalone correctness harness (21 checks: byte-exact round-trip incl. bf16, zero-copy sharing, slot lifecycle with per-reader release, event-gated reuse, exhaustion/oversize/non-contiguous fallbacks) into `tests/distributed/test_shm_broadcast.py` alongside the #48442 tests. ## Test Result **Qwen3-VL-235B (NVFP4) on 8×B300 — E2E p99, base → arena (mean of 2 repeats):** | image (`pixel_values` size) | TP=4 | TP=8 | |---|---:|---:| | 1024² (~23 MB) | 1021 → 995 ms (−2.6%) | 1025 → 999 ms (−2.5%) | | 2048² (~96 MB) | 1584 → 1298 ms (**−18.1%**) | 1807 → 1363 ms (**−24.6%**) | | 3072² (~213 MB) | 3340 → 2771 ms (**−17.0%**) | 4132 → 3083 ms (**−25.4%**) | TTFT p99 moves the same direction (TP=8: −21.5% at 2048², −11.9% at 3072²), and at these payload sizes TTFT p50 improves too (TP=8/3072²: 1185 → 948 ms). Request throughput is identical in every cell (rate-limited by design). **TP-scaling sweep (Qwen2.5-VL-3B — separate setup from the table above):** TP=1 and 512² images are neutral (negative controls behave as designed — the arena never activates); gains appear only at TP≥2 with ≥8 MB tensors and grow with TP and size. On this small model the relative effect is larger than the 235B numbers above — up to −34% E2E p99 at 3072²/TP=4 (3020 → 1992 ms) — because the same transport cost is a much bigger fraction of a 3B's per-request time, and at 2 req/s its large-image cells carry queueing that the stall removal also drains. The 235B table above is the more conservative, production-representative measurement. The disable-control arm on this setup bounds noise at roughly ±5% per cell. **Statistical honesty:** with n=2 repeats, individual p99 cells are noisy (the 235B control shows single-cell swings up to ~±17%). The claim rests on the pattern, not any one cell: the arena is ahead in **all 12** TTFT/E2E comparisons across both models, monotonically in TP degree and tensor size — the shape the mechanism predicts (per-reader transport + pageable staging replaced by one memcpy + pinned DMA), and the E2E-tail bias matches head-of-line blocking on co-scheduled decodes. **Summary:** neutral outside the large-image + TP≥2 regime; **~18–25% p99 reduction** inside it on a production-scale VLM. **Process note:** the disable-control caught a bug in an earlier revision of this PR — `--no-enable-shm-tensor-arena` was silently ignored (the gate read `get_current_vllm_config()` outside its context and fell back to the default). Fixed by threading the flag explicitly through the executor; the control was re-validated inert (0 arena activations) before the numbers above.
Historical: original measurement vs the pre-#48442 in-band baseline Same-seed interactive multimodal workload (uncapped images, TP=4 ×2 workers on one 8-GPU node, ~1.2k aligned requests): TTFT p50 93→89, p90 401→241, p99 1321→862, max 2139→1375 ms; requests >1.5 s: 9→0. These numbers predate #48442 and therefore overstate the delta vs current main; kept only as provenance for the original problem analysis.
---
Essential Elements Checklist - [x] Purpose of the PR - [x] Test plan — executed (repro commands above); unit-test port to `tests/distributed/` still outstanding - [x] Test results — A/B vs current `main` on two models × TP 1–8, with disable-control - [x] Documentation — `docs/design/shm_tensor_arena.md`
", "url": "https://github.com/vllm-project/vllm/pull/51207", "createdAt": "2026-08-05T23:30:59Z", "updatedAt": "2026-08-12T18:29:04Z", "timestamp": "2026-08-12T18:29:04Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "documentation", "nvidia" ], "author": "BolinSNLHM", "state": "open", "assignees": [ "njhill", "Isotr0py", "BolinSNLHM" ], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51208", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][AMD][Installation] add LMCache kv-connector installation and runtime packages to docker image", "text": "## Purpose Ship the LMCache KV connector in the ROCm image so it works out of the box. Note: vLLM's CUDA `docker/Dockerfile` installs LMCache when `INSTALL_KV_CONNECTORS=true`, and the release pipeline sets that for every published `vllm/vllm-openai` tag. docker/Dockerfile.rocm never referenced that arg, so vllm/vllm-openai-rocm ships without LMCache and users must build and install it themselves. This adds it so the ROCm release pipeline can enable it the same way. ### What this does - Adds a `build_lmcache` stage that compiles LMCache for HIP from source (`BUILD_WITH_HIP=1`, `CXX=hipcc`, `TORCH_DONT_CHECK_COMPILER_ABI=1`). A source build is required: LMCache publishes **no ROCm sdist on PyPI** (`>=0.4` is wheels-only), so `--no-binary` cannot force a HIP build. - Pinned to the v0.5.3 commit (140819c9d57a); LMCACHE_ROCM_ARCH defaults to gfx942;gfx950, covering MI300X/MI325X and MI350X/MI355X. - Installs the wheel with `--no-deps` plus four runtime packages. - `sortedcontainers` — hard requirement; without it `import lmcache.integration.vllm.lmcache_mp_connector` fails outright. - `opentelemetry-exporter-prometheus` — `lmcache server` calls `init_observability()` unconditionally, which imports `PrometheusMetricReader` (`mp_observability/otel_init.py:96`). Omitting it builds fine but the server dies at startup. - `cupy-rocm-7-0` — GPU cache-registration path. - `aiofile` — local-disk storage tier. - Gates everything behind build args, defaulting to off. ### Build-arg selection `INSTALL_KV_CONNECTORS` is an umbrella mirroring the CUDA Dockerfile's arg; each connector also has its own arg defaulting to it, so future connectors (like Mooncake) compose without renaming anything. | Build args | Result | | --- | --- | | *(none)* | nothing installed (default) | | `INSTALL_KV_CONNECTORS=true` | all connectors — CUDA release-pipeline parity | | `INSTALL_LMCACHE=true` | LMCache only | | `INSTALL_KV_CONNECTORS=true INSTALL_LMCACHE=false` | all except LMCache | Booleans rather than a list because the per-connector build stages are selected via `FROM _${ARG}`, which needs the value in the stage name. With a list the hipcc compile would run on every build and be discarded. ## Test Plan ```bash # (1) build docker buildx build -f docker/Dockerfile.rocm --target vllm-openai \\ --build-arg INSTALL_KV_CONNECTORS=true \\ -t vllm-rocm-lmcache:oob --load . # (2) start the LMCache server from the image, installing nothing docker run --rm --network host --ipc host \\ --device /dev/kfd --device /dev/dri --group-add video \\ -e LMCACHE_L1_SIZE_GB=906 --entrypoint bash vllm-rocm-lmcache:oob -c ' lmcache server --host 127.0.0.1 --port 5555 \\ --http-host 127.0.0.1 --http-port 8080 \\ --l1-size-gb \"$LMCACHE_L1_SIZE_GB\" --l1-init-size-gb 20 \\ --l1-read-ttl-seconds 7200 --chunk-size 1536 --max-workers 8 \\ --eviction-trigger-watermark 0.85 --eviction-ratio 0.10 \\ --eviction-policy LRU' ``` Also verified: build-arg selection matrix, and that a `false` build never runs the hipcc compile. ## Test Result Built and verified on **8x MI355X (gfx950), ROCm 7.2.3, torch 2.11**. **(1) Build** — succeeds. The in-Dockerfile smoke test passes, printing `lmcache 0.5.3`. Built wheel properties: | Check | Result | | --- | --- | | `DT_NEEDED` | `libamdhip64.so.7`, libtorch/c10 — **no CUDA libs** | | GPU archs in `c_ops` | `gfx942`, `gfx950` | | Python ABI | `cp312` | **(2) LMCache server starts out of the box**, nothing installed at runtime: ``` zmq 5555 -> accepting http 8080 -> HTTP 200 ``` Startup log shows shm capacity checked against the L1 request, L1 + L2 eviction controllers started, SessionManager cleanup thread, DeviceHostFuncDispatcher, mp-worker-reaper, and an 8-slot affinity pool matching `--max-workers 8`. **(3) No dependency downgrades** — the reason for `--no-deps`: | Package | Before | After | | --- | --- | --- | | numpy | 2.3.5 | 2.3.5 | | transformers | 5.14.1 | 5.14.1 | | opentelemetry-api / sdk | 1.44.0 | 1.44.0 | | opentelemetry-semantic-conventions | 0.65b0 | 0.65b0 | Added: `lmcache 0.5.3`, `sortedcontainers 2.4.0`, `aiofile 3.12.3`, `cupy-rocm-7-0 14.1.1`, `opentelemetry-exporter-prometheus 0.65b0`. **(4) Connectors register**: `['LMCacheConnectorV1', 'LMCacheMPConnector']`. **(5) Build-arg matrix** — all four rows behave as documented; with `INSTALL_LMCACHE=false` the `build_lmcache` stage is pruned entirely (verified by building `--target export_lmcache`: empty output, no clone, no compile). (6) Image size: 42.00 → 42.92 GB, +921 MB (+2.19%) — ~366 MB of that is CuPy, required by the GPU cache-registration path; the compiled extension is <8 MB. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51208", "createdAt": "2026-08-05T23:46:52Z", "updatedAt": "2026-08-13T15:58:15Z", "timestamp": "2026-08-13T15:58:15Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "rocm", "ready", "ci/build", "kv-connector" ], "author": "hongxiayang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51209", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] IndexCache for DeepSeek-V4 (validated on V4-Flash-0731, including DSpark on to accelerate on top of", "text": "## Summary: This work is built to support Dspark + IndexerCache to achieve best deepseek v4 flash serving performance. ## Purpose Adds DSA IndexCache support to DeepSeek-V4, so C4A layers marked shared reuse the top-k the previous C4A layer left in the shared `topk_indices_buffer` instead of running their own indexer. Same config surface as the DeepSeek-V3.2 feature (`use_index_cache` / `index_topk_freq` / `index_topk_pattern`, see [docs/features/index_cache.md](https://github.com/vllm-project/vllm/blob/main/docs/features/index_cache.md)); addresses #45350. The indexer dominates decode KV traffic on this model at long context: per decode token per C4A layer it reads `(S/4) x 132 B` of its own K cache, while the main sparse MLA reads only `index_topk x 576 B`. Across DeepSeek-V4-Flash-0731's 21 C4A / 20 C128A / 44 SWA caches that is ~65% of decode KV traffic at 32K and ~81% at 128K, and the indexer is TP-replicated so every rank pays it in full. ### Relationship to #49085 #49085 implements the same feature for Hopper and is the earlier work; its authors (@wangyicong52, @zzz3bbb3) got there first on the two non-obvious points — per-PP-rank handling, and keeping skipped indexers' KV-cache specs registered while parking their linear weights on the meta device. Two behavioural differences worth a maintainer decision: 1. This PR gates on `use_index_cache`, matching the documented V3.2 surface. #49085 activates on `index_topk_freq > 1` alone, which would make `--hf-overrides '{\"index_topk_freq\": 4}'` silently enable on V4 while silently doing nothing on V3.2. 2. This PR uses `c4a_idx % index_topk_freq`; #49085 keeps V3.2's `index_skip_topk_offset` (default 2), so its first two C4A layers always compute. Theirs is more faithful to V3.2 and I am happy to switch. ## Design notes Only C4A layers (`compress_ratio == 4`) carry an indexer in V4, so **the frequency and pattern are indexed over the C4A layers, not over all layers**. Applying the V3.2 absolute-`layer_id` formula to DeepSeek-V4-Flash marks *every* C4A layer as shared (they are all even), so nothing populates the buffer and every layer reads stale contents — silently, with no error. A regression test pins this. Three further constraints, each covered by a test: - `topk_indices_buffer` is rank-local, so the first C4A layer **of each pipeline stage** computes its own top-k (derived from the same `get_pp_indices` split `make_layers` uses). - `compress_ratios` can be longer than `num_hidden_layers` (44 entries on V4-Flash, 46 on V4-Flash-0731, for the MTP/draft slots), so selection slices to `num_hidden_layers`. - A pattern is one `F`/`S` character per C4A layer — 21 on V4-Flash-0731, not 43 — and may not start with `S`. Wrong-length patterns raise rather than silently applying a prefix. Skipped layers keep their indexer allocated and simply never run it, so the change is confined to `attention.py` and needs no KV-cache-grouping or weight-loading changes. Memory savings are left to a follow-up. ## Test Plan ```bash pytest tests/models/test_deepseek_v4_index_cache.py -v ``` E2E hardware: single node, 8xH100 80GB, TP=8, upstream vLLM 0.26.0 with this patch, `--kv-cache-dtype fp8_ds_mla`, `deepseek-ai/DeepSeek-V4-Flash-0731`. ```bash # long context / frequency sweep vllm bench latency --model <0731> --tokenizer-mode deepseek_v4 -tp 8 \\ --kv-cache-dtype fp8_ds_mla --max-model-len 33792 \\ --input-len 32768 --output-len 128 --batch-size 2 \\ --hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": N}' # quality: GSM8K 200q 5-shot, MMLU 500q 5-shot, greedy # throughput: short prompt + 512 output tokens at fixed concurrency ``` ## Test Result 14/14 unit tests pass. ### Frequency sensitivity 32768-token input x batch 2, and MMLU 500 questions 5-shot. All arms are the same build with only `index_topk_freq` varying, so `freq=1`/off is a true no-skip control (confirmed by the log line firing only on freq>1 arms): | freq | indexers run | 32K latency | Δ | MMLU | |---|---|---|---|---| | off / 1 | 21/21 | 3.004 s | — | 85.4% | | 2 | 11/21 | 2.890 s | -3.8% | 85.6% | | 3 | 7/21 | 2.842 s | -5.4% | 85.0% | | 4 | 6/21 | 2.831 s | -5.8% | 84.8% | | 6 | 4/21 | 2.808 s | -6.5% | 85.0% | The latency curve saturates quickly — freq=2 captures most of the win, and skipping three times as many indexers (freq=6) only reaches -6.5%. MMLU spans 84.8-85.6%, within the ~1.6pp binomial noise at n=500. vLLM 0.25.1 and the older `DeepSeek-V4-Flash` checkpoint reproduce the freq=2/3 numbers within noise. ### Quality and throughput matrix (GSM8K, 200q 5-shot, best of 3 timed passes) | cell | DSpark | IndexCache | accuracy | throughput | |---|---|---|---|---| | c4 | off | off | 96.0%, 96.5% | 4087, 4525 tok/s | | c2 | off | **on** | 96.0%, 96.0% | 4322, 4081 tok/s | | c1 | **on** | off | 96.0%, 97.0% | 3449, 3742 tok/s | | c3 | **on** | **on** | 97.0% | 3873 tok/s | IndexCache changes neither accuracy nor throughput here. Note this shape is ~91% prefill tokens (180K prefill vs 17K decode) and saturates the GPU, so it is not informative about speculative decoding; the concurrency sweep below is. ### Throughput vs concurrency (short prompt, 512 output tokens) | concurrency | c4 baseline | c1 DSpark | c3 DSpark + IndexCache | |---|---|---|---| | 4 | 560 tok/s | **1419** | 1201 | | 16 | 1670 tok/s | 3289 | **3419** | | 64 | 4910 tok/s | 6975 | **7187** | The two features are complementary and stack: at the concurrencies that matter for serving, DSpark + IndexCache is the fastest configuration. Draft acceptance is 63-69% (mean accepted length 4.2-4.5 of 5) with IndexCache both on and off, so IndexCache does not measurably degrade the draft. ### Long-context correctness A needle-in-a-haystack probe (18,930-token prompt, needle at depth 0.15 / 0.5 / 0.85, greedy) retrieves correctly with IndexCache off, at freq=2, and at freq=3. ### DSpark interaction DeepSeek-V4's DSpark draft builds its decoder layers with `topk_indices_buffer=None` and layer ids >= `num_hidden_layers`, so draft layers resolve to `compress_ratio == 1` and never construct an indexer; the two features are structurally orthogonal. Enabling both is exercised end to end at every concurrency above. ## Notes Not covered, stated rather than implied: no Blackwell run (#49085 hard-rejects non-SM90 for this feature; this PR does not gate, which may be wrong); no PP>1 e2e run (the per-stage rule is unit-tested only); Meanwhile, if B200 number testing needed, I will provide later follow up test in NVFP4. AI Assistance are used for this development. --- ## Full measurement set (AIPerf) Everything below is `aiperf profile` against `vllm serve` on the same node: single 8xH100 80GB, TP=8, upstream vLLM 0.26.0 + this patch, `--kv-cache-dtype fp8_ds_mla`, `deepseek-ai/DeepSeek-V4-Flash-0731`, AIPerf 0.12.0. Output length is pinned with `ignore_eos:true` + `min_tokens` (without it the model stops early and throughput deltas measure differing OSL, not speed), `--request-count` = 16x concurrency, `--warmup-request-count` = concurrency. `ISL x concurrency` is chosen to fit the ~108K-token KV pool at each point; beyond that requests queue and the run becomes a scheduling test rather than a throughput test. ### Controls Two arms exist purely to bound what a real effect must beat: - **`freq=1`** enables IndexCache but skips nothing, so it must land on top of `off`. It does, everywhere. - **`off` re-run** gives the run-to-run noise floor. | control | throughput delta vs `off` | |---|---| | `freq=1`, all five workloads | -0.1% .. +0.1% | | `off` re-run, 3 workloads | -0.1% .. +0.0% | TTFT is the exception: identical configs vary by up to 5.8% on the 4K point, so no TTFT claim below ~6% is meaningful there. ### Output token throughput, delta vs `off` | workload | ISL/OSL | conc | freq=1 | freq=2 | freq=3 | freq=4 | |---|---|---|---|---|---|---| | summarisation (prefill-heavy) | 32K / 16 | 3 | -0.0% | +3.3% | +5.2% | **+5.6%** | | long chat | 32K / 256 | 3 | +0.1% | +3.5% | +4.9% | **+5.2%** | | mid chat | 16K / 256 | 6 | +0.0% | +2.8% | +3.9% | **+4.2%** | | RAG | 7K / 172 | 12 | +0.1% | — | **+3.2%** | — | | short chat | 4K / 256 | 24 | -0.1% | +2.0% | +2.8% | **+3.1%** | Absolute baselines: 13.63, 146.34, 285.82, 466.5, 1057.42 out tok/s respectively. ### Latency, delta vs `off` (freq=4; freq=3 for RAG) | workload | e2e | TTFT | ITL | |---|---|---|---| | 32K / 16 | -5.3% | -5.1% | — | | 32K / 256 | -5.0% | -5.1% | -4.8% | | 16K / 256 | -4.0% | -4.6% | -3.7% | | 7K / 172 | -3.1% | -2.7% | — | | 4K / 256 | -3.0% | -1.9% | -3.2% | ### Accuracy per frequency | freq | GSM8K (200q, 5-shot) | MMLU (500q, 5-shot) | |---|---|---| | off | 96.0% | 85.4% | | 1 (control) | 96.0% | 85.0% | | 2 | 95.5% | 85.4% | | 3 | 96.0% | **86.2%** | | 4 | **96.5%** | 85.0% | No trend: the best GSM8K score is at freq=4 and the best MMLU at freq=3, and the `freq=1` control differs from `off` by 0.4pp on MMLU while being semantically identical to it. All deltas sit inside that noise. **This is a no-harm result, not proof of quality preservation.** Both benchmarks' prompts (~900 and ~500 tokens, i.e. ~225 and ~125 compressed) are below `index_topk=512`, so every layer selects the whole compressed sequence and IndexCache is mathematically a no-op. Long-context quality evidence is limited to the needle probe (retrieved at all three depths with IndexCache off, freq=2 and freq=3); a RULER/LongBench run at 32K is still owed. ### Where the time actually goes A torch-profiler run (eager, prefix caching off, conc 12 at 8K) attributes GPU kernel time as: | bucket | freq=0 | freq=6 | |---|---|---| | TP all-reduce + other | 6176 ms | 5254 ms | | MoE | 1003 ms | 980 ms | | norm/rope | 395 ms | 394 ms | | **indexer** | **91.7 ms (1.2%)** | **30.4 ms (0.4%)** | | compressor | 43 ms | 42 ms | The indexer is 1.2% of kernel time; the single largest kernel is `cross_device_reduce_1stage` (TP all-reduce) at ~69%. Removing 81% of the indexer work saves 61 ms directly, yet wall-clock improves several percent — the rest comes from ranks spending less time waiting inside the all-reduce. This configuration is communication-bound, which is also why the gain saturates between freq=3 and freq=4 (+0.3-0.4pp) and why a faster indexer kernel would not help much. ### Interaction with DSpark speculative decoding Decode-heavy (short prompt, 512 output tokens, `ignore_eos`), output tok/s: | concurrency | baseline | DSpark | DSpark + IndexCache | |---|---|---|---| | 4 | 560 | **1419** | 1201 | | 16 | 1670 | 3289 | **3419** | | 64 | 4910 | 6975 | **7187** | At serving concurrencies the two stack and DSpark + IndexCache is the fastest configuration. Draft acceptance is 63-69% (mean accepted length 4.2-4.5 of 5) with IndexCache both on and off, so IndexCache does not measurably degrade the draft. ### Honest summary of magnitude **+3% to +5.6%** depending on shape and frequency, saturating at freq=3-4, free at inference time, with no measurable accuracy cost in the regimes tested. That is smaller than the 7-8% reported for vllm-ascend in #45350. On this hardware the gap is explained by the profile above (communication-bound, indexer at 1.2%), by the KV pool capping `ISL x concurrency` so the indexer's share cannot grow, and by Hopper reading an FP8 indexer cache at 132 B/token where `use_fp4_indexer_cache` would halve it. ### Measurement caveats - The e2e numbers were produced by the 0.26.0-based build; upstream restructured `attention_impl` in #51430 after 0.26.0, so the rebased branch carries the same gates at the new call sites but has not itself been re-run on GPU. - One RAG data point initially showed +9.8%; its `freq=1` control read +6.6%, which flagged the baseline as un-warmed. A repeat with two independent baseline arms put the true figure at +3.2%. The table above uses the corrected value. - Single node only: no Blackwell run, and no PP>1 run (the per-stage rule is unit-tested only).", "url": "https://github.com/vllm-project/vllm/pull/51209", "createdAt": "2026-08-05T23:47:09Z", "updatedAt": "2026-08-12T18:07:48Z", "timestamp": "2026-08-12T18:07:48Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "deepseek" ], "author": "DiegoCao", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51218", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Report FULL_ATTENTION for uniform-base UniformTypeKVCacheSpecs groups instead of UNKNOWN", "text": "## Problem `get_kv_cache_spec_kind()` returns `KVCacheSpecKind.UNKNOWN` for a `UniformTypeKVCacheSpecs` whose members have more than one *inner* kind: ```python if isinstance(kv_cache_spec, UniformTypeKVCacheSpecs): inner_kinds = {get_kv_cache_spec_kind(spec) for spec in kv_cache_spec.kv_cache_specs.values()} if len(inner_kinds) == 1: return next(iter(inner_kinds)) return KVCacheSpecKind.UNKNOWN # <-- here ``` But such a group is **only ever formed when all members share one registered `uniform_type_base_spec`** — that is the merge condition itself (`UniformTypeKVCacheSpecs.is_uniform_type` → `KVCacheSpec.is_uniform_with_collection`). So `UNKNOWN` discards information the merge already established. ## Why it matters KV-cache events are tagged with this kind (`v1/core/kv_cache_manager.py`), and consumers that filter on it drop `UNKNOWN` outright. Affected models therefore lose **all** cache-aware routing credit — a KV-aware router scores them as though nothing were cached. Concrete case: a model that registers a main `FullAttentionSpec` K/V cache **and** a differently-shaped `MLAAttentionSpec` index cache. Both register `uniform_type_base_spec=FullAttentionSpec` (`v1/core/single_type_kv_cache_manager.py`), so they merge into a single group — which then reports `UNKNOWN`: ```python >>> get_kv_cache_spec_kind(UniformTypeKVCacheSpecs({\"main\": FullAttentionSpec(...), ... \"idx\": MLAAttentionSpec(...)})) KVCacheSpecKind.UNKNOWN >>> KVCacheSpecRegistry.get_uniform_type_base_spec(FullAttentionSpec(...)) >>> KVCacheSpecRegistry.get_uniform_type_base_spec(MLAAttentionSpec(...)) # same base -> the group IS uniform ``` ## Fix Fall back to the registered base spec, upgrading to `FULL_ATTENTION` only when **every** member resolves to `FullAttentionSpec`. Sliding-window, Mamba and other shapes keep `UNKNOWN` exactly as before, so this is deliberately conservative. ## Validation Verified on a live 2-worker deployment with CPU KV offloading enabled, reading the router's own `router_kv_hit_rate` metric (\"Predicted KV cache hit rate at routing time\") for one 78,163-token prefix: | step | before | after | |---|---:|---:| | cold (nobody has it) | 0.2 % | 0.0 % | | prefix warm in GPU | 99.8 % | **99.8 %** (no regression) | | prefix evicted to host, **served from host** | **0.2 %** | **74.9 %** | In the third row the worker demonstrably served the prefix from host RAM (77,952 external-prefix-cache hits, 1,066 ms vs 7,863 ms cold), yet before the fix the router scored it identically to \"nobody has it\". 74.9 % ≈ **0.75 × 99.8 %**, i.e. exactly the router's host-cache-hit weight applied to a full-prefix match — the credit path engaging at its configured multiplier. Before the fix, the group's `UNKNOWN` kind also meant that enabling self-describing KV events would stamp `UNKNOWN` on **GPU-tier** events, which the same filter drops; GPU-tier routing survived only because the field was absent from the wire. This change makes both tiers report a real kind. ## Notes - Behaviour is unchanged for every group that already returned a concrete kind. - No new imports at module scope (the registry import is local to avoid a cycle with `kv_cache_spec_registry`, which imports from this module). - Happy to add a unit test asserting `FULL_ATTENTION` for a mixed `FullAttentionSpec` + `MLAAttentionSpec` group if you'd like one — pointers to the preferred location welcome.", "url": "https://github.com/vllm-project/vllm/pull/51218", "createdAt": "2026-08-06T02:30:49Z", "updatedAt": "2026-08-13T01:25:24Z", "timestamp": "2026-08-13T01:25:24Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "bug", "ready" ], "author": "yifjiang", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51251", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Configure custom encoder cache managers from VllmConfig", "text": "## Purpose Expose custom encoder cache manager configuration through `VllmConfig` for both online and offline inference. Custom encoder cache managers may require policy-specific parameters in addition to the encoder cache size. This PR provides a generic configuration path while preserving compatibility with existing built-in and constructor-only cache managers. ## Changes This PR: - Adds an opaque `manager_config` field to `EncoderCacheManagerConfig`. Its contents are interpreted by the selected custom cache manager. - Exposes `ec_manager_config` through `EngineArgs`. - Adds the `--ec-manager-config` CLI option for online serving. - Supports the same configuration through `LLM(ec_manager_config=...)` for offline inference. - Resolves custom encoder cache managers using fully qualified class names. - Allows custom managers that need the complete `VllmConfig` to optionally implement: ```python @classmethod def from_vllm_config( cls, *, cache_size: int, vllm_config: VllmConfig, ): ... ``` - Preserves compatibility with constructor-only custom managers: ```python manager_cls(cache_size=cache_size) ``` - Keeps the existing construction paths of built-in `EncoderCacheManager` and `EncoderDecoderCacheManager` unchanged. A custom manager is not required to inherit from `EncoderCacheManager` or implement `from_vllm_config`, as long as it satisfies the interfaces consumed by the scheduler and model runner. ## Relationship to #48218 This PR is a follow-up to #48218, which was merged as commit `833483f`. #48218 introduced the generic encoder cache manager extension foundation, including: - selecting a custom encoder cache manager by fully qualified class name; - passing `EncoderCacheManagerMetadata` from the scheduler to workers; - overridable model-runner hooks for encoder cache lookup, storage, eviction, and request cleanup. However, custom managers were still constructed using only `cache_size`, and the encoder cache manager configuration was not consistently exposed through online and offline `EngineArgs`. This PR completes the configuration path without changing the lifecycle hooks introduced by #48218. Hardware-specific cache policies remain implemented in their corresponding platform plugins. ## Usage ### Online inference ```bash vllm serve MODEL \\ --ec-manager-config '{ \"encoder_cache_manager_cls\": \"vllm_ascend.ec_manager.score_ec_manager.ScoreEncoderCacheManager\", \"manager_config\": { \"cpu_cache_slots\": 100000, \"max_clock\": 15, \"clock_decay_every\": 64, \"watermark\": 0.2, \"promote_percentile\": 0.2 } }' ``` ### Offline inference ```python from vllm import LLM llm = LLM( model=\"MODEL\", ec_manager_config={ \"encoder_cache_manager_cls\": ( \"vllm_ascend.ec_manager.score_ec_manager.\" \"ScoreEncoderCacheManager\" ), \"manager_config\": { \"cpu_cache_slots\": 100000, \"max_clock\": 15, \"clock_decay_every\": 64, \"watermark\": 0.2, \"promote_percentile\": 0.2, }, }, ) ``` `manager_config` is intentionally opaque to vLLM. Validation and interpretation of manager-specific fields are owned by the selected custom manager. Specifying a non-empty `manager_config` without `encoder_cache_manager_cls` raises a configuration error.", "url": "https://github.com/vllm-project/vllm/pull/51251", "createdAt": "2026-08-06T10:36:50Z", "updatedAt": "2026-08-13T08:08:01Z", "timestamp": "2026-08-13T08:08:01Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "ready" ], "author": "hotTea123", "state": "closed", "assignees": [ "Isotr0py" ] }, { "id": "github:vllm-project/vllm:pull_request:51256", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Reserve the bonus query slot in DFlash scheduling budget", "text": "## Purpose The existing generic parallel-drafting calculation only reserves K - 1 slots for DFlash, leaving the scheduling budget short by one slot per request. For example, with: max_num_batched_tokens = 2048 max_num_seqs = 256 num_speculative_tokens = 8 the previous calculation allowed: max_num_scheduled_tokens = 2048 - 7 * 256 = 256 However, a full DFlash batch may require: 256 * (8 + 1) = 2304 query tokens, exceeding max_num_batched_tokens. DFlash has K + 1 query tokens per request, so its net drafting expansion is K slots rather than the generic parallel-drafting value of K - 1.", "url": "https://github.com/vllm-project/vllm/pull/51256", "createdAt": "2026-08-06T11:25:35Z", "updatedAt": "2026-08-13T03:50:24Z", "timestamp": "2026-08-13T03:50:24Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "bug", "ready" ], "author": "HF-001", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51274", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Kimi-K3] Add opt-in gfx942 MXFP4-to-int4 conversion", "text": "## Summary Add an explicit `int4_per_group_32` MoE override for Kimi-K3 on gfx942. When selected, the MXFP4 expert weights are requantized once at load time and served through AITER's replacement BF16 x packed-int4 FlyDSL path. The conversion is lossy and never happens by architecture alone. The default MXFP4 path is unchanged. ## Why gfx942 does not support the scaled FP4 conversion instructions used by the native MXFP4 path. This opt-in path pays the conversion cost once during model loading and then uses the replacement packed-int4 kernels from [ROCm/aiter#4646](https://github.com/ROCm/aiter/pull/4646). ## Changes - Register `int4_per_group_32` as an online MoE weight quantization key. - Require the explicit `--quantization-config.moe.weight int4_per_group_32` override on gfx942. - Require the replacement A16W4/int4 AITER entry point and fail closed unless it supports SiTUv2. - Convert eight experts at a time and release each source tensor before converting the next one, bounding transient memory during load. - Log a warning that the selected conversion is lossy. ## Validation Unit tests: ```text tests/quantization/test_quantization_config_args.py: 15 passed tests/models/kimi_k3/test_gfx942_int4.py: 5 passed ``` Strict AITER comparison against the torch reference passed for the production EP shape: ```text model_dim=3584 intermediate_dim=3072 local_experts=112 topk=15 tokens=1,4,7,16,32 activation=SiTUv2 beta=4.0 linear_beta=25.0 ``` Target-only integration on 8x MI325X with TP8, expert parallelism, Triton MLA, and full plus piecewise graph capture: - Graph capture completed successfully. - Three deterministic chat sanity checks returned correct answers. - Full five-shot GSM8K scored 0.960576 strict and flexible on all 1,319 questions, with 0.00536 standard error. - Three 1K input / 128 output serving sweeps at concurrency 1 and 16 completed 510/510 requests without server errors. The previous long-context results used the implementation superseded by ROCm/aiter#4646, so long-context validation against the replacement AITER release remains pending. ## Dependency and scope This remains a draft until ROCm/aiter#4646 is merged and available in an AITER release used by vLLM. The conversion is an explicit opt-in for deployments that accept a lossy one-time weight conversion. It does not change the default checkpoint path. ## Test plan - [x] Quantization key and opt-in selection unit tests. - [x] Strict production-shape comparison against the torch reference. - [x] Full 1,319-question GSM8K. - [x] Target-only graph capture and serving sanity. - [ ] Long-context and DSpark validation against the AITER release containing ROCm/aiter#4646. - [ ] ROCm CI on the rebased draft.", "url": "https://github.com/vllm-project/vllm/pull/51274", "createdAt": "2026-08-06T15:09:53Z", "updatedAt": "2026-08-12T20:42:42Z", "timestamp": "2026-08-12T20:42:42Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "rocm", "quantization", "kimi", "k3" ], "author": "maeehart", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51280", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Solidify entrypoint LLM lifecycle", "text": "- Replace direct `LLM(...)` construction throughout `tests/entrypoints` with the shared `VllmRunner` lifecycle. - Add an `ExitStack`-backed runner factory for tests that need one long-lived runner or several concurrent runners. - Consolidate multimodal, structured-output, offline-mode, collective-RPC, pooling, and weight-transfer cleanup onto the complete runner shutdown path. - Defer ROCm VRAM settling while multiple runners coexist, then release runner references and perform one bounded wait after every engine exits. - Preserve direct-constructor behavior where required, including pooling model length and chunked-prefill defaults. This addresses AMD CI instability caused by entrypoint tests constructing `LLM` directly and relying on partial or ad hoc cleanup. Those paths could leave engine processes or lazily reclaimed ROCm allocations alive long enough to starve a later model startup. Routing construction through `VllmRunner` guarantees engine shutdown, Dynamo reset, distributed cleanup, and bounded VRAM settling even when a test fails. The multi-engine GPU-utilization case requires deferred settling because no individual runner can reach baseline while its peers remain live.", "url": "https://github.com/vllm-project/vllm/pull/51280", "createdAt": "2026-08-06T16:25:21Z", "updatedAt": "2026-08-13T17:24:04Z", "timestamp": "2026-08-13T17:24:04Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "structured-output" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51287", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable async scheduling when VLLM_BATCH_INVARIANT=1", "text": "## Purpose `VLLM_BATCH_INVARIANT=1` pins the *kernels* so a row's result no longer depends on batch size or composition. It does nothing about **which batch the scheduler puts a request in**. Async scheduling is enabled by default (`SchedulerConfig.async_scheduling = None` -> `True`) and composes the next step's batch before the current step has retired, so batch membership depends on wall-clock timing. Timing jitter therefore reaches the numerics again, and the flag does not deliver what it promises. There is no guard, no warning, and nothing in the docs. The server boots, serves, and quietly is not reproducible. `--no-async-scheduling` fixes it, at roughly 3% throughput. This is not a corner case for the flag's intended users: batch invariance exists so research and regulated workloads can reproduce results, and those are exactly the workloads running large batches of long, variable-length requests. This PR treats `VLLM_BATCH_INVARIANT` as an async-scheduling incompatibility, following the existing structure in `VllmConfig.__post_init__`: - **explicitly enabled** (`--async-scheduling`): `ValueError` pointing at `--no-async-scheduling`, matching the ROCm DeepEP DBO and unsupported-executor cases. - **unset** (the default): disabled with `warning_once`, matching the pooling-model, spec-decode and ROCm DBO cases. The closest precedent is ROCm DeepEP high-throughput DBO, disabled \"because that combination can corrupt DP+EP generation accuracy\", which is the same argument. vLLM already auto-disables cascade attention and custom all-reduce under this same flag. Also documents the interaction in `docs/features/batch_invariance.md`, which currently says nothing about scheduler configuration. Related: #27433. ## Test Plan ### 1. Config tests (CPU only) Added to `tests/test_config.py`, next to the existing `test_draft_model_enables_async_scheduling_by_default`: - `test_batch_invariance_disables_async_scheduling`: with the flag set, `async_scheduling` resolves to `False`. - `test_batch_invariance_rejects_explicit_async_scheduling`: with the flag set and `async_scheduling=True` passed explicitly, `VllmConfig` raises. ### 2. End to end Serve, before this patch, once per row: ```bash # A: default, async scheduling on VLLM_BATCH_INVARIANT=1 vllm serve $MODEL --no-enable-prefix-caching # B: the fix VLLM_BATCH_INVARIANT=1 vllm serve $MODEL --no-enable-prefix-caching --no-async-scheduling ``` Against each, run: ```python # pip install openai import concurrent.futures as cf, hashlib, sys from openai import OpenAI MODEL, BATCHES, REPEATS = sys.argv[1], (1, 5, 10, 50), 3 client = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\") WORDS = (\"market revenue guidance segment margin pipeline capacity demand headwind \" \"backlog utilization cadence inventory logistics currency hedging\").split() def text(n, shift=0): return \" \".join(WORDS[(i + shift) % len(WORDS)] for i in range(n)) PROBE = \"Summarize this in detail, listing every theme you find.\\n\\n\" + text(9000) def answer(prompt, limit): m = client.chat.completions.create( model=MODEL, messages=[{\"role\": \"user\", \"content\": prompt}], temperature=0, max_tokens=limit).choices[0].message return (getattr(m, \"reasoning_content\", \"\") or \"\") + (m.content or \"\") def probe_hash(batch): jobs = [(PROBE, 600)] + [(text(3000, i), 400 + i % 8 * 150) for i in range(batch - 1)] with cf.ThreadPoolExecutor(batch) as pool: probe = list(pool.map(lambda j: answer(*j), jobs))[0] return hashlib.sha256(probe.encode()).hexdigest()[:12] hashes = {} for rep in range(REPEATS): for batch in BATCHES: h = probe_hash(batch) hashes.setdefault(h, []).append(f\"rep{rep} batch{batch}\") print(f\"rep{rep} batch{batch:>3} {h}\") print(\"PASS\" if len(hashes) == 1 else f\"FAIL {len(hashes)} outputs for one input\") ``` Two properties of that script are load-bearing. The **probe is long**: short prompts are reproducible even on a badly broken config. The **fillers use staggered `max_tokens`**: they retire at different decode steps, so batch composition churns while the probe is still generating. Drop either one and this passes on a server that is not batch invariant, which is exactly why the existing tests miss it (below). After this patch, A and B are the same run: async scheduling is off by default under the flag, and the extra argument is no longer needed. ## Test Result Measured on `openai/gpt-oss-120b`, single A100-80GB (sm80), `TRITON_ATTN`, temperature 0. | | probe reproducible | |---|---| | A, default (async scheduling on) | **9/10**, one request flips between runs | | B, `--no-async-scheduling` | **10/10** | Throughput cost of B: roughly 3%. The flipping request is stable when run alone (6/6 identical), so this needs co-residency: it is batch composition, not the request itself. It bites inputs sitting on a near-tie. In one case the same prompt produced either a long answer or an essentially empty one, a sub-ULP difference selecting between \"emit content\" and \"emit nothing\". ### Why the existing determinism tests do not catch this `tests/v1/determinism/test_batch_invariance.py` passes on a configuration where the above is failing. Two structural blind spots: 1. **`needle_prompt = \"There once was a \"`**: the request whose output is *checked* is 7 characters, and only the fillers are long. Short prompts are reproducible even when the config is broken. 2. **`max_tokens = 128` for every request**: all requests decode the same number of steps and retire *together*, so decode batch composition is constant for the whole test. The condition that breaks is batch composition **changing mid-decode**, not batch size. ### Caveat The end-to-end numbers were taken on v0.23.0 with #46639's Marlin MoE kernel back-ported and compiled for sm80, since MXFP4 MoE has no batch-invariant kernel on sm80 otherwise. The config change here is architecture independent and covered by the CPU-only tests. A maintainer re-running the determinism suite on a natively supported configuration would be worth doing before merge. The serve configuration, the build recipe for that sm80 kernel, and a fuller version of the test script above are at https://github.com/GiesDSRS/batch-invariant-vllm-a100.", "url": "https://github.com/vllm-project/vllm/pull/51287", "createdAt": "2026-08-06T17:27:06Z", "updatedAt": "2026-08-13T07:11:21Z", "timestamp": "2026-08-13T07:11:21Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "documentation", "needs-rebase" ], "author": "fsaudm", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51291", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Bugfix] Restore AITER-compatible Triton 3.7 pin", "text": "## Purpose Restore an AITER-compatible ROCm Triton 3.7 pin. [vLLM #50607](https://github.com/vllm-project/vllm/pull/50607) moved the ROCm image to Triton `0263a6a` from `release/internal/3.7.x`. That revision rejects the `DistributedLinearLayout` offsets produced by AITER's CDNA4 Gluon MLA kernel: ```text expected offsets type layout to be BlockedLayout or SliceLayout ``` The Kimi-K3 DSpark K=7 stress reproducer reaches this path during piecewise graph capture because the draft verifies eight query tokens across 12 local heads. Pinning ROCm Triton to `89002410` restores the distributed-layout lowering used by AITER without changing vLLM or AITER runtime code. The production InferenceX configuration uses K=2 and is tested separately below. This restores compilation and execution of AITER's actual Gluon MLA kernel; it does not disable Gluon or route the workload to a different attention backend. At the package-version level, this is a compatibility rollback from Triton 3.7.1 to the validated 3.7.0 revision. The commits are on divergent branches, so this is not a linear Git revert. A corrected internal 3.7.x revision can replace this pin after the complete layout support is backported and validated. [AITER v0.1.19's default installer](https://github.com/ROCm/aiter/blob/v0.1.19/.github/scripts/install_triton.sh#L77-L83) pins `triton==3.7.0`. On AITER's ROCm 7.2 package index, that resolves to `triton-3.7.0+amd.rocm7.2.0.git89002410`, so this change matches AITER's production wheel exactly. AITER's older `756afc06` pin is used only by its opt-in source-build CI path. I searched open vLLM, AITER, and ROCm Triton issues and pull requests for the error, the two Triton revisions, and the AITER MLA layout. I found no existing fix. [Triton #9513](https://github.com/triton-lang/triton/pull/9513) is related: cherry-picking only its relaxed `DistributedLayout` type check onto ROCm Triton's `release/internal/3.7.x` branch at the exact vLLM-pinned base commit [`0263a6a6203cf27c441c57a6c808ea87ffb8f654`](https://github.com/ROCm/triton/commit/0263a6a6203cf27c441c57a6c808ea87ffb8f654) made the exact Kimi-K3 shape compile, but the default two-split path still produced nondeterministic incorrect output. In a deterministic seed sweep from 0 through 15, seeds 7, 10, 14, and 15 failed at `atol=rtol=0.01`; seeds 7 and 10 had about 10.3% mismatching elements with maximum absolute errors of 3.67 and 3.84. Repeated seed-7 runs used identical input and reference hashes but produced different GPU output hashes, with mismatches confined to query position 2 across all 12 heads. Forcing one split on the same inputs passed with maximum absolute error `0.00390625`, isolating the remaining problem to the multi-split Stage-1/Stage-2 dependency rather than the frontend layout check. Therefore the isolated #9513 backport is not sufficient; the validated `89002410` revision is used instead. AI assistance was used for root-cause investigation and drafting. The submitter reviewed the Dockerfile changes and test evidence before opening this pull request. ## Test Plan ### 1. Docker build Build the same stage changed by this PR: ```bash docker build \\ -f docker/Dockerfile.rocm_base \\ --target build_triton \\ -t vllm-rocm-build-triton:89002410 . ``` The stage must fetch the full immutable commit, complete the Triton build, and produce a `triton-3.7.0+git89002410` wheel. ### 2. AITER MLA operator reproducer Compare the immutable ROCm nightly image `sha256:726321b39e1dbbe24d2735160be54d679496ef5d2a8479a264394f7296908617` with only its Triton package changed from `0263a6a` to `89002410`. Run the AITER MLA operator at the Kimi-K3 DSpark verification shape: ```bash python3 op_tests/test_mla.py -c 128 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal ``` Also run context length 256 while forcing the one- and two-split reduction paths. ### 3. TP8 endpoint smoke test Build a new ROCm base image directly from this PR's `docker/Dockerfile.rocm_base`. Then build the runnable vLLM server image from the same PR checkout using that base. Do not install or copy Triton into an existing nightly image. ```bash git rev-parse HEAD # 2d1bb40fb4a07e65706749b77c1701eed771695a DOCKER_BUILDKIT=1 docker build \\ --progress=plain \\ -f docker/Dockerfile.rocm_base \\ -t rocm/vllm-dev:pr51291-base . DOCKER_BUILDKIT=1 docker build \\ --progress=plain \\ -f docker/Dockerfile.rocm \\ --build-arg BASE_IMAGE=rocm/vllm-dev:pr51291-base \\ -t vllm/vllm-openai-rocm:pr51291-source . ``` Verify that the runnable image contains the PR source revision and the Triton wheel produced by the base-image build: ```bash docker run --rm --entrypoint /bin/bash \\ vllm/vllm-openai-rocm:pr51291-source -lc ' cat /app/versions.txt python3 - <<\"PY\" from importlib.metadata import version import triton import vllm print(\"vLLM:\", vllm.__version__) print(\"AITER:\", version(\"amd-aiter\")) print(\"Triton:\", triton.__version__) PY ' ``` `/app/versions.txt` must contain `TRITON_BRANCH: 89002410dee054a1207a3d88cfbe1748dacfd819`, and the installed Triton version must be `3.7.0+git89002410`. Start Kimi-K3 and DSpark on 8x MI355X. This uses InferenceX's production DSpark level 2 and its token-aligned capture sizes: with K=2, each sequence has a decode width of three tokens, so the capture list is `3,6,...,48` for `max_num_seqs=16`. This validation fixes both the model and KV cache to BF16. The command uses `--kv-cache-dtype bfloat16` explicitly rather than relying on dtype inference. ```bash docker run -d --name vllm-kimi-k3-pr51291-source-k2-ix \\ --network host --ipc host \\ --device /dev/kfd --device /dev/dri --group-add video \\ --cap-add SYS_PTRACE \\ --security-opt seccomp=unconfined --security-opt label=disable \\ -v /data/huggingface/Kimi-K3:/model:ro \\ -v /data/huggingface/Kimi-K3-DSpark:/draft:ro \\ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\ -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\ -e AITER_ROCM_ARCH=gfx950 \\ -e VLLM_ROCM_USE_AITER=1 \\ -e VLLM_ROCM_USE_AITER_FP4BMM=1 \\ -e SAFETENSORS_FAST_GPU=1 \\ -e AITER_SITUV2_A8W4=1 \\ -e AITER_BF16_FP8_MOE_BOUND=0 \\ -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \\ --entrypoint vllm \\ vllm/vllm-openai-rocm:pr51291-source \\ serve /model \\ --served-model-name moonshotai/Kimi-K3 \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 8 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --load-format auto --moe-backend auto \\ --gpu-memory-utilization 0.95 \\ --kv-cache-memory-bytes 51539607552 \\ --kv-cache-dtype bfloat16 \\ --mm-encoder-tp-mode data \\ --max-num-seqs 16 \\ --max-num-batched-tokens 4096 \\ --max-model-len 1048576 \\ --enable-prefix-caching \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"],\"cudagraph_capture_sizes\":[3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48]}' \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"/draft\",\"num_speculative_tokens\":2,\"attention_backend\":\"TRITON_MLA\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' ``` Wait for readiness and send a chat smoke request: ```bash until curl --fail --silent http://127.0.0.1:8000/health; do test \"$(docker inspect -f '{{.State.Status}}' vllm-kimi-k3-pr51291-source-k2-ix 2>/dev/null)\" = running || { docker logs vllm-kimi-k3-pr51291-source-k2-ix exit 1 } sleep 30 done curl --fail --silent --show-error \\ http://127.0.0.1:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{\"model\":\"moonshotai/Kimi-K3\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with one short word.\"}],\"temperature\":0,\"max_tokens\":16}' ``` ### 4. Repository checks ```bash pre-commit run --files docker/Dockerfile.rocm_base ``` ## Test Result ### Docker build The `build_triton` target fetched and checked out `89002410dee054a1207a3d88cfbe1748dacfd819`, completed all 503 compilation steps, and generated the `triton-3.7.0+git89002410` wheel. ### Operator result MI355X, TP8 system, BF16 inputs and KV cache: ```bash # Context 128, automatic split selection. python3 op_tests/test_mla.py -c 128 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal # Context 256, force the single-split path. python3 op_tests/test_mla.py -c 256 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal --gluon-num-kv-splits 1 # Context 256, force the two-split Stage-1/Stage-2 path. python3 op_tests/test_mla.py -c 256 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal --gluon-num-kv-splits 2 ``` | Test | `0263a6a` | `89002410` | |---|---|---| | MLA, context 128, automatic two splits | compile error | pass, 10.30 us | | MLA, context 256, forced one split | compile error | pass, 10.64 us | | MLA, context 256, forced two splits | compile error | pass, 10.80 us | All passing operator cases matched the PyTorch reference with maximum absolute error `0.00390625` at `atol=rtol=0.01` and zero failing elements. These tests executed `aiter.ops.triton.gluon.mla_gluon`, including the Kimi-K3 12-head, eight-query-token verification shape. ### TP8 endpoint result The image build and endpoint smoke were run with images built from this PR's source tree and Dockerfiles; no package was replaced in an existing nightly image. No other vLLM pull request or AITER overlay was included. Explicit BF16 KV startup was reconfirmed on 2026-08-07. - Base image: `rocm/vllm-dev:pr51291-base` (`sha256:a53d6c7dae565b2c7df3e530c2eeec496b259d22195e3115ed4ca82737aa4a3b`) - Runnable image: `vllm/vllm-openai-rocm:pr51291-source` (`sha256:4212a65ac86027e370b18ba2ad8acbb53e0f9d72c6d8a64ddf611bf75abf061b`) - Runtime versions: vLLM `0.1.dev19627+g2d1bb40fb.d20260806`, Triton distribution `3.7.0+git89002410`, AITER `0.1.19`, PyTorch `2.12.0+git6bbd260` - Hardware: 8x MI355X (`gfx950`), TP8 - KV cache: BF16, selected explicitly with `--kv-cache-dtype bfloat16` - Target backend: `ROCM_AITER_MLA`; DSpark draft backend: `TRITON_MLA` - With K=2 and the explicit InferenceX token-aligned capture list, piecewise graph capture completed for all 16 capture sizes on all eight ranks in 66 seconds. - `/health` returned HTTP 200. A chat-completions smoke request returned HTTP 200 with 16 completion tokens. - Speculative decoding was active: the K=2 smoke metrics recorded 11 drafts, 22 draft tokens, and four accepted tokens; both draft positions recorded at least one accepted token. The chat request is an endpoint/runtime smoke test, not an accuracy evaluation. All pre-commit hooks applicable to `docker/Dockerfile.rocm_base` passed.", "url": "https://github.com/vllm-project/vllm/pull/51291", "createdAt": "2026-08-06T17:37:39Z", "updatedAt": "2026-08-13T14:50:27Z", "timestamp": "2026-08-13T14:50:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build" ], "author": "JohnQinAMD", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51302", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] deepseek-vl2: restore original DeepseekV2Config defaults for omitted language_config fields", "text": "## Purpose Loading `deepseek-ai/deepseek-vl2-small` (and `deepseek-ai/deepseek-vl2`) currently crashes during model initialization: ``` TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' ``` at `DeepseekV2Attention.__init__` (`self.kv_lora_rank + self.qk_rope_head_dim`). Working around that surfaces a second failure while loading `embed_tokens`: ``` AssertionError # loaded_weight.shape[output_dim] == self.org_vocab_size ``` Both failures share one root cause. DeepSeek-VL2 checkpoints ship an incomplete `language_config` that relies on the defaults of the *original* DeepSeek-VL2 `DeepseekV2Config` (`vocab_size=102400`, `kv_lora_rank=512`, `qk_nope_head_dim=128`, `qk_rope_head_dim=64`, `v_head_dim=128`). For example, `deepseek-vl2-small`'s `language_config` contains neither `vocab_size` nor `kv_lora_rank`. vLLM does not use the model's remote config; `DeepseekVLV2Config` parses `language_config` with the built-in Transformers `DeepseekV2Config` (`DeepseekVLV2TextConfig`), whose generic defaults differ: - `DeepseekV2Config().vocab_size == 32000` (not 102400) - `DeepseekVLV2TextConfig.kv_lora_rank` is forced to `None` So omitted fields silently resolve to wrong values: 1. `kv_lora_rank=None` makes `ModelConfig.is_deepseek_mla` return `False` (`use_mla=False`); combined with the non-zero `qk_*_head_dim` defaults the decoder falls through to `DeepseekV2Attention`, which computes `kv_lora_rank + qk_rope_head_dim` → `None + int` → `TypeError`. 2. `vocab_size=32000` mismatches the checkpoint's 102400-row `embed_tokens`, tripping the assertion in `VocabParallelEmbedding.weight_loader`. This fix restores the original DeepSeek-VL2 defaults **only for fields the checkpoint does not set**, before building the text config: ``` deepseek_vl2_reference_defaults = { \"vocab_size\": 102400, \"kv_lora_rank\": 512, \"qk_nope_head_dim\": 128, \"qk_rope_head_dim\": 64, \"v_head_dim\": 128, } for key, value in deepseek_vl2_reference_defaults.items(): language_config.setdefault(key, value) ``` ## Test Plan Serve deepseek-ai/deepseek-vl2-small and confirm it no longer crashes ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51302", "createdAt": "2026-08-06T19:26:22Z", "updatedAt": "2026-08-12T23:24:11Z", "timestamp": "2026-08-12T23:24:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "deepseek" ], "author": "shepark", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51309", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Skip redundant sparse index remap on non-indexer layers", "text": "## Summary With cross-layer index sharing, only some attention layers update the shared top-k buffer. The old path still remapped that buffer on every layer. This PR runs the remap only when the layer owns an indexer. For GLM-5.2 with `index_topk_freq = 4`, this reduces remaps from 78 to 21 per forward pass. ## Safety - Layer 0 always owns an indexer, so the shared output is initialized before reuse. - Non-indexer layers do not change the inputs to the remap. - The default `index_topk_freq = 1` path is unchanged. - Speculative decoding still remaps on every layer. ## Results MI355X, TP=4, rank 0, same serving workload: | remap kernel | before | after | change | |---|---:|---:|---:| | launches | 139,378 | 37,254 | **-73.3%** | | GPU time | 2,392.1 ms | 645.7 ms | **-73.0%** | | launches per layer | 1.000 | 0.269 | matches 21/78 | The red box below is a redundant remap on a layer with no indexer. In the second capture, the same layer enters the MLA kernel directly. ![Before: redundant remap](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr2_before.png) ![After: remap removed](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr2_after.png) ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 81.8 | **+0.7%** | 1392 | 1385 | 10.80 | 10.73 | -0.6% | | 1024 | 4 | 32 | 330.5 | 332.4 | **+0.6%** | 131 | 130 | 11.66 | 11.60 | -0.5% | | 1024 | 8 | 64 | 562.4 | 567.0 | **+0.8%** | 142 | 141 | 13.08 | 12.96 | -0.9% | | 1024 | 16 | 96 | 875.0 | 879.9 | **+0.6%** | 211 | 210 | 16.44 | 16.37 | -0.4% | | 1024 | 32 | 128 | 1333.8 | 1329.9 | **-0.3%** | 333 | 337 | 20.58 | 20.63 | +0.2% | | 1024 | 64 | 256 | 2026.2 | 2057.8 | **+1.6%** | 442 | 437 | 26.96 | 26.45 | -1.9% | | 1024 | 128 | 384 | 2987.2 | 2999.1 | **+0.4%** | 986 | 973 | 35.19 | 35.01 | -0.5% | | 1024 | 256 | 768 | 4338.7 | 4378.1 | **+0.9%** | 1630 | 1633 | 50.59 | 50.17 | -0.8% | | 8192 | 1 | 16 | 84.5 | 85.1 | **+0.7%** | 339 | 338 | 11.50 | 11.42 | -0.7% | | 8192 | 4 | 32 | 302.2 | 304.3 | **+0.7%** | 447 | 442 | 12.44 | 12.36 | -0.6% | | 8192 | 8 | 64 | 473.4 | 474.2 | **+0.2%** | 666 | 640 | 15.10 | 15.16 | +0.4% | | 8192 | 16 | 96 | 665.3 | 668.3 | **+0.4%** | 825 | 824 | 21.44 | 21.32 | -0.6% | | 8192 | 32 | 128 | 949.8 | 949.6 | **-0.0%** | 1861 | 1749 | 29.40 | 29.58 | +0.6% | | 8192 | 64 | 256 | 1269.4 | 1276.5 | **+0.6%** | 2991 | 3060 | 44.80 | 44.31 | -1.1% | | 8192 | 128 | 384 | 1634.2 | 1649.1 | **+0.9%** | 6450 | 6453 | 68.91 | 68.02 | -1.3% | | 8192 | 256 | 768 | 1909.0 | 1927.7 | **+1.0%** | 14096 | 13901 | 116.63 | 115.29 | -1.1% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.6%** | -0.5% | -0.7% | | ISL 8192 | **+0.6%** | -1.3% | -0.6% | | **all 16** | **+0.6%** | -0.9% | -0.6% | Throughput improves on 14 of 16 cases; the two exceptions are both at concurrency 32, at -0.3% and -0.0%. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9447 | | gsm8k flexible-extract | 0.9507 | 0.9469 | | RULER niah_single_2 @16k | 0.870 | 0.884 | Unchanged within run-to-run variation. The skipped remap rewrites `paged_kv_indices` with bit-identical values, so no output change is expected. ## Tests ```bash pytest tests/kernels/attention/test_rocm_aiter_mla_sparse_index_remap.py ``` The tests cover the indexer/speculative-decoding gate and verify whether the remap is called.", "url": "https://github.com/vllm-project/vllm/pull/51309", "createdAt": "2026-08-06T21:10:14Z", "updatedAt": "2026-08-13T07:53:17Z", "timestamp": "2026-08-13T07:53:17Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51311", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[K3 Perf] Flash kda out kernel for prefill, 1.1~1.4x kernel performance improvement", "text": "## Purpose Use workspace manager to preallocate the memory for `_flashkda_prefill`, avoid re-allocate each time when we call the kernel. ## Test Acc covered in unit tests Perf can be seen in this AI generated script ```py import statistics import time import torch import vllm._flashkda_C # noqa: F401 from vllm.models.kimi_k3.nvidia.kda import _flashkda_prefill TOKENS = (1, 32, 64, 128, 256, 512, 1024, 2048) H, D, N = 12, 128, 8 WARMUP, ITERATIONS, REPEATS = 10, 100, 5 DEVICE = torch.device(\"cuda\") def benchmark(fn) -> float: for _ in range(WARMUP): fn() torch.accelerator.synchronize() samples = [] for _ in range(REPEATS): start = time.perf_counter() for _ in range(ITERATIONS): fn() torch.accelerator.synchronize() samples.append((time.perf_counter() - start) * 1000 / ITERATIONS) return statistics.median(samples) def run_case(T: int) -> tuple[float, float]: num_seqs = min(N, T) q, k, v, g = [ torch.randn(1, T, H, D, dtype=torch.bfloat16, device=DEVICE) for _ in range(4) ] beta = torch.randn(1, T, H, dtype=torch.bfloat16, device=DEVICE) A_log = torch.randn(H, dtype=torch.float32, device=DEVICE) dt_bias = torch.randn(H, D, dtype=torch.float32, device=DEVICE) initial_state = torch.randn(num_seqs, H, D, D, dtype=torch.float32, device=DEVICE) cu_seqlens = torch.arange(num_seqs + 1, dtype=torch.int32, device=DEVICE) * ( T // num_seqs ) workspace_size = torch.ops._flashkda_C.get_workspace_size(T, H, num_seqs) core_out = torch.empty_like(v) state_cache = torch.empty_like(initial_state) final_state = torch.empty_like(initial_state) workspace = torch.empty(workspace_size, dtype=torch.uint8, device=DEVICE) def before() -> None: out = torch.empty_like(v) final = torch.empty_like(initial_state) ws = torch.empty(workspace_size, dtype=torch.uint8, device=DEVICE) torch.ops._flashkda_C.fwd( q, k, v, g, beta, D**-0.5, out, ws, A_log, dt_bias, -3.0, initial_state, final, cu_seqlens, ) core_out.copy_(out) state_cache.copy_(final) def after() -> None: _, final = _flashkda_prefill( q, k, v, g, beta, A_log, dt_bias, -3.0, initial_state, cu_seqlens, core_out, final_state, workspace, ) state_cache.copy_(final) return benchmark(before), benchmark(after) @torch.inference_mode() def main() -> None: print(f\"{'tokens':>8} {'before (ms)':>12} {'after (ms)':>11} {'speedup':>9}\") for tokens in TOKENS: before_ms, after_ms = run_case(tokens) print( f\"{tokens:>8} {before_ms:>12.3f} {after_ms:>11.3f} \" f\"{before_ms / after_ms:>8.3f}x\" ) if __name__ == \"__main__\": main() ``` And we can get ```bash tokens before (ms) after (ms) speedup 1 0.037 0.026 1.406x 32 0.042 0.034 1.263x 64 0.044 0.033 1.307x 128 0.042 0.033 1.278x 256 0.043 0.033 1.285x 512 0.043 0.033 1.324x 1024 0.042 0.033 1.283x 2048 0.048 0.044 1.113x ```", "url": "https://github.com/vllm-project/vllm/pull/51311", "createdAt": "2026-08-06T21:19:06Z", "updatedAt": "2026-08-12T18:51:50Z", "timestamp": "2026-08-12T18:51:50Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "ready", "kimi", "k3" ], "author": "yewentao256", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51314", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Skip cleaning sparse prefill MQA logits", "text": "## Summary The sparse prefill path creates the MQA-logits buffer with `torch.full(..., -inf)`. This launches an FP32 fill before every logits kernel. This PR passes `clean_logits=False` to AITER, so the buffer uses `torch.empty` and the fill is removed. ## Safety The only consumer is `top_k_per_row_prefill`. It reads each row only inside `[cu_seqlen_ks, cu_seqlen_ke)`. Its vectorized path also guards the final partial load, so padding values are never read. The top-k result is unchanged. ## Results MI355X, TP=4, rank 0, comparable before/after captures: | metric | before | after | |---|---:|---:| | FP32 fill launches | 18,162 | **0** | | FP32 fill GPU time | 1,250.2 ms | **0 ms** | | MQA-logits launches | 18,108 | 17,930 | The removed fill was **0.89% of total GPU kernel time**. In the sample below it takes 74.6 µs and is fully serialized before the logits kernel. ![Before: FP32 fill before MQA logits](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr4_before.png) ![After: fill removed](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr4_after.png) ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 81.2 | **-0.1%** | 1392 | 1387 | 10.80 | 10.82 | +0.2% | | 1024 | 4 | 32 | 330.5 | 330.5 | **+0.0%** | 131 | 131 | 11.66 | 11.66 | +0.0% | | 1024 | 8 | 64 | 562.4 | 561.0 | **-0.3%** | 142 | 142 | 13.08 | 13.10 | +0.2% | | 1024 | 16 | 96 | 875.0 | 880.4 | **+0.6%** | 211 | 211 | 16.44 | 16.35 | -0.5% | | 1024 | 32 | 128 | 1333.8 | 1333.5 | **-0.0%** | 333 | 333 | 20.58 | 20.56 | -0.1% | | 1024 | 64 | 256 | 2026.2 | 2042.7 | **+0.8%** | 442 | 438 | 26.96 | 26.66 | -1.1% | | 1024 | 128 | 384 | 2987.2 | 2985.9 | **-0.0%** | 986 | 979 | 35.19 | 35.17 | -0.1% | | 1024 | 256 | 768 | 4338.7 | 4337.8 | **-0.0%** | 1630 | 1628 | 50.59 | 50.55 | -0.1% | | 8192 | 1 | 16 | 84.5 | 84.3 | **-0.3%** | 339 | 336 | 11.50 | 11.53 | +0.3% | | 8192 | 4 | 32 | 302.2 | 302.4 | **+0.0%** | 447 | 444 | 12.44 | 12.44 | +0.0% | | 8192 | 8 | 64 | 473.4 | 471.4 | **-0.4%** | 666 | 636 | 15.10 | 15.27 | +1.1% | | 8192 | 16 | 96 | 665.3 | 668.7 | **+0.5%** | 825 | 826 | 21.44 | 21.33 | -0.5% | | 8192 | 32 | 128 | 949.8 | 947.2 | **-0.3%** | 1861 | 1770 | 29.40 | 29.59 | +0.6% | | 8192 | 64 | 256 | 1269.4 | 1270.4 | **+0.1%** | 2991 | 3077 | 44.80 | 44.52 | -0.6% | | 8192 | 128 | 384 | 1634.2 | 1641.3 | **+0.4%** | 6450 | 6423 | 68.91 | 68.55 | -0.5% | | 8192 | 256 | 768 | 1909.0 | 1917.4 | **+0.4%** | 14096 | 14072 | 116.63 | 115.82 | -0.7% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.1%** | -0.3% | -0.2% | | ISL 8192 | **+0.1%** | -1.1% | -0.0% | | **all 16** | **+0.1%** | -0.7% | -0.1% | The removed fill is on the prefill path, so the signal lands in TTFT rather than throughput: **-1.1% geomean at ISL 8192**, reaching -4.9% at concurrency 32 (1861 -> 1770 ms). Throughput and TPOT are flat, as expected for a change worth 0.89% of GPU kernel time in a decode-bound sweep; every case sits within +/-0.8% on throughput, i.e. inside run-to-run spread. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9575 | | gsm8k flexible-extract | 0.9507 | 0.9621 | | RULER niah_single_2 @16k | 0.870 | 0.838 | Unchanged. Both tasks run at the checkpoint's default sampling, so these deltas are inside run-to-run variation in both directions. ## Tests ```bash pytest tests/v1/attention/test_rocm_aiter_mla_sparse.py ``` The new test verifies that vLLM forwards `clean_logits=False` to AITER.", "url": "https://github.com/vllm-project/vllm/pull/51314", "createdAt": "2026-08-06T21:41:17Z", "updatedAt": "2026-08-13T07:53:15Z", "timestamp": "2026-08-13T07:53:15Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51315", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fuse the DSA indexer prologue with AITER", "text": "## Summary The DSA indexer prologue uses four kernels for K normalization, Q/K RoPE, FP8 quantization, and the K-cache write. This PR uses AITER's fused `indexer_qk_rope_quant_and_cache` kernel instead. The fused path is enabled only for its supported BF16/FP8 shape; other configurations keep the existing path. ## Results MI355X, TP=4, rank 0: | indexer stage | before | after | change | |---|---:|---:|---:| | launches | 4 | 1 | **-75%** | | decode time | 17.4 µs | 5.9 µs | **-66%** | | prefill time | 163.9 µs | 270.6 µs | **+65%** | The fused kernel is faster for decode but slower for prefill. Across the full trace, prologue GPU time changes from 1,093.6 ms to 1,041.4 ms (**-4.8%**). One decode indexer stage at the same zoom: ![Before: four prologue kernels](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr3_before.png) ![After: one fused prologue kernel](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr3_after.png) End-to-end geomean: | metric | before | after | change | |---|---:|---:|---:| | throughput | 811.75 | 817.58 | **+0.72%** | | TPOT | 24.12 ms | 23.96 ms | **-0.66%** | | ITL | 23.85 ms | 23.69 ms | **-0.67%** | ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 82.2 | **+1.1%** | 1392 | 1401 | 10.80 | 10.66 | -1.3% | | 1024 | 4 | 32 | 330.5 | 334.0 | **+1.0%** | 131 | 132 | 11.66 | 11.54 | -1.0% | | 1024 | 8 | 64 | 562.4 | 570.2 | **+1.4%** | 142 | 143 | 13.08 | 12.90 | -1.4% | | 1024 | 16 | 96 | 875.0 | 884.8 | **+1.1%** | 211 | 211 | 16.44 | 16.29 | -0.9% | | 1024 | 32 | 128 | 1333.8 | 1345.3 | **+0.9%** | 333 | 340 | 20.58 | 20.41 | -0.8% | | 1024 | 64 | 256 | 2026.2 | 2037.0 | **+0.5%** | 442 | 443 | 26.96 | 26.83 | -0.5% | | 1024 | 128 | 384 | 2987.2 | 3011.6 | **+0.8%** | 986 | 989 | 35.19 | 34.90 | -0.8% | | 1024 | 256 | 768 | 4338.7 | 4353.5 | **+0.3%** | 1630 | 1628 | 50.59 | 50.52 | -0.1% | | 8192 | 1 | 16 | 84.5 | 85.3 | **+0.9%** | 339 | 338 | 11.50 | 11.39 | -1.0% | | 8192 | 4 | 32 | 302.2 | 304.6 | **+0.8%** | 447 | 445 | 12.44 | 12.34 | -0.8% | | 8192 | 8 | 64 | 473.4 | 478.1 | **+1.0%** | 666 | 666 | 15.10 | 14.95 | -1.0% | | 8192 | 16 | 96 | 665.3 | 669.9 | **+0.7%** | 825 | 824 | 21.44 | 21.34 | -0.5% | | 8192 | 32 | 128 | 949.8 | 951.5 | **+0.2%** | 1861 | 1798 | 29.40 | 29.42 | +0.1% | | 8192 | 64 | 256 | 1269.4 | 1274.2 | **+0.4%** | 2991 | 3148 | 44.80 | 44.47 | -0.7% | | 8192 | 128 | 384 | 1634.2 | 1640.1 | **+0.4%** | 6450 | 6414 | 68.91 | 68.69 | -0.3% | | 8192 | 256 | 768 | 1909.0 | 1908.9 | **-0.0%** | 14096 | 14005 | 116.63 | 116.91 | +0.2% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.9%** | +0.5% | -0.9% | | ISL 8192 | **+0.5%** | -0.0% | -0.5% | | **all 16** | **+0.7%** | +0.2% | -0.7% | Per-case expansion of the geomean above. The gain is larger at ISL 1024 (+0.9%) than at ISL 8192 (+0.5%), consistent with the fused kernel being faster for decode and slower for prefill. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9530 | | gsm8k flexible-extract | 0.9507 | 0.9545 | | RULER niah_single_2 @16k | 0.870 | 0.852 | Unchanged within run-to-run variation. ## Tests ```bash pytest tests/kernels/attention/test_rocm_indexer_qk_rope_quant_cache.py ``` The test compares the fused and existing paths across both RoPE layouts, block sizes 16/64, and token counts 1/37/256.", "url": "https://github.com/vllm-project/vllm/pull/51315", "createdAt": "2026-08-06T21:41:29Z", "updatedAt": "2026-08-13T07:53:11Z", "timestamp": "2026-08-13T07:53:11Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "deepseek" ], "author": "amd-sriram", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51316", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend][gRPC] Add RL lifecycle control", "text": "## Purpose Add reinforcement-learning lifecycle operations to the Rust frontend's existing gRPC `Control` service. - Add pause and resume RPCs for scheduler control. - Add sleep and wake RPCs for GPU memory management. - Add weight-transfer initialization, start, update, finish, and version RPCs. - Advertise RL capabilities through the EngineCore ready handshake and reject unsupported operations before dispatch. - Serialize state mutations and require generation to be paused before weight updates. - Reuse the configured NCCL, IPC, or sparse-NCCL weight-transfer backend. - Document the administrative trust boundary for the unauthenticated control surface. This lets trusted out-of-process RL sidecars use the native Rust gRPC frontend instead of the development HTTP server. It adds no listener, port, or separate weight-transfer implementation. ## Test Plan - Run Rust formatting, compilation, and Clippy checks for the affected crates. - Run the focused gRPC control tests. - Run the Python-to-Rust MessagePack compatibility test. - Compile the changed Python engine modules. - Validate the complete diff for whitespace errors. ## Test Result - `cargo fmt --all -- --check`: passed. - `cargo check -p vllm-server -p vllm-engine-core-client`: passed. - `cargo +1.96.1 clippy -p vllm-server -p vllm-engine-core-client --all-targets -- -D warnings`: passed. - `cargo test -p vllm-server grpc::tests::control_ -- --nocapture`: 4 passed. - `cargo test -p vllm-engine-core-client python_msgpack_fixtures_match_rust_encoding -- --nocapture`: 1 passed. - `uv run --no-sync python -m compileall -q vllm/v1/engine rust/src/engine-core-client/src/tests/python_compat.py`: passed. - `git diff --check upstream/main...HEAD`: passed. **AI assistance disclosure:** This PR was authored with AI assistance and reviewed by the submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described. - [x] The test plan is included. - [x] The test results are included. - [x] The necessary security and weight-transfer documentation is updated.
", "url": "https://github.com/vllm-project/vllm/pull/51316", "createdAt": "2026-08-06T21:45:34Z", "updatedAt": "2026-08-13T03:33:46Z", "timestamp": "2026-08-13T03:33:46Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "documentation", "rust" ], "author": "connorcarpenter15", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51318", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][DSv4] Make the C128A decode topk row stride capture-stable", "text": "# [Bugfix][DSv4] Make the C128A decode topk row stride capture-stable Fixes the intermittent output corruption on DeepSeek-V4-Flash under concurrent long-context load reported in the #41834 thread (mid-generation `<|begin_of_sentence|>` bursts, multilingual token salad — one request per batch, never the first). Root-cause analysis and A/B evidence: https://github.com/vllm-project/vllm/pull/41834#issuecomment-5209098845. The same fix is open against the GB10 deployment fork as jasl/vllm#41. ## Root cause `_build_c128a_metadata` derives the row stride of the persistent C128A decode buffer from the **batch's** `max_seq_len`: ```python active_topk_width = min(max(next_pow2(cm.max_seq_len // 128), 128), c128a_max) ``` The decode consumers of that layout run **inside FULL cudagraphs**, and cudagraph capture builds attention metadata with `max_seq_len = max_model_len` (`gpu_model_runner.py`, `for_cudagraph_capture` branch). The captured kernels therefore bake the widest stride — 8192 for a 1M-context model — while runtime builds re-lay rows out at whatever the current batch gives (128…4096 in practice). Row 0 lines up at offset 0 under any stride and is always read correctly. Every later decode row is read at the capture-time offset — stale bytes from earlier, differently-strided builds — which the decode kernel treats as global compressed-KV slot ids: the victim request attends over pages it does not own (token salad), or over uninitialized fp8 where ~2/256 random bytes decode to NaN. One NaN turns the whole logits row NaN, and `argmax` over an all-NaN row returns index 0 — token 0 is BOS. This explains the field observations that made the bug hard to pin down: the victim is never the first request in the batch; single-request traffic never reproduces (no row 1); `--enforce-eager` is clean (no baked stride); deployments whose `max_model_len` is close to their tested context are clean (the widths coincide); and the corruption is deterministic per shape, so run-it-twice determinism checks pass. ## Fix Lay decode rows out at `c128a_max_compressed` always. The only cost is wider `-1` padding writes in the build kernel; kernel reads stay bounded by `decode_lens` via `topk_length`, so decode-side work is unchanged. ## Reproduction / verification * `test_c128a_decode_row_addresses_survive_batch_width_changes` reproduces the mechanism directly on one GPU in seconds: build metadata once at `max_seq_len = max_model_len` (what capture does) and once for a small batch, then assert decode row 1's stride and address did not move. On unfixed `main` it fails with *\"stride changed with the batch's max_seq_len (8192 at capture vs 128 at runtime)\"*; on this branch it passes. Verified both ways on GB10 (SM121). * `test_c128a_decode_stride_is_batch_independent` is a no-GPU AST gate on the assignment itself, in the spirit of not asserting on a value the builder could stop computing. Also verified to fail on the unfixed tree. * System-level A/B on 4× GB10 / TP=4 / 1M context / FULL_AND_PIECEWISE / spec-off, 2×300k-token concurrent streams per round, with engine-side NaN detection at the sampling-row gather: **unfixed, 39 NaN events in round 1; fixed, zero events across 8 rounds / 16 streams / ~105k decode steps**, zero leaked specials, zero multi-script output. An in-graph trace (device ops captured into the FULL graphs, so replays record what they compute) additionally confirmed the NaN is born only at compress-ratio-128 layers, only in rows ≥ 1, while the builder's row 1 held valid slot ids and the captured kernel read a stale row. Test commands run (GB10 / SM121, CUDA container): ``` pytest tests/v1/attention/test_dsv4_c128a_capture_stable_stride.py -v # unfixed main: 1 failed (stride), 1 failed (AST gate), behavioral repro fails # this branch: all pass ``` ## Contribution notes * Not a duplicate: no open PR touches the C128A metadata build (searched `c128a`, `sparse_mla deepseek`, `41834 in:body`); jasl/vllm#41 is the same fix in the deployment fork, referenced here intentionally. * AI assistance was used throughout (instrumentation, analysis, and the patch); the change and its verification were reviewed end-to-end by the submitter, who operates the affected production deployment.", "url": "https://github.com/vllm-project/vllm/pull/51318", "createdAt": "2026-08-06T22:15:50Z", "updatedAt": "2026-08-12T18:04:38Z", "timestamp": "2026-08-12T18:04:38Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "tobymao", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51323", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[1/N] HiSparse: host-resident sparse-MLA decode hot-buffering", "text": "Initial implementation based on #46326, with substantial improvements; notably, a heirarchical design which includes a standard device resident KV cache in addition to the hot buffer and host cache. Includes support for DeepSeek v4. Supersedes #46326 Preliminary results on B300: \"hisparse_pd_staged_pareto\" \"hisparse_blog_shapes\"", "url": "https://github.com/vllm-project/vllm/pull/51323", "createdAt": "2026-08-07T00:21:12Z", "updatedAt": "2026-08-13T16:09:56Z", "timestamp": "2026-08-13T16:09:56Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "documentation", "needs-rebase", "ci/build", "v1", "deepseek", "kv-connector", "nvidia", "mrv2" ], "author": "MatthewBonanni", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51332", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization][Humming] Support MXFP4 weight + block-FP8 activation for MoE", "text": "## Purpose Enable Humming MoE GEMM with **MXFP4 weights + block-FP8 (group-128) activations**, and — on the expert-parallel (DeepEP) path — quantize activations to block FP8 **before** the all-to-all dispatch so FP8 rather than BF16 crosses the interconnect. The Humming kernel already implements MXFP4 weights (group-32 `e8m0`) with a grouped FP8 activation via WGMMA software dequant, and runs on SM90 (Hopper/H200) — the `grouped-fp8` cases in Humming's `tests/kernels/humming/test_mxfp4.py` cover it. Only the vLLM plumbing to select it was missing. Two fixes: ## EP path: pre-dispatch block-FP8 quantization By default Humming quantizes activations internally, so EP dispatches **BF16**. For block-FP8 (group-128) we now quantize **before** the DeepEP dispatch, so only FP8 + fp32 block scales cross the interconnect (~half the dispatch traffic) and Humming consumes the pre-quantized input directly: 1. `get_humming_moe_quant_config` surfaces the block-FP8 dtype + `[1, 128]` group shape so the config reports `is_block_quantized`. 2. `expects_unquantized_inputs` returns `False` for block-FP8, so prepare/finalize does not defer quantization and emits the `(fp8, scales)` tuple. 3. `apply()` forwards the dispatch scale (`a1q_scale`) into `may_quant_input` (a no-op when a scale is supplied), so Humming skips the redundant w13 quantization; the scale is permuted/reshaped alongside tokens. No DeepEP-side change is needed: the dispatch byte count is data-driven off the quant config (`use_fp8_dispatch`), the same block-FP8 v2 path other backends already use. Every other scheme leaves `activation_group_shape=None` → unchanged BF16-dispatch behavior. ## Cap MoE tuning K-block at 128 Small-M tiles can select a K-block of 256 (`block_shape[2]`), whose TMA descriptor the driver rejects at `cuLaunchKernelEx` (`CUDA_ERROR_MISALIGNED_ADDRESS`). It fires in `profile_run`, so `--enforce-eager` does not avoid it. `_cap_tuning_config_k_block` caps `block_shape[2]` at 128 — what Humming already uses for larger M — so the W4A8 MoE runs under CUDA graphs. ### Usage On an MXFP4-weight checkpoint (e.g. `nm-testing/Qwen3-30B-A3B-MXFP4A16`): ``` VLLM_HUMMING_INPUT_QUANT_CONFIG='{\"dtype\":\"float8e4m3\",\"group_size\":128}' ``` The per-token variant `{\"dtype\":\"float8e4m3\"}` and BF16 dispatch for non-block schemes are unchanged. ## Test Plan - `tests/quantization/test_humming_mxfp4_block_fp8.py`: schema→`QuantKey` mapping, `(kMxfp4Static, kFp8Dynamic128Sym)` now allowed, block-FP8 config is `is_block_quantized` with a `[1, 128]` block, and default (no block shape) stays per-token/deferred. Schema-mapping cases skip when Humming isn't installed. - `tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8-block.yaml` — GSM8K config, added to `config-act-fp8.txt`. ## Test Result QuantKey-level verification passing locally: ``` grouped-128 float32 == kFp8Dynamic128Sym : True grouped-128 uint8 == kFp8Dynamic128Sym : False (old behavior) per-token float32 == kFp8DynamicTokenSym: True group-32 uint8 == kMxfp8Dynamic : True ``` > [!NOTE] > Draft: end-to-end GSM8K accuracy on H200 still to be run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51332", "createdAt": "2026-08-07T00:33:30Z", "updatedAt": "2026-08-13T11:33:03Z", "timestamp": "2026-08-13T11:33:03Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "elvircrn", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51338", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RFC][Bug Fix][Spec Decode] Require explicit speculative methods", "text": "This PR removes speculative-method guessing and makes the method an explicit contract. ## Problem On current `main`, vLLM guesses the speculative method from model path/name substrings such as `eagle3`, `dflash`, and `dspark`. A DFlash or DSpark checkpoint at a neutral training path can therefore be treated as the generic autoregressive `draft_model`, while renaming or moving the same checkpoint changes serving behavior. For example: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15}' ``` Previously, this could silently select `draft_model`. In the original end-to-end Qwen3-8B + DFlash reproduction, the misdetected path produced acceptance length 1.02; adding explicit `\"method\": \"dflash\"` produced 3.88. Serving a specialized checkpoint through the wrong proposer is worse than requiring one extra field, so this PR now fails closed. ## Design - A method is required for every explicit speculative configuration. It can be supplied as `\"method\"` in `--speculative-config` or with `--spec-method`. - The explicit method selects the proposer and any structural loader normalization. vLLM does not replace or second-guess it from the model path, checkpoint architecture, structural `model_type`, or other metadata. - Checkpoint metadata is used only after method selection for implementation details, such as choosing the embedded DeepSeek-V4 DSpark loader for `method=\"dspark\"` and defaulting its proposal depth from `dspark_block_size`. - Known checkpoint schemas that explicitly declare their algorithm may populate the method before `SpeculativeConfig` is constructed. The speculators format uses `speculators_model_type`; PEagle is normalized to `method=\"eagle3\"` with parallel drafting. - `num_speculative_tokens` remains optional when the selected checkpoint format declares a proposal depth, such as speculators `speculative_tokens`, `n_predict`, or DSpark `dspark_block_size`. Otherwise the existing clear error remains. - DFlash and DSpark parallel-drafting behavior is derived from the explicit method. - Explicit shorthand values are folded into the config before checkpoint-declared defaults, so `--spec-method`, `--spec-model`, and `--spec-tokens` follow the same precedence rule as JSON fields. For an embedded DeepSeek-V4 checkpoint, the same checkpoint metadata is therefore handled according to the requested method: `draft_model` keeps the ordinary model loader, `mtp` uses `DeepSeekV4MTPModel`, and `dspark` uses `DSparkDraftModel`. The metadata does not choose among them. ## Breaking change Configurations that relied on implicit selection must add a method. Before: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` After: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"method\": \"dflash\", \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` For a generic autoregressive draft model, use `\"method\": \"draft_model\"`. Legacy EAGLE/EAGLE3, Medusa, MLP-speculator, n-gram, and custom-proposer configurations must likewise state their method. A speculators-format checkpoint used as the top-level model still works without an explicit `--speculative-config`, because its schema declares both the algorithm and proposal depth. ## Validation - End-to-end reproduction recorded before this design change: - misdetected `draft_model`: acceptance length 1.02 - explicit `dflash`: acceptance length 3.88 - This PR now rejects the missing-method configuration instead of silently serving it autoregressively. - Explicit method, schema default, loader selection, and callable override tests: 18 passed. - Full draft HF override module: 11 passed. - DeepSeek EAGLE loader compatibility regression: 1 passed. - Earlier focused config, DSpark, engine-argument, scheduler, and structured-output suites: 50 passed. - Full `tests/test_config.py` before the final refinement: 179 passed; 2 failed only because this environment cannot access gated `meta-llama/Meta-Llama-3-8B-Instruct` (HTTP 403). - All pre-commit hooks passed on every changed file, including Ruff, mypy, Markdown lint, SPDX, DCO sign-off, and configuration validation. - GPU model evaluation was not rerun for this follow-up; the serving change is fail-fast configuration and loader-selection behavior, and the earlier DFlash measurement above is retained as the motivating model result.
Commands run ```bash .venv/bin/python -m pytest -q \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/test_config.py \\ -k 'method_selects_draft_override or callable_overrides_reach_the_draft_config or arch_mapping_applies_before_callable_override or composed_override_is_picklable or explicit_method_selects_deepseek_v4_loader or speculative_config_requires_method or speculators_format' .venv/bin/python -m pytest -q \\ 'tests/config/test_model_arch_config.py::test_draft_model_arch_config[eagle618/deepseek-v3-random-eagle618/eagle-deepseek-v3-random-eagle-True]' .venv/bin/python -m pytest -q tests/config/test_speculative_draft_hf_overrides.py .venv/bin/pre-commit run --files ```
## Contribution notes - This is not a duplicate of #47490: that PR preserves auto-detection and validates explicit-method mismatches; this PR removes automatic method selection and fails when an explicit configuration omits the method. - This is not a duplicate of #44566: that PR only propagates token depth from a speculators-format draft; this PR changes method-selection semantics while preserving declared token defaults. - This is not a duplicate of #52165: that PR detects embedded DeepSeek-V4 DSpark metadata and uses it for automatic routing; this PR makes the explicit method authoritative and uses metadata only to configure the selected loader. - AI assistance was used. I reviewed every changed line and ran the validation listed above.", "url": "https://github.com/vllm-project/vllm/pull/51338", "createdAt": "2026-08-07T00:53:10Z", "updatedAt": "2026-08-13T17:47:21Z", "timestamp": "2026-08-13T17:47:21Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "documentation", "new-model", "structured-output", "speculative-decoding", "rust" ], "author": "WindChimeRan", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51346", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Add sequence parallelism support for DeepSeek V4", "text": "## Purpose Add sequence parallelism (SP) support to the XPU DeepSeek V4 model path. Attention activations are kept sharded along the sequence dimension across TP ranks, so each rank only materializes `num_tokens / tp_size` rows for the MoE and hyper-connection stages. This reduces activation memory and the volume moved by the post-attention collective on XPU. Implementation notes: - Reuses the existing `vllm.models.common.ops.sequence_parallel` helpers (`sp_shard` / `sp_all_gather` / `sp_reduce_scatter` / `sp_padding_mask`) rather than introducing an XPU-specific mechanism. - `_use_sequence_parallel()` gates SP on `pp == 1 and enable_expert_parallel and tp > 1 and dp > 1`, following the existing upstream convention for this family of models. - Inside the decoder layer, the attention input is all-gathered and the attention output is reduce-scattered; `wo_b.reduce_results` is disabled so the reduce-scatter replaces the all-reduce instead of duplicating it. - `is_sequence_parallel` is propagated to `FusedMoE` and the shared experts. - The `VLLM_MOE_SKIP_PADDING` `forward_context.is_padding` mask is sharded consistently with the activations. - Covers the main model, MTP, and DSpark paths.", "url": "https://github.com/vllm-project/vllm/pull/51346", "createdAt": "2026-08-07T02:31:41Z", "updatedAt": "2026-08-13T00:35:33Z", "timestamp": "2026-08-13T00:35:33Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu", "deepseek" ], "author": "majian4work", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51349", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Render][1/n] Paged shared memory storage for mm tensor ipc.", "text": "## Purpose Paged shared memory storage for mm tensor ipc. 0. This compares two approaches for IPC of mm tensors: - **Paged shared memory (pshm):** mm tensor → shm → swap_blocks_batch → GPU - **ZMQ RPC:** mm tensor → ZMQ IPC → CPU Buffer → Pin Memory → GPU 1. CPU‑to‑GPU (H2D) bandwidth saturates when transferring 1 MiB blocks, so a page size of 1 MiB is a reasonable choice for shared memory storage. 2. A typical multimodal data payload is around 10 MiB, which is larger than page size of 1 MiB. Therefore, using paged shared memory for storing and transferring mm tensors is practically meaningful. 3. For payloads smaller than one page (e.g., 0.5 MiB), shared memory space is wasted, and the performance is nearly identical to that of ZMQ RPC. Hence, such small payloads offer no practical advantage. 4. Every copy operation along the mm data path introduces overhead, with dynamic memory allocation and pin memory being particularly expensive. Thus, removing the ZMQ IPC → CPU Buffer → Pin Memory steps in the ZMQ RPC path would bring significant performance gains. \"cpu_to_gpu_bandwidth\" Supplement↓
### 1. CPU‑to‑GPU (H2D) bandwidth saturates when transferring 1 MiB blocks PTAL: https://github.com/vllm-project/vllm/pull/48936 \"image\" ### 2. A typical multimodal data payload is around 10 MiB Qwen2VL outputs 10,764 patches when the input is a 1920×1080 image. Stored in bf16, this takes about 24 MiB; stored in uint8 (see #50411), it uses about 12 MiB. ### 3. For payloads smaller than one page (e.g., 0.5 MiB), the performance is nearly identical to that of ZMQ RPC. Hence, such small payloads offer no practical advantage. https://github.com/noooop/snippet/tree/main/benchmarks/pshm/h2d | Scenario | Data Path | ZMQ Usage | |----------|-----------|-----------| | **zmq** | CPU Source → ZMQ Send → ZMQ IPC → ZMQ Receive → CPU Buffer → Pin Memory → GPU | **Transfers full data payload** | | **zmq+shm (copy)** | Shared Memory → CPU Read → Clone → Pin Memory → GPU | **Control only** | | **zmq+shm (no-copy)** | Shared Memory → Zero-copy CPU Read → GPU | **Control only** | \"cpu_to_gpu_bandwidth\" ### 4. Removing the ZMQ IPC → CPU Buffer → Pin Memory steps in the ZMQ RPC path would bring significant performance gains. https://github.com/vllm-project/vllm/blob/5ac2684976ee22c04fe0d2f968c6cf6096b383f2/vllm/v1/serial_utils.py#L416-L425
## Let's do some calculations. https://github.com/noooop/snippet/tree/main/benchmarks/pshm/offline In my tests, for a typical Qwen2-VL-2B-Instruct model, the TTFT for one image is 558 ms. Transferring a 12 MiB image via ZMQ takes about 12 MiB / 2 GiB/s = 5.8 ms. Transferring a 12 MiB image via PSHM takes about 12 MiB / 5 GiB/s = 2.3 ms. 5.8 ms - 2.3 ms = 2.5 ms. Compared to 558 ms, this is completely negligible. Even if the speed were increased to 20 GiB/s, 12 MiB / 20 GiB/s = 0.58 ms — hm, that would only save 5 ms, still completely negligible. I feel like I'm tilting at windmills. (╯‵□′)╯︵┻━┻ ## Test Plan tests/models/multimodal/generation_ppl_test/test_qwen.py::test_pshm ## Test Result Model: Qwen/Qwen2-VL-2B-Instruct mm_device_do_normalize: True VLLM: dtype:torch.bfloat16 head_dtype:torch.bfloat16 41096328.0 Transformers: Constant 41081356.0 Difference (%): 0.03644475610785584 PASSED ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51349", "createdAt": "2026-08-07T03:14:17Z", "updatedAt": "2026-08-13T10:27:22Z", "timestamp": "2026-08-13T10:27:22Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "frontend", "multi-modality", "qwen", "nvidia", "mrv2" ], "author": "noooop", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51350", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "support rl feature : weight checker", "text": "## Purpose This PR extends the POST /weight_checker test suite so the weight-checker flow is exercised under both single-GPU (TP=1) and tensor-parallel (TP=2) configurations)Skip all cases now run for both TP settings via pytest.mark.parametrize. The coverage includes snapshot, compare, checksum, reset, and invalid-request error handling, plus a new assertion that the server reports the expected world_size for each TP configuration. ## Test Plan A single parameterized test file drives the dev server for each TP setting and verifies: -snapshot returns a structured response with a positive tensor count, and repeated snapshots are consistent. -compare requires a prior snapshot (400 otherwise), matches when weights are unchanged, and still matches after a sleep/wake cycle and after a generate request. -checksum returns per-tensor 64-character SHA-256 digests, is stable across calls, and does not clobber an existing snapshot. -reset rewrite the model weight to check update is correct. -error handling rejects missing/unknown actions cleanly while /health stays up. -world size matches the configured tensor_parallel_size. | Priority| Test Case | Main Coverage | | --- | --- | --- | | P0 | test_compare_without_snapshot_returns_400 | 400 when comparing without a snapshot; server stays healthy | | P0 | test_snapshot_then_compare_matches | Full match after snapshot (match=True, no mismatches) | | P0 | test_world_size_matches_tp | Server world_size equals the target TP (1/2) | | P0 | test_snapshot_returns_n_tensors | snapshot returns positive integer n_tensors, status=snapshotted | | P0 | test_compare_is_one_shot | compare is one-shot: a second compare (no new snapshot) returns 400 | | P1 | test_snapshot_twice_overwrites | Repeated snapshot overwrites the old one; n_tensors stays consistent | | P1 | test_checksum_returns_dict_of_hex | checksum returns a dict of 64-char hex SHA-256 digests | | P1 | test_checksum_stable_across_calls | checksum is idempotent/stable while weights are unchanged | | P1 | test_checksum_does_not_affect_snapshot | checksum does not write to or corrupt the stored snapshot | | P1 | test_sleep_wake_does_not_change_weights | Weights unchanged across sleep/wake (compare still matches) | ## Test Result default test tp=1 and 2 VLLM_SERVER_DEV_MODE=1 python -m pytest tests/entrypoints/serve/dev/test_weight_checker.py -v result： ```bash 52.35s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 35.10s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 4.07s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 4.06s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 3.47s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 2.95s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 2.83s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 2.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 2.71s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 2.66s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 2.58s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] 2.57s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 2.22s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 1.96s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 1.88s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 1.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 1.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 1.70s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 1.40s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 1.37s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 1.35s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 1.33s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 1.33s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 1.03s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.84s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.48s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.37s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 0.32s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.28s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 0.27s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 0.27s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 0.25s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.24s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.24s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 0.19s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.17s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] ``` or you can test tp =1 faster VLLM_TEST_TP=1 python -m pytest tests/entrypoints/serve/dev/test_weight_checker.py result ```bash 36.11s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 3.94s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 3.93s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 3.37s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 2.87s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 2.74s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 2.64s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 2.64s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 2.63s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 1.40s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 1.35s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 1.32s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 1.29s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.45s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.34s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 0.27s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.25s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.19s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.17s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51350", "createdAt": "2026-08-07T03:32:38Z", "updatedAt": "2026-08-13T12:24:41Z", "timestamp": "2026-08-13T12:24:41Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "frontend", "needs-rebase" ], "author": "shiyuan680", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51351", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Declare EAGLE cache-peek capability for KV cache managers", "text": "## Summary - declare whether each KV cache manager can safely peek past an EAGLE/MTP candidate boundary - resolve EAGLE cache-peek groups through the shared KV cache spec registry - exclude recurrent and otherwise unsupported cache managers from the fallback - fail closed when an unsupported group is explicitly annotated - apply the shared capability in the core, Mooncake store, and offloading paths Fixes #50630. ## Root cause When no KV cache group was explicitly annotated for EAGLE, several paths conservatively treated every group as an EAGLE group. This is safe only for managers that can drop the extra matched cache unit. Recurrent Mamba state cannot be rewound this way, so each caller had to exclude Mamba manually. The new manager capability makes this invariant explicit and defaults to unsupported. Full-attention and sliding-window managers opt in. ## Duplicate-work check Before opening this PR, issue #50630 and its comments were checked. Open PR searches by issue number and by the EAGLE/cache-peek/Mamba area found no PR addressing the same change. ## Tests Linux, Python 3.10.12: - Focused regression tests: `10 passed, 14 warnings in 2.71s` - Related test files: `109 passed, 14 warnings in 21.33s` - `ruff check`: passed - `ruff format --check`: 9 files already formatted - `git diff --check`: passed The warnings are existing PyTorch `torch.jit.script_method` deprecation warnings. No model evaluation or GPU benchmark was run because this is a control-plane KV-cache capability and boundary-selection change covered by unit tests. ## AI assistance AI assistance was used for test design, and PR drafting. The human submitter reviewed the changes and ran the reported tests.", "url": "https://github.com/vllm-project/vllm/pull/51351", "createdAt": "2026-08-07T03:36:18Z", "updatedAt": "2026-08-13T17:15:26Z", "timestamp": "2026-08-13T17:15:26Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "needs-rebase", "kv-connector" ], "author": "xinyuran", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51359", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Initialize DeepGemmQuantScaleFMT oracle lazily; bound QuantFP8 UE8M0 packed path to group_size 128", "text": "## Purpose Fix a crash and a latent kernel-contract violation in `QuantFP8`'s DeepGEMM UE8M0 fast path, found by running the FP8 kernel suite on an RTX PRO 6000 (SM120). `DeepGemmQuantScaleFMT.from_oracle()` asserts its cache is populated, but the cache is only filled as a side effect of `_lazy_init()` running for some other DeepGEMM wrapper (it's called at the end of `_lazy_init`, #30898). A `QuantFP8` built with an explicit `use_ue8m0=True` — directly, or via the DeepSeek V3.2 attention path — can hit `from_oracle()` before any DeepGEMM kernel wrapper has run and crashes: ```text AssertionError: DeepGemmQuantScaleFMT oracle cache not initialized ``` On Hopper this path is dormant (the oracle resolves to `FLOAT32_CEIL_UE8M0`, not `UE8M0`), which is why CI never sees it; on Blackwell (SM100/SM120) `tests/kernels/quantization/test_fp8_quant_group.py` fails with this assertion today. Fixing the init order then surfaces two more problems in the same fast path (added in #33047): 1. the packed branch was taken for **any** group size, but `per_token_group_quant_8bit_packed` only supports `group_size == 128` — `group_size=64` raises `RuntimeError: ... only supports group_size==128, got 64`; 2. the test still asserted plain float32 `[B, num_groups]` scales, so it never passed on Blackwell where the fast path legitimately returns packed int32 UE8M0 scales. This PR makes `from_oracle()` initialize on first use (running `_lazy_init()` first so the decision is based on resolved DeepGEMM symbols; without DeepGEMM it yields `FLOAT32`, matching `is_deep_gemm_e8m0_used()`), gates the packed branch on `group_size == 128`, and updates the test to unpack the int32 UE8M0 scales and compare them against `forward_native`. I didn't find existing work on this: searched `oracle cache not initialized`, `DeepGemmQuantScaleFMT`, `init_oracle_cache`, and `test_fp8_quant_group` across issues/PRs — only the original #30898 (refactor that introduced the assert) and #33047 (refactor that introduced the fast path) touch this code. AI assistance (GitHub Copilot CLI) was used to prepare this PR. ## Test Plan On an RTX PRO 6000 Blackwell (SM120, torch 2.11.0+cu130), where the oracle resolves to `UE8M0`: ```bash pytest tests/kernels/quantization/test_fp8_quant_group.py -q # fails before, passes after pytest tests/kernels/quantization/test_fp8_quant.py -q # adjacent suite pre-commit run --files vllm/utils/deep_gemm.py \\ vllm/model_executor/layers/quantization/input_quant_fp8.py \\ tests/kernels/quantization/test_fp8_quant_group.py ``` Plus a standalone repro (fresh process, `QuantFP8(group_shape=GroupShape(1,128), use_ue8m0=True).forward_cuda(...)` under a bare `set_current_vllm_config`), a bit-exactness check of the packed path against `forward_native` for group_size 128 shapes, and an e2e smoke of `Qwen/Qwen3.6-35B-A3B-FP8` (block-FP8, DeepGEMM MoE backend) before/after. ## Test Result - `test_fp8_quant_group.py`: **3 failed on main** (assert crash for `[64,1024,64]` / `[128,2048,128]`, then shape mismatch once the crash is fixed) → **11 passed** with this PR; still 11 passed with the two source fixes reverted-and-reapplied individually confirming each layer. - Standalone repro: crash → works; `group_size=64` with `use_ue8m0=True` now takes the unpacked path instead of raising. - Packed path bit-exactness vs `forward_native` (group_size 128): quantized data 100% equal, unpacked UE8M0 scales exactly equal, for `[16,1024]` and `[128,2048]`. - `test_fp8_quant.py`: 241 passed. - Qwen3.6-35B-A3B-FP8 greedy e2e smoke: output identical before/after. - ruff / ruff-format / typos / mypy hooks and `git diff --check`: clean. Note: `tests/kernels/quantization/test_block_fp8.py` has 100+ pre-existing failures on this SM120 machine (DeepGEMM matmul tolerances, unrelated to this change — identical failures on clean `main`). ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51359", "createdAt": "2026-08-07T05:06:37Z", "updatedAt": "2026-08-12T18:27:51Z", "timestamp": "2026-08-12T18:27:51Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "bug", "ready", "quantization" ], "author": "BabyDrangoner", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51360", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add reusable TP1 initialized-engine snapshots", "text": " ## Purpose @elizabetht's RFC #34303 and #37921/#37925 split CUDA checkpoint work into in-process primitives and leave CRIU disk persistence for a later phase. #35934 proposes a CRIU-safe engine suspend and resume path, while PRs #42790 and #46089 propose provider-owned pod snapshot hooks. @fergusfinn's data in #34303 also showed that full initialized snapshots can work, with large images and stale external TCP connections as central costs. @simon-mo's closed, reference-only #40323 autoresearch showcase also included inert snapshot scaffolding for a pre-model image of Python imports and a CUDA context. It explicitly excluded model weights, model configuration, compile state, and CUDA graphs. This PR instead implements and measures a host-owned artifact captured after TP1 engine initialization and a deterministic generation canary, before HTTP bind. It is also the measured successor to my closed PR #48996. PR #48996 captured only the import stage. Its matched warm comparison was flat at about 29.08 seconds versus 29.09 seconds, n=3 per arm, because the expensive engine state still had to be constructed after restore. This PR moves the boundary after engine initialization and a deterministic generation canary, but before the HTTP listener binds. The user-facing workflow is: ```bash vllm snapshot create MODEL --snapshot-dir PATH --revision MODEL_SHA [serve args] vllm snapshot inspect PATH vllm snapshot restore PATH --host 127.0.0.1 --port 8000 ``` Creation captures the initialized TP1 process tree and CUDA state with CRIU and NVIDIA's CUDA plugin. Restore validates the saved host and software identity before CRIU runs, restores the tree, binds the requested HTTP address, and checks the saved token again before returning. It does not silently fall back to ordinary startup. The lightweight `vllm_bootstrap` wrapper is intentional. Importing the normal CLI loads the vLLM runtime and PyTorch before CRIU can restore the saved process, consuming much of the activation win. Only `vllm snapshot restore` takes the lightweight path. Every other `vllm` command delegates to the existing CLI entry point unchanged. This is adjacent to, but does not replace, the in-process suspend and resume work. The current product boundary stays TP1 so it can preserve the initialized process tree without claiming multi-GPU communicator restore. ## Product envelope and tradeoffs This feature is opt-in and intentionally exchanges local disk and host privilege for activation time. - Linux x86-64, one NVIDIA GPU, TP1, PP1, and DP1 only. - Same host, GPU, driver, kernel, Python, PyTorch, vLLM source and binary, model revision, engine arguments, selected environment, and plugin identity. - Root or passwordless `sudo`, CRIU, `cuda-checkpoint`, and NVIDIA's CRIU CUDA plugin. - `io_uring` disabled before vLLM starts. For an unprivileged process, `kernel.io_uring_disabled=1` is sufficient; mode `2` disables it host-wide. - One plaintext HTTP frontend without API authentication, TLS, custom middleware, or a Unix domain socket. - No external established TCP peer at capture. Tree-owned internal connections are recorded. - Private local artifact that may contain process memory, model state, and secrets. - No concurrent restore of one artifact and no automatic normal-start fallback. The documentation explains these constraints, the sensitive-artifact boundary, and the disk, preparation-time, privilege, compatibility, and portability costs. ## Update, August 8 After the original `16999dbe788f` head passed full CI, I added one opt-in artifact-size path to the same product. `vllm snapshot create --minimize-snapshot-state` uses the existing sleep level 2 lifecycle after the correctness canary and before capture. It releases the model-weight and KV-cache allocations, then reloads the weights and recreates the KV cache after restore and before binding HTTP. The default full-state path is unchanged. The dense A10 decision cell used experimental source `4154f9b61775048f7cb679c4581706fd620426ff`, Qwen3-8B FP16 TP1, warm model-weight pages, and a cold snapshot artifact: | Path | Allocated artifact | Activation to first correct token | | --- | ---: | ---: | | ordinary | 24,231,231,488 B | 23.7633 s median, n=3 | | minimized | 5,144,281,088 B | 21.2396 s median, n=3 | The minimized path removed 19,086,950,400 allocated bytes, or 78.8 percent, and reduced the activation clock by 2.5237 seconds, or 10.6 percent. Every accepted restore returned the exact oracle token, completed at least six post-restore requests, and left no GPU process behind. The public option was then tested on source `e790b9b2018a6727bea069fa11bf1d1e64626e2b` with Qwen3-0.6B FP16 TP1 on an RTX 4090 Laptop GPU. It reduced the allocated artifact from 11,623,870,464 to 3,698,221,056 bytes, or 68.2 percent. The ordinary artifact restored once and the minimized artifact restored twice. Each restore passed the exact-token oracle plus three additional requests and drained the GPU. Page residency was uncontrolled, so this is a mechanism smoke and not another performance claim. Feature production-code changes since that tested source are a reviewer-requested hardening of existing CRIU link-remap validation and the lightweight package rename from `vllm_cli` to `vllm_bootstrap` after a reviewer found a collision with the independent `vllm-cli` PyPI project. Neither changes the measured activation path. Deterministic regressions cover the prior remap check/read race and the console entry-point/package-discovery boundary. Current head `f86a1e269b` rebases the tested implementation plus those two review fixes onto vLLM main `8a9f9f762a`; the remaining feature changes since the tested source are documentation and tests. The five commits added since the previous reviewed base touched none of this PR's 16 files. The state-minimization measurements use a different model and starting state from the original 9.4234-second A10 restore result below. The savings are not additive. Warm model-file pages can make the smaller artifact faster, while cold or slow storage can make reloading weights slower than restoring the full process image. ## Test plan Focused CPU tests: ```bash PYTHONPATH=$PWD pytest -q \\ tests/snapshot \\ tests/entrypoints/unit_tests/test_launch_cli.py ``` Changed-file repository checks: ```bash pre-commit run \\ --from-ref 8a9f9f762a47e24ed8b48a8472f6757346cb30ae \\ --to-ref HEAD ``` The RTX 4090 Laptop GPU cell used: - source `6d4507450d6ad2b226b5883904e009b9d11617ec` - base `dd856e48bbf969e3f0e561e8c76f6e92c76e0795` - RTX 4090 Laptop GPU, 16,376 MiB, driver 595.71.05 - Ubuntu kernel 7.0.0-28, Python 3.12.3, torch 2.13.0+cu132, CRIU 4.2.1 - `Qwen/Qwen3-0.6B` at revision `c1899de289a04d12100db370d81485cdf75e47ca` - FP16, maximum model length 512, GPU memory utilization 0.50, TP1, PP1, and DP1 - warm Hugging Face, TorchInductor, Triton, and vLLM caches - `VLLM_NO_USAGE_STATS=1` The Python source was the exact clean feature commit. The precompiled native payload was reused from `adc3e03517d2e7333a3bb2083bb4d394a2986876` after a path-limited diff found no C++, CUDA, Rust, FlashAttention, or vendored native-source change. The only `setup.py` change was an optional dependency minimum. The final NVIDIA A10 acceptance cell used: - exact source `16999dbe788f2a1d7e6b0de243e874193388e352` - base `c810e5ee9976ad86b81d1277b53e76d0ee639414` - NVIDIA A10, 23,028 MiB, driver 570.148.08 - Ubuntu 22.04.5, kernel 6.8.0-60, Python 3.10.12, torch 2.13.0+cu129, CUDA 12.9, CRIU 4.2.1 - the same model revision and engine settings as the 4090 cell - warm Hugging Face, TorchInductor, Triton, and vLLM caches - documented unprivileged `kernel.io_uring_disabled=1` This exact head contains the two repairs selected by the first A10 run. It captures the caller's environment before vLLM injects internal defaults, and it rejects active `io_uring` descriptors with an actionable error before CRIU runs. ## Test result Focused tests: ```text 65 passed ``` All applicable changed-file pre-commit hooks passed, including Ruff, formatting, typos, markdownlint, Python 3.10 mypy, SPDX, lazy imports, forbidden imports, and configuration checks. Each hardware cell restored one artifact five times and ran five matched ordinary warm starts. Both arms used the same source, model revision, caches, engine settings, console entry point, clock boundary, and correctness request within that cell. ### RTX 4090 Laptop GPU | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 15.7790 s | 5 | | initialized snapshot restore | 5.5488 s | 5 | | reduction | 10.2302 s, 64.8% | 5 per arm | Snapshot creation took 37.50 seconds and recorded 11,595,128,620 artifact bytes, about 11.60 GB. The measured break-even was the fourth repeated activation. ### NVIDIA A10 | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 23.9121 s | 5 | | initialized snapshot restore | 9.4234 s | 5 | | reduction | 14.4888 s, 60.6% | 5 per arm | Snapshot creation took 42.3594 seconds and recorded 15,580,855,788 artifact bytes, about 15.58 GB. The measured break-even was the third repeated activation. Every restore and warm control in both cells returned token ID `12095` with text value \" Paris\" for the prompt `The capital of France is`. The final A10 cell exercised both product defects found by the first A10 run. With the host default `kernel.io_uring_disabled=0`, creation stopped before CRIU and named the owning PIDs. Mode `1` then completed creation and all five restores. The saved manifest omitted vLLM-injected internal defaults, while changing caller-provided `VLLM_USE_FLASHINFER_SAMPLER` still produced `snapshot mismatch: environment` before CRIU ran. The runner restored the host sysctl and left the GPU process list empty. These break-even points belong only to their measured machine, model, and cache state. Larger models, slower disks, or different GPU memory settings can change both artifact size and restore time. Both hardware cells disabled usage reporting. #51219 separately closes the default telemetry HTTP session, so this PR does not duplicate that fix or claim a default-telemetry hardware result. This PR does not claim a model-quality improvement. The generation canary checks that the restored engine reproduces the saved deterministic token before it is declared ready. AI assistance was used. I reviewed the changes and can defend the implementation, constraints, and measurements. ## Checklist - [x] The purpose names the RFC and adjacent open work. - [x] The test plan includes focused tests, repository checks, and hardware. - [x] The result includes matched before and after measurements with n labels. - [x] The documentation covers setup, security, compatibility, and tradeoffs.", "url": "https://github.com/vllm-project/vllm/pull/51360", "createdAt": "2026-08-07T05:06:49Z", "updatedAt": "2026-08-13T18:00:06Z", "timestamp": "2026-08-13T18:00:06Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "frontend", "ready" ], "author": "matteso1", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51382", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Build] Pin minimal DeepGEMM SM120 and SITU revision", "text": "## Purpose Addresses #47436 with a minimal DeepGEMM dependency revision that combines: - `a6b593d2826719dcf4892609af7b84ee23aaf32a`, the revision previously used by vLLM for SM120 support. - The SITU change from `f5a76426fa084087169693fd0cd815223576d6e9`, cherry-picked directly on top as `586032eba787dd99e3d1c069f8514ab01ca64906`. - A one-line follow-up importing `Union`, which is referenced by the cherry-picked MegaMoE annotations. The resulting DeepGEMM branch is [`nv_dev+situ+0810`](https://github.com/vllm-project/DeepGEMM/tree/nv_dev%2Bsitu%2B0810), pinned at [`9e8903799beb0b65d88e5ca08940dd5cd712c7d2`](https://github.com/vllm-project/DeepGEMM/commit/9e8903799beb0b65d88e5ca08940dd5cd712c7d2). Both vLLM DeepGEMM pins are updated together. No other vLLM files are changed. ### Duplicate-work check #50796 targets the same high-level SM120 + SITU outcome, but pins the later `nv_dev+situ` merge commit `5f33a180`. This PR instead keeps the smaller `a6b593d` base and the focused SITU cherry-pick. Other open DeepGEMM pin PRs found by the required duplicate search do not use this minimal construction. ## Test plan and results Passed locally in vLLM: ```bash bash -n tools/install_deepgemm.sh .venv/bin/pre-commit run --files cmake/external_projects/deepgemm.cmake tools/install_deepgemm.sh git diff --check ``` The two vLLM pins were checked for exact equality at `9e8903799beb0b65d88e5ca08940dd5cd712c7d2`. Passed on the DeepGEMM branch: ```bash uvx ruff check --select F821 deep_gemm/mega/__init__.py uv run --no-project python -c 'import ast, pathlib; p=pathlib.Path(\"deep_gemm/mega/__init__.py\"); ast.parse(p.read_text(), filename=str(p))' git diff --check ``` Not run locally: CUDA compilation, SM120/B200 hardware validation, or model evaluation. A fresh vLLM CI run is required for the new pin. ## AI assistance AI assistance was used for CI-log analysis, the one-line dependency fix, static validation, and drafting this update. The submitting human must review every changed line and confirm the CUDA/model results before merge.", "url": "https://github.com/vllm-project/vllm/pull/51382", "createdAt": "2026-08-07T09:18:25Z", "updatedAt": "2026-08-13T05:23:31Z", "timestamp": "2026-08-13T05:23:31Z", "metrics": { "reactions": 4, "comments": 11 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "khluu", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51403", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Route non-divisor small-head MLA decode to Gluon on gfx950", "text": "Fixes #51232. Kimi-K3 has 96 MLA query heads; at TP8 each rank gets 12 local heads. 12 is not a divisor of 16, so `use_gluon_decode` returned `False` in \"auto\" mode, routing single-token decode to the padded ASM persistent path. That path pads 12 heads to 16 and asks AITER for a `{bf16, fp8, gqa=16, qseqlen=1}` kernel, which does not exist on gfx950 — every rank crashes with `cannot get heuristic kernel` before the server starts. The Gluon MLA decode kernel handles any head count natively (no padding), so the fix is to prefer Gluon for ALL small-head counts on gfx950 in \"auto\" mode, not just divisors of 16. On gfx942 (no Gluon build) the ASM path is still used for all counts — no behavior change. `VLLM_ROCM_AITER_MLA_ASM_PADDING=asm` still forces the padded ASM path if needed. ### Changes - `rocm_aiter_mla.py`: remove the `m % num_heads == 0` gate from the \"auto\" return in `use_gluon_decode` - `test_rocm_aiter_mla_head_padding.py`: update 3 assertions so non-divisor counts (12, 6) assert `is on_gfx950` instead of `not`", "url": "https://github.com/vllm-project/vllm/pull/51403", "createdAt": "2026-08-07T14:38:14Z", "updatedAt": "2026-08-13T15:20:41Z", "timestamp": "2026-08-13T15:20:41Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "zvigrinberg", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51406", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Enable fused QK-norm+RoPE+gate Triton kernel for Qwen3-Next/Qwen3.5", "text": "## Purpose `Qwen3NextAttention` fuses the gated QKV split + QK-RMSNorm + (partial) NeoX RoPE + gate copy into a single Triton kernel (`fused_qk_rmsnorm_rope_gate`, added in https://github.com/vllm-project/vllm/pull/44176). The kernel is pure Triton and backend-agnostic, but it is gated behind `current_platform.is_cuda()`, so ROCm falls back to the slow multi-op eager path, even though the kernel runs correctly and fast on AMD GPUs. This PR relaxes that one-line guard to `is_cuda_alike()` so ROCm gets the same fusion. It affects the whole `Qwen3NextAttention` family: Qwen3-Next, Qwen3.5, Qwen3.5-MoE Measured on MI325X (gfx942), bf16, against the model's real eager branch (`GemmaRMSNorm` + the `rotary_emb` custom op), 300 timed iters after 50 warmup: | Seq len (tokens) | Eager (µs) | Fused (µs) | Speedup | |---|---|---|---| | 1 (decode) | 279.4 | 33.0 | 8.47× | | 8 | 279.3 | 33.6 | 8.32× | | 64 | 281.7 | 33.9 | 8.32× | | 512 | 277.4 | 33.7 | 8.22× | | 4096 | 562.3 | 138.7 | 4.05× | Note: https://github.com/vllm-project/vllm/pull/47719 is an opt-in AITER path that additionally fuses the KV-cache write. ## Test Plan Add a test case in `tests/kernels/core/test_fused_qk_norm_rope.py` ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51406", "createdAt": "2026-08-07T15:04:43Z", "updatedAt": "2026-08-13T04:09:32Z", "timestamp": "2026-08-13T04:09:32Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "qwen" ], "author": "xuebwang-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51415", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual act quant fusion llama", "text": "Starts the `ActivationQuantFusionPass` manual-fusion migration (RFC #43224, specific tracker https://github.com/vllm-project/vllm/issues/43501) on the producer side of the `QuantizedActivation` contract (#44260). Adds `maybe_fused_act_quant`: given an activation and the linear it feeds, it emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kernel when the linear advertises a consumable `input_quant_key`, and falls back to the plain activation otherwise. Llama's MLP forward calls it for `down_proj` — the model-code change is one line. When manual fusion fires the `silu_and_mul` pattern is already consumed, so the compiler `ActivationQuantFusionPass` finds nothing and the two never double-fuse. Only `(SiluAndMul, kFp8StaticTensorSym)` is registered for now; other schemes/activations are one table row each. Not a duplicate: builds on the landed contract (#44260) and is the activation-quant counterpart to the AR+RMSNorm manual fusion in #45855; no open PR covers the producer side. ## Test Smoke tested on `RedHatAI/Llama-3.2-1B-Instruct-FP8` (default compiled + CUDA-graph path): all 16 `mlp.down_proj` layers take the fused branch (`input_quant_key == kFp8StaticTensorSym`) and generations are coherent — the `QuantizedActivation` flows through the compiled `down_proj.forward` into `apply_weights` with no graph break. TP2 coherence and an A/B vs. the compiler pass are follow-ups. AI assistance (Claude) was used; all changed lines are human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/51415", "createdAt": "2026-08-07T15:44:04Z", "updatedAt": "2026-08-13T17:07:15Z", "timestamp": "2026-08-13T17:07:15Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "llama", "quantization" ], "author": "mgoin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51437", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 latent-MoE: overlap the shared all-reduce with the routed up-projection", "text": "## Purpose Kimi-K3's latent MoE combines a **routed** expert output — produced in a compressed latent space, then up-projected back to hidden dim — with a **shared** expert output. Under tensor parallelism both need an all-reduce. Today the ROCm runner (`ROCmLatentMoERunner`) only implements the column-parallel tail (`_shard_up_proj_tail`, \"Tier 2\"), which folds a sharded up-projection into the shared partial before a single reduce. That tail is efficient for prefill-sized batches but gives up all overlap: the reduce must follow the accumulate. This PR adds a token-count-dispatched overlap tail so decode-sized batches stop leaving the shared all-reduce on the critical path: - **New Tier 1 — `_overlap_allreduce_tail`.** All-reduce + RMSNorm the latent, then run the full replicated up-projection GEMM on the default stream while the shared-expert all-reduce runs on the aux stream (`maybe_execute_in_parallel` + `aux_stream()`), and add. For batches under `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` the shared all-reduce is hidden behind the up-proj GEMM. - **Tier selection — `_select_tail_tier`.** Small batches take the overlap tail; larger batches keep the column-parallel tail; when the up-projection rows do not divide evenly across ranks (or the routed scale is non-unit) the runner falls back to the overlap tail, which is correct at any shape. - **`_allreduce_norm_latent_out`.** Fused AR+RMSNorm for the latent — collapses the latent all-reduce and its RMSNorm into a single AITER op, with a plain `all_reduce` + `RMSNorm` fallback. This helper is backed by a small addition to the base runner (`moe_runner.py`): a module-level `_aiter_fused_ar_rmsnorm` binding (via `rocm_aiter_ops.get_fused_allreduce_rmsnorm_op()`) plus a `_get_zero_residual` buffer helper. The underlying AITER op already exists in `_aiter_ops.py`; the base runner just did not expose it before. The native (non-fused) path is unchanged: the fused path is still gated on `tp_size > 1`, an un-reduced combine output, a shared expert, no sequence parallelism, and unit routed scale — anything else defers to the base `MoERunner.forward`. No existing base-runner behavior changes; the only base-runner edit is the additive AR+RMSNorm helper above, covered by a new unit test. Relates to #50682 (ROCm/AMD Kimi-K3 gap and roadmap tracking → Performance Optimization). Tracked as PR #51437. ### Files changed - `vllm/models/kimi_k3/amd/latent_moe_runner.py` — the overlap tier, tier selection, and `_allreduce_norm_latent_out`. - `vllm/model_executor/layers/fused_moe/runner/moe_runner.py` (+90/−7) — additive `_aiter_fused_ar_rmsnorm` binding + `_get_zero_residual` helper (no behavior change to existing paths). - `tests/models/kimi_k3/test_amd_latent_moe_runner.py` — tier-selection unit tests + multi-GPU parity for both tails. - `tests/model_executor/test_moe_runner_fused_ar_rmsnorm.py` — unit test for the new base-runner helper. ### This feature has no kernel dependency The overlap and column-parallel tails contain **no activation** — they are reductions, an up-projection GEMM, and an add. The `situ`/`situv2` activation lives entirely in the stage-1 expert GEMM (`_forward_entry`), which runs **before** tier selection and is identical across Tier 1, Tier 2, and the upstream baseline. So this PR does not depend on any activation-kernel change, and the unit + multi-GPU parity tests — which feed synthetic routed/shared tensors straight into the tail methods and never invoke the expert GEMM — pass regardless of the AITER build. **Environment note (not a feature dependency):** serving Kimi-K3 int4 on gfx942 at all requires **ROCm/aiter#4471** (\"[FlyDSL] Support SiTUv2 in the packed-int4 MoE stage1 epilogue\"), because Kimi-K3 uses `hidden_act = situ` and the current packed-int4 FlyDSL stage1 otherwise raises `NotImplementedError: split-K stage1 activation supports only 'silu', got 'situv2'` (or silently computes SiLU). This affects the **upstream Tier-2 baseline (#51253) exactly as much as this PR** — it is a prerequisite for reproducing the *end-to-end serving* numbers below, not something this feature introduces or relies on. On an AITER without split-K SiTUv2 support, force the non-split-K stage1 with `AITER_KSPLIT=1`. ### Relationship to #50657 #50657 (\"[ROCm][Perf] Fuse and FP8-pack Kimi-K3 latent MoE output tail\") targets the same boundary but with a different mechanism: a dedicated AITER FlyDSL fused-tail kernel plus optional FP8 packing of the up-projection weight, which depends on unmerged AITER PRs (ROCm/aiter#4496, #4503). This PR is complementary — it schedules existing in-tree collectives/GEMMs across two streams and adds an AITER fused AR+RMSNorm helper to the base runner (backed by an op already present in `_aiter_ops.py`, so no new kernel). The two overlap only on the `runner_cls=` wiring line in `amd/linear.py` (already present on `main`); if both land, that single line needs reconciling to a runner that can select between the two strategies. ## Test Plan Unit (no GPU) — tier-selection and fused-path gates: ```bash .venv/bin/python -m pytest -q \\ tests/models/kimi_k3/test_amd_latent_moe_runner.py \\ -k \"not tp4 and not tp8\" ``` Multi-GPU arithmetic parity — each tail must equal the replicated up-projection: ```bash # 4- and 8-GPU ROCm host .venv/bin/python -m pytest -q tests/models/kimi_k3/test_amd_latent_moe_runner.py ``` The multi-GPU suite covers, at TP4 and TP8: - `test_overlap_tail_tp{4,8}_matches_replicated_projection` — new Tier 1 output equals `shared + up_proj(rms_norm(all_reduce(routed)))`. - `test_shard_tail_tp{4,8}_matches_replicated_projection` — Tier 2 unchanged. - `test_shard_tail_tp4_writes_only_its_own_shard` — the column-parallel tail writes only this rank's hidden shard before the final collective. End-to-end serving parity (8× MI355X, `gfx950`, `moonshotai/Kimi-K3`): ```bash vllm serve /model --served-model-name moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 --trust-remote-code --moe-backend auto \\ --gpu-memory-utilization 0.9 --max-num-seqs 32 \\ --max-num-batched-tokens 4096 --reasoning-parser kimi_k3 ``` Compare the candidate against `main` (baseline = Tier 2 only). Exercise both tiers by sweeping batch size around `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` (small decode → overlap tier; large prefill → column-parallel tier) and force the column-parallel path with `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`. Note: EP must be **off** for the fused tail to engage — with expert parallel enabled the MoE combine already reduces the fused output, so the runner takes the native path and neither tail runs. ## Test Result _All numbers below are placeholders — fill from the runs described in the Test Plan. Environment: 8× MI325X (`gfx942`), `moonshotai/Kimi-K3`, `int4_per_group_32`, TP=8, EP off. Serving requires an AITER that handles SiTUv2 (ROCm/aiter#4471), or `AITER_KSPLIT=1` to force the non-split-K stage1 — an environment prerequisite for K3 on gfx942, not specific to this feature._ **Correctness** - Unit (`-k \"not tp4 and not tp8\"`): _TODO PASS/FAIL_ (`N passed`). - Multi-GPU parity (TP4 + TP8, overlap & shard tails): _TODO PASS/FAIL_. - Accuracy parity vs `main` (fused tail must not change outputs within tolerance): _TODO — e.g. gsm8k / small lm_eval, baseline vs candidate._ **Decode latency / throughput** — batch below `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` so the overlap tail (Tier 1) is active. Baseline = `main` (Tier 2 only); Candidate = this PR. | Batch (tokens) | Tail tier | Baseline TPOT (ms) | Candidate TPOT (ms) | Δ % | Baseline tok/s | Candidate tok/s | Δ % | |---:|:--|---:|---:|---:|---:|---:|---:| | 1 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 8 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 32 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 128 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 256 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 4096 (prefill) | column-parallel | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | **Overlap evidence** — trace excerpt showing the shared-expert `all_reduce` running on the aux stream concurrently with the routed up-projection `mm` on the default stream (confirms the reduce is hidden, not merely reordered): _TODO — paste kernel timeline._ ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/51437", "createdAt": "2026-08-07T19:43:05Z", "updatedAt": "2026-08-13T01:41:58Z", "timestamp": "2026-08-13T01:41:58Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "mpashkovskii", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51459", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Fix and extend PR/issue auto-labeling", "text": "## Purpose Auditing the labeling setup against the actual repo turned up rules that had stopped matching the code they were written for, plus issue-side labeling that covered only 6 of 62 labels. ### Rules that were missing real work - **`deepseek` missed the `vllm/models/` tree.** Newer models live in `vllm/models//` rather than `vllm/model_executor/models/`. The rule only knew the old location, so only the title match applied — **64 of 99 PRs** touching `vllm/models/deepseek_v4/` went unlabeled. Same gap for `deepseek_v32`. - **`speculative-decoding` missed the Model Runner V2 subtree.** `vllm/v1/worker/gpu/spec_decode/` has seen 94 PRs in 6 months, more than the original `vllm/v1/spec_decode/` at 81. This also folds `dflash` and `dspark` under the umbrella label, per review feedback that both are spec decoding techniques. ### Rules that could never fire Four conditions used `files=` (exact match) with regex syntax, or carried a doubled backslash that YAML passes through literally: ```yaml - files=^examples/features/speculative_decoding/ # never matches - files~=^\\\\.buildkite/intel_jobs/ # never matches ``` A further 13 conditions pointed at files that have since moved or been deleted — `harmony` moved to the Rust frontend, `pallas` is gone. Repointed where the code moved, removed where it is gone. ### Retired: `new-model` The rule fired on any PR touching a model file **and** `registry.py`. `registry.py` is edited for renames, aliases, refactors and removals, not just additions. Of the 25 most recent PRs carrying the label, 10 added a model. The rest included: - `[Model] Remove Ouro`, `[Model] Remove Plamo2`, `Remove PersimmonForCausalLM and FuyuForCausalLM` — PRs that **delete** models - `[model registry] some simple typos` - `[Perf] Optimize clamp to clamp_`, `[Kernel] ReplaySSM: cache SSM inputs`, `Revert \"Blackwell decode optimizations\"` Whether a new architecture was registered depends on what changed *inside* `registry.py`. Mergify matches file paths, not diff contents, so this rule cannot be made accurate — tightening the paths does not help. 111 open PRs currently carry the label. The label remains on issues, applied by the New Model template, where the requester states the intent explicitly. ### Rule that fired too often `nvidia` matched `requirements/cuda.txt`, so dependency bumps were tagged as NVIDIA backend work. Measured across 200 merged PRs, 17 carried the label and this was the one clear miss. ### New rules `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`, `DSv4`, `dflash`, `torch.compile`, `ray`, `vllm-ir`. ### Issue labeling Extended from 6 labels to 31. Terms match **titles only** by design: issue bodies carry pasted `collect_env` output that names unrelated hardware and libraries, which tags the reporter's environment rather than the topic. ### Guard against recurrence `tools/pre_commit/check_label_rules.py` fails when a file condition matches nothing in the tree, and runs via pre-commit when `mergify.yml` changes. It caught two bugs in this change while it was being written. ### Also removed: the New Model issue template `.github/ISSUE_TEMPLATE/600-new-model.yml` is deleted. - Usage fell from 155 issues in 2025 to 30 in 2026, only 3 since July - It applied a label named `new model`, but the label is `new-model`. GitHub silently skips a label that does not exist, so requests went untagged from the March rename onward — five months, unnoticed - 292 of the 304 ever filed are closed - Model support arrives as planned work rather than as community requests `blank_issues_enabled` is `false`, so model requests route to Feature Request. **Nothing applies `new-model` after this change.** The label is left in place so it stays on the 917 PRs and 304 issues carrying it; deleting the label itself is a separate call. ## Before merging **The five new labels must be created first** — `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`. Mergify errors on a label that does not exist. Two related items a PR cannot do: - `mrv1-only`'s description currently reads \"apply only to Model Runner V2 (not applicable to Model Runner V2)\" — says V2 twice. - `DSv4` has a rule here mirroring the `kimi`/`k3` split, but whether to keep it as a distinct label is a maintainer call. ## Test Plan Rules were validated against the real file tree and against real issues and PRs, rather than by inspection. ```bash # every file condition in mergify.yml matches at least one tracked file python tools/pre_commit/check_label_rules.py pre-commit run check-label-rules --files .github/mergify.yml pre-commit run ruff-check --files tools/pre_commit/check_label_rules.py pre-commit run ruff-format --files tools/pre_commit/check_label_rules.py ``` ## Test Result ``` tracked files: 6430 rules: 44 file conditions matching >=1 file: 154 file conditions matching NOTHING: 0 Check auto-label rules still match real files............................Passed ruff check...............................................................Passed ruff format..............................................................Passed ``` Negative test: injecting a condition pointing at a nonexistent path makes the check exit 1 and name the offending rule. New rules were spot-checked against the tree — `glm` matches 21 files and correctly ignores `midashenglm.py`, which contains the substring \"glm\" but is an unrelated model. Issue rules were replayed over 600 real issues: 43% now receive at least one label, and the existing rules agreed with hand-applied labels (`kimi` 35 fires / 31 already labeled, `rocm` 34 / 33, `k3` 27 / 26). No model evaluation is included because this change touches only CI labeling configuration. It does not affect inference, model output, accuracy, or serving. ## Note on AI assistance AI assistance (Claude Code) was used to audit the existing rules, measure PR and issue volumes, and draft these changes. Every measurement quoted above is reproducible with the commands shown. I have reviewed each changed line.", "url": "https://github.com/vllm-project/vllm/pull/51459", "createdAt": "2026-08-07T22:57:38Z", "updatedAt": "2026-08-13T16:22:38Z", "timestamp": "2026-08-13T16:22:38Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "ready", "ci/build" ], "author": "jcotant-inferact", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51464", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] update triton in base docker for gluon compatibility", "text": "## Purpose To pick up https://github.com/ROCm/triton/pull/960 for fixing the gluon mla kernel compilation problem related to DistributedLinearLayout was rejected in the previous triton commit. With this, the gluon kernel can compile and serve K3 with accuracy. Context: Initially vllm/vllm-openai-rocm:nightly docker failed at the first MLA decode with a Gluon frontend error inside aiter/ops/triton/gluon/mla_gluon.py: triton.compiler.errors.CompilationError: at 319:8: ... gl.amd.cdna4.async_copy.buffer_load_to_shared(buf_q_pe, Q_pe, offs_q_pe, ...) ^ expected offsets type layout to be BlockedLayout or SliceLayout ## Test Plan Verified aiter op test passed. Verified Kimi-K3 worked without accuracy issue after building triton in vllm nightly docker. ## Test Result K3 no-spec serve, gluon-MLA path Startup complete, no compile error, no memory fault GSM8K full set (1319q)+ 94.5% ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51464", "timestamp": "2026-08-12T16:26:18Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "hongxiayang", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:51466", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Fix DSD K-lookup to count sampling-only requests, preventing K thrashing at batch boundaries", "text": "> [!IMPORTANT] > **Withdrawn after controlled follow-up (2026-08-13).** This PR correctly > demonstrates that total-request and sampling-request lookup can choose > different K values, but I had not established that sampling-only was the > correct policy or that it reduced K thrashing. A later 25-pair controlled > experiment found no target workload meeting the predeclared benefit bar, > heavy-prefill workloads favored the existing total-count policy in every > pair, and complete-trace K transitions were 50 with sampling-only versus 40 > with total-count. I am closing the PR and retaining the original description > below as historical context. See the closure comment for details. ## Description The dynamic speculative decoding (DSD) K-lookup in `scheduler.py` uses `len(num_scheduled_tokens)` to index into `dynamic_sd_lookup`. This counts **all** scheduled requests, including mid-prefill/chunked-prefill requests. When prefills are mixed with decodes, they temporarily inflate the batch size and cause K to thrash between values as requests enter and leave prefill — a contributing factor to the catastrophic aggregate-throughput collapse reported in #49548. ### Example With schedule `[[1, 2, 2], [3, 16, 0]]` (K=2 for batch ≤ 2, K=0 for batch ≥ 3): | Step | Requests | Old count | Old K | New count | New K | |---|---|---|---|---|---| | 2 decode + 1 prefill (512 tok) | 3 | 3 | **0** | 2 (decode-only) | **2** | | 2 decode + 1 prefill (1 tok chunk) | 3 | 3 | **0** | 2 (sampling-only) | **2** | The old code flips K from 2→0 every time a prefill joins 2 decodes, then back to 2 when it leaves. Each flip changes the verify-token count, forward shape, and CUDA graph dispatch. ### Fix Count only requests that will **sample this step** (decode or final-prefill-chunk that reaches sampling), using the same phase-transition predicate the scheduler uses elsewhere: ```python req.num_computed_tokens + num_tokens >= req.num_tokens + req.num_output_placeholders ``` This correctly handles: - Pure decode requests - Mid-prefill chunked requests (excluded — will not sample) - Final prefill chunks that reach sampling (included — will sample) - 1-token prefill chunks that are NOT final (excluded) - Spec-padded decode requests (impossible in DSD mode — spec padding is gated on `dynamic_sd_lookup is None`) If no requests reach sampling (all mid-prefill), falls back to `len(num_scheduled_tokens)` to preserve original behavior. ## Benchmark Dual RTX 5090, TP=2, Qwen3.6-35B-A3B-Uncensored-FP8, MTP k=2, `VLLM_USE_V2_MODEL_RUNNER=1` + #49652 to preserve FULL CUDA graphs. | Scenario | Before (±stdev) | After (±stdev) | Δ | |---|---|---|---| | 4-session × 256 tok | 195 ± 164 t/s | **190 ± 49 t/s** | Variance −70% | | 8-session × 256 tok | 308 ± 67 t/s | **412 ± 163 t/s** | +34% mean | | Single-stream 32k | 285 ± 3 t/s | **289 ± 1 t/s** | Unchanged | The variance reduction confirms K thrashing is eliminated. The multi-session mean remains below no-spec baseline under long context — the cause is under investigation (see our correction comments on #49986 and #49548). The gap is context-length dependent and its root cause has not yet been identified. ## Complementary to #49652 This fix targets the scheduler-side K-selection pathology (#49548 H3). PR #49652 fixes the worker-side MRV2 CUDA graph capture crash (#48494). Both are needed for functional DSD with FULL CUDA graphs on MRV2. ## Test - `py_compile vllm/v1/core/sched/scheduler.py` — PASS - `pytest test_dynamic_sd_sampling_only_k_lookup` — 1 passed (8/8 edge cases) - Live runtime verified on dual RTX 5090 with Qwen3.6-35B-A3B FP8 + MTP + MRV2 + DSD Note: vLLM native scheduler tests require full dev install (`pip install -e .`). The standalone test validates the core predicate logic. Fixes part of #49548. ## AI Assistance Disclosure AI tools (codex CLI) were used in the development of this PR. All changes were reviewed and tested by the human submitter (Greg Weyer).", "url": "https://github.com/vllm-project/vllm/pull/51466", "createdAt": "2026-08-08T00:14:53Z", "updatedAt": "2026-08-13T17:35:34Z", "timestamp": "2026-08-13T17:35:34Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "speculative-decoding" ], "author": "Suppressor72", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51484", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MM][CG][WIP] Encoder CUDA graph support for HunyuanVL", "text": "## Summary Working on encoder CUDA graph support for HunYuanVL (`HunYuanVLForConditionalGeneration`), tracked in #38175. Claiming this model in the tracker. This PR starts with a prerequisite refactor: `HunYuanVisionTransformer` currently processes images by splitting the batch into a Python list and running each image through every layer separately (`O(num_layers * num_images)` split/cat calls). This commit packs all images into a single sequence and uses `cu_seqlens`-scoped attention instead, matching the pattern used by Qwen2-VL/Qwen3-VL. This is a standalone perf win and a prerequisite for graph capture (a per-image Python loop across layers isn't graph-friendly). **Not yet included:** the actual `SupportsEncoderCudaGraph` wiring. HunyuanVL's patch merger reshapes tokens into an explicit `(h, w)` spatial grid per image before a Conv2d-based merge step (with per-row newline tokens and begin/end/sep tokens). Unlike Qwen2-VL's merger, which works on a flat sequence and only cares about total token count, HunyuanVL's merger needs the real aspect ratio, not just the token budget — two images with the same token count but different `(h, w)` produce different merge outputs. This means the \"uniform dummy shape per token budget\" trick used for Qwen2-VL/Qwen3-VL doesn't directly apply here. This looks like the same class of problem noted for MiniCPM-V in #38175 (needs a secondary capture axis / per-shape graph pool rather than token-budget-only grouping). Will follow up with the capture/replay/forward methods once that's sorted out — flagging here in case anyone has thoughts on approach before I go further. ## Why this isn't a duplicate Checked #38175's checklist and comments, `gh pr list --search \"hunyuan\"` — no open PR or issue comment claims HunyuanVL for encoder CUDA graph work as of this PR. ## Test plan - [x] `ruff check` / `ruff format --check` pass on the changed file - [x] Standalone (non-vLLM) PyTorch script confirming packed `cu_seqlens`-masked SDPA attention is numerically equivalent to the old per-image split-attention loop (max abs diff ~1.8e-7) - [ ] Unit tests (`tests/models/multimodal/...`) — not yet run, no GPU available in the environment this was written in - [ ] E2E benchmark — not yet run, same reason I don't have GPU access in my current dev environment, so the actual model forward pass (patch embed → packed attention → merger) is unverified end-to-end. I'll need to run this on GPU hardware before it's mergeable — treating this as a draft/WIP starting point, not something ready for review yet. ## AI assistance disclosure Portions of this PR (the refactor and this description) were written with AI assistance (Claude). I've reviewed the diff and will run and verify tests myself before taking this out of draft.", "url": "https://github.com/vllm-project/vllm/pull/51484", "createdAt": "2026-08-08T07:10:47Z", "updatedAt": "2026-08-13T08:54:48Z", "timestamp": "2026-08-13T08:54:48Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "documentation", "multi-modality", "nvidia" ], "author": "gangula-karthik", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51506", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Hoist tensor-descriptor build out of unified-attention TD ti…", "text": "## Summary Part of the TD adoption strategy proposed in #42545. `USE_TD`'s K/V tile loader rebuilt the tensor descriptor every tile iteration, since the descriptor's `base` baked in the per-tile physical block index. That forces a `tensormap_create` per iteration, which the pipeliner can't predicate: - On Hopper (sm_90), this only compiles at `num_stages=1`, costing roughly 50% throughput vs the raw-pointer default. - On sm_100/sm_120, it fails to compile at all: `'ttng.tensormap_create' op pipeliner doesn't know how to predicate this op`. ## Fix Move the loop-invariant `kv_head_idx` into the descriptor base and build one descriptor per K/V cache before the tile loop, addressing the per-tile physical block via a `load()` coordinate instead. This makes the descriptor loop-invariant, restoring `num_stages > 1` compilation on Hopper and fixing the sm_100/sm_120 compile failure outright. The hoisted 3D descriptor also needs a registered scratch allocator on sm_100/sm_120 (`Kernel requires a runtime memory allocation, but no allocator was set` otherwise). Wired up `set_triton_allocator()`, matching the existing pattern in `fused_moe` / GDN / LoRA's TD paths, cached per device to avoid a redundant call on every forward pass. ## Testing ### Correctness ``` pytest tests/kernels/attention/test_triton_unified_attention.py -k use_td ``` `290 passed` on H100/H200. ### Throughput `vllm bench throughput`, Qwen3-4B bf16, TRITON_ATTN, H100/H200: | Workload | in/out | raw ptr | TD hoisted | hoisted/raw | |---|---|---|---|---| | prefill_heavy | 2048/128 | 17.57 req/s | 15.85 req/s | 0.90x | | decode_heavy | 128/1024 | TBD | TBD | TBD | | balanced | 1024/256 | 28.25 req/s | 25.13 req/s | 0.89x | Repro: ``` COMMON=\"--model Qwen/Qwen3-4B --dtype bfloat16 --attention-config '{\\\"backend\\\":\\\"TRITON_ATTN\\\"}' --max-model-len 4096 --gpu-memory-utilization 0.85 --seed 42\" VLLM_TRITON_USE_TD=0 vllm bench throughput $COMMON --dataset-name random --random-input-len 2048 --random-output-len 128 --num-prompts 200 VLLM_TRITON_USE_TD=1 vllm bench throughput $COMMON --dataset-name random --random-input-len 2048 --random-output-len 128 --num-prompts 200 ``` Note: use `--random-input-len` / `--random-output-len`, not `--input-len` / `--output-len`, with `--dataset-name random`, the latter are silently overridden by the random defaults. B200 (sm_100), same model/backend/workloads: | Workload | in/out | raw ptr | TD hoisted | hoisted/raw | |---|---|---|---|---| | prefill_heavy | 2048/128 | 26.01 req/s | 22.05 req/s | 0.85x | | decode_heavy | 128/1024 | 23.25 req/s | 17.98 req/s | 0.77x | | balanced | 1024/256 | 40.22 req/s | 32.34 req/s | 0.80x | Correctness on B200: `290 passed` as well, same suite. Close to oonyshch's own B200 numbers (0.85x/0.83x/0.78x), independently confirmed on our own commit. cc: joint effort with @oonyshch @BabyDrangoner,coauthored the repro and root cause on H200/b200", "url": "https://github.com/vllm-project/vllm/pull/51506", "createdAt": "2026-08-08T16:03:02Z", "updatedAt": "2026-08-13T13:44:40Z", "timestamp": "2026-08-13T13:44:40Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [], "author": "cinnamonica02", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51520", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RL] Add nccl-m2n sharding-aware weight transfer (worker side)", "text": "## Purpose Adds `nccl_m2n`, the inference-side half of a sharding-aware weight transfer backend built on [NCCL M2N](https://github.com/NVIDIA/nccl-extensions), per [RFC #46439](https://github.com/vllm-project/vllm/issues/46439). The broadcast NCCL backend assumes both sides hold the same layout, so a trainer that shards its parameters (FSDP, EP) must all-gather each one to a full tensor before sending. `nccl_m2n` moves each parameter with a single `reshard` that redistributes it between two disjoint meshes inside one communicator — trainer ranks `[0, T)`, inference workers `[T, T + N)`, which is exactly the interval vLLM's existing `rank_offset` convention already produces. The trainer sends its local shards and never materializes a full tensor. This PR is the **worker side only**. Each worker receives the whole tensor and hands it to `load_weights`, which is what the broadcast backend does for every parameter — so the two are directly comparable, and the destination layout is the same for every parameter, derived from the rank split with no negotiation. An external trainer can drive it today through the existing four-phase control plane (HTTP or Ray). Two follow-ups, already written and stacked behind this: 1. the in-tree trainer engine (`M2NWeightSource` / `DTensorModuleSource`, `M2NTrainerWeightTransferEngine`) plus a runnable FSDP → TP example; 2. per-parameter destination layouts, so a worker receives only its own shard. Selected with `--weight-transfer-config '{backend: nccl_m2n}'`. Default behavior is unchanged, and the runtime is imported lazily, so vLLM is unaffected unless the backend is selected. ## Not a duplicate Checked per `AGENTS.md`: ```bash gh pr list --repo vllm-project/vllm --state open --search \"46439 in:body\" # none gh pr list --repo vllm-project/vllm --state open --search \"m2n\" # none relevant gh pr list --repo vllm-project/vllm --state open --search \"weight transfer backend\" ``` No open PR references RFC #46439 or adds an m2n backend. The two adjacent ones solve a different problem: - **#40828 (WPI)** — a Kubernetes-native zero-copy transport for moving weights from external trainers. It is a transport substitution; it does not reshard between differing parallelism layouts, which is the entire point here. - **#43375 (RDT weight sync, draft)** — a WIP prototype for RDT sharded transfer. Overlapping in goal, but a different transport and still a prototype with no test plan. If it lands first, the two backends coexist in the factory registry the same way `nccl` / `ipc` / `sparse_nccl` do. ## Test Plan ```bash python -m pytest tests/distributed/test_weight_transfer_m2n.py -v ``` The suite covers how a transfer is *described*, not the transfer itself: layout encoding (including the size-1-axis spelling of full replication), the dtype and tensor-rank limits, init/update-info validation, and backend registration. Those are the failures worth catching cheaply — a bad layout or a ragged plan puts the two sides into mismatched collectives, which hangs rather than raises, so everything is validated during `init_transfer_engine` before any rendezvous. The transfer path needs the `nccl-extensions` runtime and multiple GPUs, so it is exercised out of tree against a real trainer. ## Test Result - `ruff check` / `ruff format --check` — clean on all touched files. - Layout invariants and the replicated-destination derivation verified against a stubbed `torch`, confirming both sides compute the same destination layout independently. - `pytest tests/distributed/test_weight_transfer_m2n.py` — **14/14 passed** (first run; this suite had never been executed before). Python 3.12.13, pytest 9.1.1, vLLM editable install on branch `nccl-m2n-worker` @ `1e7b332a6`. ## Model evaluation This PR has only the worker side change. We will need the trainer side change as well to make an end-to-end model run. The latter will be submitted in a stack PR. ## Known limitations Inherited from the m2n build and documented in `docs/training/weight_transfer/m2n.md`: - tensor rank 1..3; no fp4; `Partial` placements unsupported - the trainer's device mesh must cover a contiguous rank interval from 0, 1-D or 2-D - m2n caps per-destination-shard fan-in at 16 source shards (`MAX_SOURCES`, a compile-time bound in `reshard_limits.h`; raising it needs a library rebuild). This is not a rank limit in general — with a sharded destination each shard is fed by only a few sources. It binds here because a *replicated* destination is a single shard fed by every source shard, so a trainer that shards its parameters is limited to 16 ranks until the sharded-destination follow-up lands. A replicated trainer is unaffected at any scale. - requires `nccl-extensions` (NCCL ≥ 2.30.5) and `VLLM_NCCL_SO_PATH` pointing at the same `libnccl.so` that `libnccl_m2n.so` was linked against — torch bundles an older NCCL, so this is load-bearing rather than cosmetic. ## AI assistance This PR was written with AI assistance (Claude Opus 5 via Claude Code). Every line has been reviewed by the submitter, who understands and can defend the change end to end.", "url": "https://github.com/vllm-project/vllm/pull/51520", "createdAt": "2026-08-08T21:29:42Z", "updatedAt": "2026-08-13T17:23:55Z", "timestamp": "2026-08-13T17:23:55Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "needs-rebase" ], "author": "kwen2501", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51538", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark", "text": "## Purpose DeepSeek-V4-Flash-0731 could not run reliably through the SM120 sparse MLA backend. This fixes the seven defects that blocked it across all three decode modes -- plain decode, MTP, and DSpark -- verified end-to-end on 8xRTX PRO 6000 Blackwell across in-flight batching and prefill/decode disaggregation. Commits 1-5 unblock DSpark. Commits 6-7 fix a hang that is **not** DSpark-specific: it strands any MTP (`next_n > 1`) server on this backend once the batch drains, and is a pre-existing defect on `main` rather than a regression from this PR. It is filed as #51593, with the full root-cause analysis in [this comment](https://github.com/vllm-project/vllm/issues/51593#issuecomment-5237934676). ## Why this is not a duplicate This consolidates #51042 after coordination with the maintainers (see https://github.com/vllm-project/vllm/pull/51042#issuecomment-5230492529). @ilmarkov is `Co-authored-by` on the SWA-width commit, whose `amd/rocm.py` hunk and `decode_swa_width` field are his work. Two deliberate design differences are described in that comment. Duplicate checks run for `dspark`, `sparse MLA SWA width`, `mxfp4 gemm1_alpha`, and `deepseek v4 expert placement`; no other open PR covers the remaining four commits. For commits 6-7, checks run for `persistent_topk`, `indexer seq_lens spec decode`, and `51593 in:body`. Two open PRs are adjacent but do not overlap: - **#43970** (MLA indexer / MTP) touches the same two files. Its \"drop padded MTP decode slots\" applies to the variable-length flatten branch, which computes `seq_len - decode_len` = `0 - 0` = `0` and is already safe. The uniform and native spec-decode paths that produce the negative length are untouched by it. The two changes are complementary and do not conflict. - **#49139** fixes a different bug in `persistent_topk.cuh` (radix histogram reuse after short rows). Worth flagging the interaction: its precondition is a CTA group processing rows that cross `RADIX_THRESHOLD`, and the out-of-range lengths fixed here cause exactly that spuriously. That PR is still needed for genuinely long rows. ## What is fixed 1. **SWA widths** — non-causal draft batches allocate `decode_swa_indices` wider than `window_size`, but the FlashInfer DSV4 path reshaped with `window_size` and crashed the draft. The dense width is now carried on the metadata. The non-causal width pads to a multiple of 64 (192 for the K=5 shape: 128 sliding-window + 5 draft entries) rather than 128 (256), matching the kernel's 64-entry tile; both dispatch after flashinfer-ai/flashinfer#4380, and 192 measures 13-16% faster at >=8 tokens. 2. **Workspace lanes** — the DSpark target and draft CUDA graphs retain workspace views concurrently, so one buffer per ubatch let a resize for one orphan the other's live tensor. The second lane is allocated only for V2 DSpark. 3. **Graph replay and draft KV** — `sample_idx_mapping` was zero-filled, so capture executed padding rows that scattered into request slot 0; captured backbone outputs could be freed before replay read their storage; draft KV could be written into physical block 0, the null block. Draft sampling also moves to a disjoint Philox counter range, since the rejection sampler keys both its acceptance uniform and its recovery Gumbel noise by token position. 4. **MXFP4 SwiGLU parameters** — `FlashInferExperts` injected the GPT-OSS activation constants (`gemm1_alpha=1.702`, `gemm1_beta=1.0`, `gemm1_clamp_limit=7.0`) for every mxfp4 weight dtype. DeepSeek V4 uses this path under `--moe-backend flashinfer_cutlass`, so its SwiGLU was evaluated with GPT-OSS constants and generation collapsed. GPT-OSS is unaffected: `GptOssMxfp4MoEMethod` supplies the same constants through its quant config, which the added test pins. 5. **SM120 gate** — a FlashInfer build can expose the sparse MLA decode API without carrying the DSV4 specialization a configuration needs. That now fails at model init with the required `(num_q_heads, top_k)` shape instead of an opaque kernel launch failure at the first decode. 6. **Negative indexer context lengths under MTP** — padded decode slots carry `seq_len == 0`, and with `next_n > 1` both spec-decode paths computed a negative per-token context length for the first token of each padded request (`0 - 2 + 0 + 1 = -1`). The sparse-MLA top-k kernels consume `lengths` as `uint32`, so `-1` is read as ~4.29e9. Clamped at 0 in both paths, matching the variable-length path which already yields `0`. With `next_n == 1` the expression collapses to `seq_len`, which is why plain decode never hit this. 7. **Top-k kernels hardened against out-of-range lengths** — `persistent_topk_kernel` cast `lengths` to `uint32` *before* testing `RADIX_THRESHOLD`, so the bogus ~4.29e9 forced the row onto the multi-CTA radix path. Because the `cta_in_group != 0` early exit is decided from the host-side `max_seq_len` while the per-row branch reads device memory, the two disagreed and stranded the group leader on the inter-CTA barrier forever — the kernel never retired, the async output-copy event never fired, and the engine hung waiting for a response that was never sent. The row length is now clamped to `min(stride, max_seq_len)` before any decision, which also removes an out-of-bounds read into the next row. `cooperative_topk` compares signed (so it cannot hang) but then casts to `uint32`, emitting indices `0..TopK-1` instead of `-1` padding; clamped as well. ## Model evaluation DeepSeek-V4-Flash-0731, `--moe-backend flashinfer_cutlass`, `--attention-backend FLASHINFER_MLA_SPARSE_DSV4`, gsm8k `n=1319`: | configuration | strict-match | flexible-extract | acceptance | FlashInfer | | --- | --- | --- | --- | --- | | target-only, TP4 | 0.9454 | 0.9454 | — | main @ 7f614b86 | | DSpark, TP4 | 0.9477 | 0.9477 | 65.5% | main @ 7f614b86 | | DSpark, TP4+EP4 | 0.9462 | 0.9462 | 67.1% | dc963cc0 | | DSpark + P/D (mooncake) | 0.9492 | 0.9500 | — | dc963cc0 | Standard rejection sampling is distribution-preserving, and DSpark lands 0.0023 from target-only against a difference-stderr of ~0.009 (0.26 sigma), on the same checkpoint and the same tree. MTP on the base checkpoint (`num_nextn_predict_layers = 1`), the configuration that #51593 hangs on, gsm8k `n=1319`: | configuration | strict-match | | --- | --- | | MTP, TP4 (commits 6-7 applied) | 0.9477 | That sits inside the 0.9454-0.9492 band spanned by the target-only and DSpark rows above, on the same checkpoint and tree, so the clamp does not move accuracy. No unfixed baseline is quoted for this row: without commits 6-7 the same server wedges under the drain reproducer within 30-180 s (see Test plan), and I did not obtain a clean unfixed eval run to compare against. Isolating fix 4 — identical configuration with only that commit removed: | configuration | strict-match | flexible-extract | | --- | --- | --- | | target-only, TP4 | 0.0000 | 0.0220 | | DSpark, TP4 | 0.0000 | 0.0174 | The TP4 pair was measured on both FlashInfer revisions with no significant change (DSpark 0.9454 -> 0.9477, target-only 0.9462 -> 0.9454), so the two rows measured on the earlier revision are directly comparable. ## Test plan ``` pytest tests/v1/worker/test_workspace.py \\ tests/v1/spec_decode/test_dflash_prepare_inputs.py \\ tests/v1/worker/gpu/spec_decode/test_utils.py \\ tests/kernels/attention/test_flashmla_sparse.py \\ tests/kernels/test_compressor_kv_cache.py \\ tests/v1/attention/test_flashinfer_sparse_mla_sm120_api.py \\ tests/kernels/moe/test_ocp_mx_moe.py ``` Result: **71 passed, 141 skipped**. For commits 6-7, a drain-shaped reproducer (waves of concurrent completions that retire unevenly, wedging the unfixed build in 30-180 s) against `deepseek-v4-flash` with `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`, TP4: | build | runs | result | | --- | --- | --- | | unfixed | — | hangs within 30-180 s; all 4 GPUs pinned at 100% util / 0% memory util | | commit 6 only | 13 | all clean | | commit 7 only (commit 6 reverted, so negative lengths still reach the kernel) | 3 | all clean | The commit-7-only run is the one that shows the kernel guard is load-bearing rather than dead code: with commit 6 in place nothing negative ever reaches the kernel, so it would pass either way. Root cause was confirmed on a live hang with cuda-gdb: exactly one of 16 CTAs resident, spinning in `wait_ge`, with `arrival_counter == 1` against `target_val == 2` — a group leader waiting on a peer that had already taken the `max_seq_len` early exit. ## AI assistance This work was produced with AI assistance. Every changed line has been reviewed by the submitter, and the tests and evaluations above were executed on the hardware described, with results reported as measured.", "url": "https://github.com/vllm-project/vllm/pull/51538", "createdAt": "2026-08-09T08:05:15Z", "updatedAt": "2026-08-13T06:41:54Z", "timestamp": "2026-08-13T06:41:54Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "bug", "speculative-decoding", "ready", "nvidia", "mrv2" ], "author": "lucifer1004", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51540", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Avoid a per-prefill-step GPU->CPU sync in the KDA chunk kernels", "text": "`prepare_chunk_indices` derives per-sequence chunk counts with `.tolist()`, so passing it a device `cu_seqlens` blocks the caller. The count also fixes the Triton grid, so it has to reach the host either way; compute it from the host-side cu_seqlens we already have and copy the result across, as `gdn_attn` already does for its own prefill slice. Models from `vllm/models/kimi_k3` using FLA hit this on prefill batches, on NVIDIA and ROCm. The existing `chunk_indices` describes the decode-peeled `prefill_query_start_loc`, so add `non_spec_chunk_indices` alongside it and thread an optional `chunk_indices` through `chunk_kda_with_fused_gate`. It defaults to None and falls back to deriving on device, so other callers are unaffected. Verified host- and device-derived indices match in value, dtype and length. Not run on Kimi hardware. Note this applies to Kimi Linear but **not** Kimi K3 with cuda since the latter uses FlashKDA rather than FLA. Claude was used for this.", "url": "https://github.com/vllm-project/vllm/pull/51540", "createdAt": "2026-08-09T08:23:52Z", "updatedAt": "2026-08-13T10:15:44Z", "timestamp": "2026-08-13T10:15:44Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "ready", "needs-rebase", "kimi", "k3" ], "author": "njhill", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51545", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][NVFP4 MoE] Support CuTeDSL for ModelOpt W4A16 and pad non-gated experts", "text": "**Closing since this is not a clean W4A16 as it will still quantize activations, so the comparison between the 2 backends is not apples to apples, and the correctness results below show degradation. I might open a separate PR for the padding fix** ## Purpose Enable the FlashInfer CuTeDSL NVFP4 MoE backend for ModelOpt W4A16 NVFP4 checkpoints on Blackwell, and fix the non-gated CuTeDSL shape handling needed by Nemotron-style MoE layers. This patch enables the existing CuTeDSL API for ModelOpt W4A16 by treating missing activation scales as identity scales at the wrapper boundary. Longer term, CuTeDSL should expose W4A16/no-activation-scale support as an explicit backend mode instead of requiring identity activation-scale tensors. This avoids the Marlin fallback for supported Blackwell W4A16 case; on GB200 with `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4`, auto-selected CuTeDSL improves output throughput by **38.9%** over forced Marlin and reduces mean TPOT by **30.3%**. This PR does three things: 1. Allows the NVFP4 MoE oracle to consider CuTeDSL for ModelOpt W4A16 checkpoints under `--moe-backend auto`. - ModelOpt W4A16 still enters backend selection as static NVFP4 weights with missing activation scales: `weight_key=kNvfp4Static`, `activation_key=None`. - CuTeDSL can run this case through its runtime activation-scale path, but because ModelOpt W4A16 checkpoints do not provide activation scales, the CuTeDSL wrapper materializes identity activation scales. - The oracle therefore uses `kNvfp4Dynamic` only for CuTeDSL's activation support predicate; other backends keep the original `activation_key=None` semantics. 2. Makes CuTeDSL tolerate missing W4A16 activation scales. - W4A16 checkpoints do not carry calibrated activation scales. - CuTeDSL materializes identity activation global scales for this case, matching the dtype/device of the global scale tensors they fold into. 3. Pads non-gated NVFP4 MoE intermediate rows to CuTeDSL's 128-row tile requirement. - The original unpadded intermediate size is preserved in the MoE config. - Padded regions are zero-filled so model outputs are unchanged. With this change, users can run ModelOpt W4A16 NVFP4 MoE with the normal backend selection path: ```bash vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\ --moe-backend auto ``` On supported Blackwell systems, auto selects: ``` Using 'FLASHINFER_CUTEDSL' NvFp4 MoE backend ``` ## Test Plan ### Unit tests Add and run focused cases in: 1. tests/quantization/test_modelopt.py 2. tests/quantization/test_trtllm_nvfp4_hidden_dim_padding.py ### Correctness Full GSM8K using lm-eval local chat completions: ``` python3 -m lm_eval \\ --model local-chat-completions \\ --apply_chat_template \\ --tasks gsm8k.yaml \\ --log_samples \\ --model_args \"model=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4,base_url=http://.../v1/chat/completions,max_length=32768\" \\ --gen_kwargs \"max_tokens=16384,temperature=0,top_p=1\" ``` ### Performance I compared `--moe-backend auto` after this PR, which selects CuTeDSL, against `--moe-backend auto` before this PR, which selects Marlin. Both tests ran with `vllm/vllm-openai:nightly-65b7662d3fcb773afaf751ab29ac6960a0cf011d`, with and without this PR applied. I used nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (W4A16) with GB200 (SM100) for the benchmark. Serve command: ```bash vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\ --moe-backend auto \\ --reasoning-parser nemotron_v3 ``` Bench command: ```bash vllm bench serve \\ --endpoint /v1/completions \\ --dataset-name random \\ --random-input-len 64 \\ --random-output-len 512 \\ --ignore-eos \\ --request-rate inf \\ --max-concurrency 128 \\ --num-prompts 512 ``` ## Test Result ### Unit tests Passed. ### Correctness ``` gsm8k strict-match exact_match: 0.9424 ± 0.0064 gsm8k flexible-extract exact_match: 0.9447 ± 0.0063 ``` ### Performance Backend config | Selected MoE backend | Successful requests | Output tok/s | Mean TPOT -- | -- | -- | -- | -- --moe-backend auto | FLASHINFER_CUTEDSL | 1536 / 1536 | 15235.59 ± 94.15 | 7.45 ± 0.07 ms --moe-backend marlin | MARLIN | 1536 / 1536 | 10972.30 ± 106.03 | 10.69 ± 0.04 ms ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51545", "createdAt": "2026-08-09T08:48:12Z", "updatedAt": "2026-08-12T18:01:08Z", "timestamp": "2026-08-12T18:01:08Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia", "quantization" ], "author": "fsaady", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51551", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] Add guarded gfx942 FP8 context prefill", "text": "## Summary Adds an opt-in gfx942 FP8 context-prefill path for Kimi-K3 MLA using the D192/V128 kernel from ROCm/aiter#4645. It also supports checkpoint-specific static per-head scales through a versioned external artifact, removing dynamic amax reductions while preserving the existing attention boundary. ## Background The existing AITER FP8 ASM kernels do not cover Kimi-K3's asymmetric D192 Q/K and D128 V shape on gfx942. ROCm/aiter#4645 adds a packed variable-length Gluon implementation and static per-head quantization for this shape. ## What this PR does - Enables the AITER FP8 path only with `VLLM_ROCM_KIMI_K3_FP8_PREFILL=1`. - Restricts dispatch to BF16 Kimi-K3 MLA on gfx942 with 12 heads, D192 Q/K, and D128 V. - Quantizes Q once per context pass and reuses its per-head scale across context segments. - Collects layer-owned Q/K/V maxima after warmup in eager calibration mode. - Writes atomic TP-rank shards and merges them into a versioned `safetensors` artifact. - Validates model revision, TP/PP topology, cache mode, dimensions, dtype, layer coverage, and finite positive descales before compilation. - Uses a one-kernel static quantizer when an artifact is configured. Missing or incompatible artifacts fail instead of silently falling back. - Keeps the BF16 path for unsupported shapes and short contexts. ## Validation Model: `moonshotai/Kimi-K3`, TP8, MI325X. - Full five-shot gsm8k: 0.9606 exact match with 0.0054 stderr. - Needle retrieval: 9 of 9 passed at 128K, 512K, and 1,047,552 tokens with needle depths 10%, 50%, and 90%. - Deterministic completion sanity checks: 3 of 3 passed. - Matched 1,047,552-token dynamic TTFT: 434.65 s, 404.80 s, and 404.83 s. - Matched static TTFT: 431.38 s, 402.44 s, and 402.44 s. - Mean static TTFT improved by 0.64%. The two warm repeats improved by 0.59%. - AITER synthetic gfx942 tests: 9 passed. - vLLM artifact and configuration tests: 8 passed. ## Notes / scope This PR depends on ROCm/aiter#4645. The feature remains disabled by default. Calibration and validation prompt sets are separate, and calibration is eager-only. ## Test plan - [x] Full gsm8k, five-shot, 1319 samples - [x] 128K, 512K, and one-million-token needle retrieval - [x] Three matched one-million-token dynamic and static repeats - [x] API sanity checks - [x] Artifact compatibility and merge tests", "url": "https://github.com/vllm-project/vllm/pull/51551", "createdAt": "2026-08-09T10:16:12Z", "updatedAt": "2026-08-13T17:10:51Z", "timestamp": "2026-08-13T17:10:51Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "needs-rebase", "nvidia", "kimi", "k3" ], "author": "maeehart", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51560", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fail fast on Inkling's unsupported GPU architectures", "text": " ## Purpose Fixes the startup crash reported in #51405. Serving Inkling on SM120 (RTX PRO 6000 Blackwell, GeForce Blackwell, DGX Spark) died during startup profiling with ``` AssertionError: Paged KV not supported on SM 12.0 in this PR RuntimeError: Worker failed with error 'Paged KV not supported on SM 12.0' ``` raised from the vendored FA4 relative-attention forward, after weights had already been loaded and the KV cache sized. The FA4 kernel has no paged-KV forward on SM12x, and vLLM always attends over the paged KV cache, so there is no configuration that makes the model run there. **Scope note:** an earlier revision also rejected SM8x, since FA4 asserts `page_table is None` there too. @TheVerdman demonstrated Inkling serving end-to-end on 4×A100 through a FlexAttention fallback carrying the relative bias via `score_mod`, so SM8x is deliberately left open for that work and this PR rejects SM12x only. The capability tables are a deny-list of broken architectures rather than an allow-list of working ones, so adding a fallback means adding a path, not deleting a guard. This does not add SM120 support. The kernel is fetched at build time from `vllm-project/tml-fa4` (pinned in `cmake/external_projects/tml_fa4.cmake`) and `vllm-project/flash-attention`, so a paged-KV SM120 forward cannot be implemented from this repo. What this PR does is take the second option #51405 asks for — *\"a documented supported-hardware note ... would save others the investigation\"* — and reject those architectures up front instead of asserting inside a worker after a multi-GiB weight load. **Changes** - `vllm/platforms/cuda.py` — `CudaPlatformBase.verify_model_arch` rejects the three Inkling architectures, driven by a new `_CAPABILITY_RESTRICTED_MODELS` table. Mirrors the existing `RocmPlatform.verify_model_arch` / `_ROCM_UNSUPPORTED_MODELS` pattern. The capability query is wrapped in `try/except RuntimeError` because on non-NVML installs it goes through torch and can initialize CUDA in the resolving process. - `vllm/config/model.py` — calls `verify_model_arch` from `ModelConfig` once the architecture is resolved. Without this the hook never runs in the front end: `_try_load_model_cls` is its only caller, and front-end resolution goes through `inspect_model_cls` instead, so on SM120 the guard fired inside the worker off the usage-stats path in `init_device()` (see @Rawsejet's [test report](https://github.com/vllm-project/vllm/pull/51560#issuecomment-5257205501)). Both TP workers spawned and dumped a traceback before `EngineCore` shut them down — still before `load_model()`, but noisier and later than it needs to be. The `_try_load_model_cls` call stays as a backstop for paths that build a model without a `ModelConfig`. On ROCm this is a no-op today: `_ROCM_UNSUPPORTED_MODELS` and `_ROCM_PARTIALLY_SUPPORTED_MODELS` are both empty. - `vllm/models/inkling/nvidia/attention.py` — `InklingAttention.__init__` keeps a check of its own for direct construction, naming the device and the missing kernel. This constructor is the single chokepoint for both backbone (`nvidia/model.py`) and MTP (`nvidia/mtp.py`) layers; the vision/audio towers are Triton-only and never reach FA4. - `vllm/models/inkling/nvidia/ops/fa4_rel_attention.py` — adds `check_inkling_fa4_support()`, and folds the three arch predicates scattered through the file into named constants (`_PAGED_KV_MAJORS`, `_SHEARED_BIAS_MAJORS`, `_SPLIT_KV_MAJORS`) with one comment pointing at the tml-fa4 pin they all track. This also fixes `inkling_fa4_num_splits`, which keyed off `major == 9` and so returned a split count of up to 128 on architectures whose kernel has no split-KV support at all - `docs/models/supported_models.md` — Inkling had no rows in either table. Adds `InklingForCausalLM` and `InklingForConditionalGeneration`, plus a note recording that SM120/SM121 are unsupported, why, and that ROCm uses a separate Triton implementation and is unaffected. - Tests — coverage for the guard, for the platform hook, a test that `InklingAttention.__init__` actually invokes the guard before touching anything else, and `tests/test_config.py::test_platform_rejects_unsupported_arch_during_config`, which pins the front-end call so the hook cannot silently regress to worker-only again. Adds `skip_if_capability_restricted` to `tests/models/utils.py`, used by `test_initialization.py` and `test_registry.py`: both resolve every registered architecture, and `_try_load_model_cls` calls `verify_model_arch` outside its `try/except`, so without the skip they would error rather than skip on a GPU that cannot run Inkling. **No behavior change on any architecture that could already run Inkling.** On SM8x/SM90/SM100/SM110 the guard is a no-op and `inkling_fa4_num_splits` returns exactly what it returned before (`major == 9` → 1, unchanged; 10/11 → the same computed value). No kernel is added and no numerics are altered on any architecture that could run Inkling previously, so there is no accuracy or serving delta to evaluate. The only behavioral change is on architectures where the model could not run at all **Not duplicating existing work.** `gh pr list --repo vllm-project/vllm --state open --search \"51405 in:body\"` returns nothing, and #51405 is open with no comments. The open Inkling PRs are unrelated: #51517 and #50528 (parser), #49120 (fp32 head dtype), #49315 (warmup infrastructure), #48768 (multi-depth MTP ) ## Test Plan Lint and static checks over every changed file: ```bash pre-commit run --files docs/models/supported_models.md \\ tests/models/inkling/test_fa4_rel_attention.py \\ tests/models/test_initialization.py tests/models/test_registry.py \\ tests/models/utils.py tests/test_config.py vllm/config/model.py \\ vllm/models/inkling/nvidia/attention.py \\ vllm/models/inkling/nvidia/ops/fa4_rel_attention.py vllm/platforms/cuda.py ``` Unit tests covering the new guard, the platform hook, and the arch constants: ```bash .venv/bin/python -m pytest tests/models/inkling/test_fa4_rel_attention.py -v .venv/bin/python -m pytest tests/test_config.py -k platform_rejects_unsupported_arch -v ``` Regression check on the two suites that resolve every registered architecture, since `verify_model_arch` now runs in that path: ```bash .venv/bin/python -m pytest tests/models/test_registry.py -v .venv/bin/python -m pytest \"tests/models/test_initialization.py::test_can_initialize_large_subset\" -v ``` End-to-end on an unsupported device (SM120) — the failure should now appear during startup, naming the device, rather than as a worker assertion after the weight load: ```bash vllm serve thinkingmachines/Inkling-Small-NVFP4 \\ --tensor-parallel-size 2 \\ --tokenizer-mode inkling \\ --trust-remote-code ``` And on a supported device (SM90/SM100), to confirm serving is unaffected: ```bash vllm serve thinkingmachines/Inkling-Small-NVFP4 --trust-remote-code ``` ## Test Result **Lint — all hooks pass, and the run is idempotent (a second pass modifies nothing):** ``` ruff check.......................................................Passed ruff format......................................................Passed typos............................................................Passed markdownlint-cli2................................................Passed Run mypy for Python 3.10.........................................Passed Check SPDX headers...............................................Passed Check root lazy imports..........................................Passed Check for spaces in all filenames................................Passed Update Dockerfile dependency graph...............................Passed Check for forbidden imports......................................Passed Prevent new 'torch.cuda' APIs call...............................Passed Validate configuration has default values and that each field has a docstring................................................Passed Check for boolean ops in with-statements.........................Passed Suggestion.......................................................Passed ``` **Before, on SM120** (as reported in #51405):weights load, the FP4 MoE backend selects, the KV cache is sized, and the worker then dies in the forward pass with `AssertionError: Paged KV not supported on SM 12.0 in this PR`. **After, on SM120** — tested by @Rawsejet on the #51405 reporter's hardware ([full report](https://github.com/vllm-project/vllm/pull/51560#issuecomment-5257205501)): 2× RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, driver 595.58.03, CUDA 13.2, vLLM `0.26.1rc1.dev376+g613411a90`, torch `2.13.0+cu130`, TP=2, runtime hunks applied onto the installed nightly. Both guards behave as intended on the live device: ``` check_inkling_fa4_support() -> ValueError (names the device, 12.0) verify_model_arch(InklingForConditionalGeneration) -> ValueError (12.x) verify_model_arch(InklingForCausalLM) -> ValueError (12.x) verify_model_arch(InklingMTPModel) -> ValueError (12.x) verify_model_arch(LlamaForCausalLM) -> OK (no raise) ``` `vllm serve ... --tensor-parallel-size 2 --tokenizer-mode inkling --trust-remote-code` exits in ~25 s with the clean `ValueError` and **no weights loaded**, against a full ~152 GiB load before the assertion previously. That run also showed the guard firing **in the worker**, not the front end, which the description previously claimed — `_try_load_model_cls` was the only caller of `verify_model_arch`, and front-end resolution does not reach it. The `vllm/config/model.py` change above fixes that; the worker-side check remains as a backstop. Re-test on SM120 welcome. **Everything else — not yet run.** I don't have access to SM90/SM100 hardware or a GPU build, so I have not executed the pytest suites or the supported-device `vllm serve` command above, and I'd rather say so than paste results I can't stand behind. Requesting CI to run the unit and registry suites. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51560", "createdAt": "2026-08-09T13:32:22Z", "updatedAt": "2026-08-13T07:01:19Z", "timestamp": "2026-08-13T07:01:19Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "bug", "documentation", "nvidia" ], "author": "m4xkushnir", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51575", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MRV2] Respect dynamic K=0 from scheduler in AutoRegressiveSpeculator", "text": "## Problem When Dynamic Speculative Decoding (DSD) selects K=0 at higher batch sizes, `AutoRegressiveSpeculator` ignores the scheduler's runtime K and executes the configured maximum number of draft steps. Those drafts are not used, so this wastes GPU work. The scheduler already publishes the selected K in `SchedulerOutput.num_spec_tokens_to_schedule`; MRV2 did not propagate that value to proposal execution. Fixes #51510. ## Solution - Propagate `num_spec_tokens_to_schedule` through `ExecuteModelState` into `speculator.propose()`. - For K=0, run draft prefill to keep the draft KV cache synchronized, then return an empty-width draft tensor without running draft decode. - Pass only the returned draft width to `DraftTokensHandler`, preventing stale columns in the reusable draft buffer from becoming phantom draft slots. - Preserve prior behavior when the runtime K is `None` or positive. Other speculator implementations accept the optional argument but do not act on it. ## Related work / duplicate check Searches for #51510 and MRV2 dynamic-K fixes found no other PR addressing this bug. #49652 is complementary, not duplicate. It fixes CUDA graph capture-shape derivation for autoregressive draft decode under dynamic SD. This PR carries the scheduler's runtime K into proposal execution and skips draft decode when that K is zero. A deployment using FULL CUDA graphs needs #49652's capture fix as well as this runtime behavior fix. ## Tests ```bash .venv/bin/python -m pytest \\ tests/v1/worker/test_gpu_autoregressive_speculator.py \\ tests/v1/worker/test_gpu_model_runner_v2_eplb.py -q ``` Result: `23 passed`. The new regression tests verify that: - K=0 still runs draft prefill but does not prepare or execute draft decode; - the K=0 proposal has shape `(num_requests, 0)`; - the scheduler-selected K reaches `speculator.propose()`; and - stale columns in the persistent draft buffer do not reach `DraftTokensHandler`. As a red/green check, removing the K=0 early return and active-width slice made the two new tests fail with width 2 instead of 0. Restoring the fix made them pass. ```bash uvx pre-commit run --files \\ tests/v1/worker/test_gpu_autoregressive_speculator.py \\ tests/v1/worker/test_gpu_model_runner_v2_eplb.py \\ vllm/v1/worker/gpu/model_runner.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ vllm/v1/worker/gpu/spec_decode/dflash/speculator.py \\ vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py \\ vllm/v1/worker/gpu/spec_decode/speculator.py ``` Result: all applicable hooks passed. `git diff --check` passed. ## Benchmark 2× RTX 5090, ThinkingCap-Qwen3.6-27B-FP8, MRV2, TP=2, `FULL_AND_PIECEWISE` CUDA graphs, and DSD `[[1,2,2],[3,16,0]]`: | Sessions | Before | After | Change | No-spec reference | |---:|---:|---:|---:|---:| | 1 | 107 t/s | 108 t/s | +1% | 107 t/s | | 4 | 193 t/s | 211 t/s | +9% | 209 t/s | | 8 | 364 t/s | 399 t/s | +10% | 393 t/s | At one session the schedule selects K=2, so no K=0 improvement is expected. At four and eight sessions, K=0 is active. Instrumentation showed that the unpatched path spent approximately 7 ms on two unused draft-decode passes per K=0 step. ## AI assistance OpenAI Codex assisted with analysis, code drafting, and test drafting. I reviewed the changes and validation results and can explain and maintain the patch.", "url": "https://github.com/vllm-project/vllm/pull/51575", "createdAt": "2026-08-09T18:49:09Z", "updatedAt": "2026-08-13T15:43:07Z", "timestamp": "2026-08-13T15:43:07Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "mrv2" ], "author": "Suppressor72", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51590", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Memory] Measure complete CUDA graph capture footprint for KV budgeting", "text": "## Purpose Fix CUDA graph memory undercounting in `profile_cudagraph_memory()`. The old path profiled only two descriptors per graph mode and extrapolated the rest. It also summed per-mode deltas even though FULL and PIECEWISE share one runtime pool, and it did not budget memory allocated during profiling setup or retained after profiling cleanup. The shortfall was assigned to the KV cache, so `--gpu-memory-utilization` could be exceeded during capture or serving while the KV pool was still mostly empty. This PR: - Profiles **every** capture descriptor in **one end-to-end span** across all decoder modes instead of extrapolating from two samples per mode. - Includes profiling KV-cache / attention-metadata setup and **`cudagraph_profiling_retained_memory`** in the estimate (scratch the real capture reuses). - Skips the profiling capture entirely when `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` (previously ran and discarded the result). - Guards the unaccounted-graph-memory warning so it only fires when graphs will actually be captured (`--enforce-eager` / `cudagraph_mode=NONE` no longer trigger it). - Preserves cleanup and graph-pool restoration when profiling raises. On Kimi-K3 (8× MI355X, TP8), the estimate rose from ~**1.3 GiB/GPU** to **~4.7–16.9 GiB/GPU** depending on config — matching what capture actually needs. This is accounting only; no kernel or graph-capture logic changes. Fixes agentic OOMs where `HSA_STATUS_ERROR_OUT_OF_RESOURCES` occurred at **3–16%** KV usage because graph/setup memory was budgeted to the KV cache. ## Test Plan **Unit tests** (CPU-only; fake device allocator, no GPU required): ```bash pytest tests/v1/cudagraph/test_cudagraph_memory_profiling.py \\ tests/v1/cudagraph/test_cudagraph_memory_worker.py -q ``` Coverage: - `test_cudagraph_memory_profiling.py` — all descriptors and modes; shared-pool span accounting; negative deltas and zero clamping; encoder graphs; profiling-KV setup; retained memory; cleanup on capture failure. - `test_cudagraph_memory_worker.py` — estimate profiled and subtracted from KV budget; `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` skips profiling; no warning when `cudagraph_mode=NONE`. **Serving smoke** (ROCm, optional for reviewers): - Kimi-K3, TP8, FP8 KV, `FULL_AND_PIECEWISE`, `gpu-memory-utilization=0.95` - Confirm log line `Estimated CUDA graph memory: … GiB total` reflects full capture footprint and serve completes without HSA OOM at previously failing batch/seq limits. ## Test Result **Unit tests (this branch):** ``` 13 passed in 5.89s ``` Negative control: 8 behavior-changing tests fail on pristine `upstream/main`. **Serving (Kimi-K3, MI355X TP8):** | config | metric | before (old sampler) | after (this PR) | |---|---|---:|---:| | cm16k s128, c16, util 0.95 | graph memory estimate | ~1.3 GiB/GPU | 4.7–7.1 GiB/GPU | | cm16k s128, c16, util 0.95 | agentic output tok/s | 61 | **227 (+272%)** | | cm16k s128, c16, util 0.95 | TTFT / ITL avg | 56s / 187ms | **2.9s / 50ms** | | cm8k s128, c24, util 0.95 | graph memory estimate | ~1.3 GiB/GPU | **16.90 GiB/GPU** | | cm8k s128, c24, util 0.95 | run health | OOM (transient workspace) | **440 reqs, 0.23% err** | | cm8k s128, c24, util 0.93 | GPU KV cache size | — | **3,660,123 tokens**, clean run | Startup: profiling every descriptor adds ~8s on Kimi-K3 TP8 (21s → 29s for profiling + capture). Warmup reuse across profiling and real capture is intentionally deferred to a follow-up PR.", "url": "https://github.com/vllm-project/vllm/pull/51590", "createdAt": "2026-08-09T23:55:48Z", "updatedAt": "2026-08-12T23:12:21Z", "timestamp": "2026-08-12T23:12:21Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm", "intel-gpu", "nvidia", "verified", "k3" ], "author": "xiaohuguo2023", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51598", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][RFC] Enable scoped AITER W8A8 support on gfx1100", "text": "## Summary Enable scoped gfx1100-compatible AITER W8A8, GDN decode/causal-conv1d single-token, and sampler paths without widening the CDNA/RDNA4 master gates to unsupported CK/ASM operations. Addresses vllm-project/vllm#51136. ## Changes - Add independent fail-closed gates for gfx1100 W8A8, GDN decode, and lazy sampling. - Route W8A8 through public `aiter.gemm_a8w8`, accepting only dynamic per-token activation and channelwise weight scales. - Probe only the two AITER Triton symbols required for GDN decode and keep per-request generators on the native sampler fallback. - Register only the required W8A8 custom op; preserve CDNA/RDNA4 behavior and keep unrelated AITER features disabled. - Require `is_fused_moe_enabled()` before an explicit AITER MoE backend can be selected. This is a fallback fix, not gfx1100 MoE support. ## Validation The tests below were run locally on a W7900; test-file changes are intentionally excluded from this production-only PR. - gfx1100 gate, registration, layout, and MoE fallback tests: 21 passed. - Existing unquantized MoE backend-selection tests: 7 passed. - Official lazy sampler routing test: 2 passed. - AITER GDN recurrent tests: 12 passed; causal-conv1d single-token/fused-reshape tests: 22 passed. - Real top-k, top-p, and joint sampler integration: passed. - Ruff, format, and `git diff --check`: passed. ### W8A8 versus baseline - Baseline: existing vLLM Triton W8A8 kernel. - Public AITER matched the baseline numerically on all 18/18 model-observed shapes and measured `1.119x–3.786x`. - The vLLM custom op to public `aiter.gemm_a8w8` passed on a real gfx1100 GPU. ## Final-source status - Qwen3.6 W8A8 startup selected `AiterInt8ScaledMMLinearKernel`, lazy AITER sampling, and Triton/FLA GDN prefill. - An unrelated base-image vLLM ROCm skinny-GEMM assertion blocked request completion before all decode/sampling paths ran. - Final-source E2E correctness and TPS are not claimed. ## Dependencies and limitations - Depends on ROCm/aiter#4512 and the gfx1100 A8W8 config patch. - Requires an AITER-compatible Triton release and a matching gfx1100-capable vLLM ROCm extension build. - Released-package validation, final model serving, and upstream CI remain pending. ## Publication Draft PR: [vllm-project/vllm#51598](https://github.com/vllm-project/vllm/pull/51598).", "url": "https://github.com/vllm-project/vllm/pull/51598", "createdAt": "2026-08-10T02:24:11Z", "updatedAt": "2026-08-13T02:26:52Z", "timestamp": "2026-08-13T02:26:52Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm" ], "author": "01xjw", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51599", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(v1): decouple async Mamba align D2H counts from InputBatch row shifts (#51571)", "text": "### Target Issue Closes #51571 ### Description When running speculative decoding / MTP in `align` mode with `use_async_scheduling=True`, `GPUModelRunner._update_states_after_model_execute()` passes `input_batch.num_accepted_tokens_cpu_tensor` as the D2H target for `postprocess_mamba_align_gpu()`. However, while the GPU D2H copy is in flight on the CUDA stream, the CPU prepares the next step and calls `InputBatch.condense()`. This compacts finished requests and shifts row indices in `input_batch` *before* `num_accepted_tokens_event` is synchronized in `_prepare_inputs()`. When `_prepare_inputs()` later gathers from `input_batch.num_accepted_tokens_cpu`, it reads corrupted/shifted counts, causing Mamba hidden state copy offsets to misalign for shifted requests. This PR fixes the race condition by: 1. Directing D2H writes to `self.num_accepted_tokens.cpu` (the runner's unmutated CPU buffer snapshot) when `use_async_scheduling=True`. 2. Gathering historical accepted counts from `self.num_accepted_tokens.np` in `_prepare_inputs()`. 3. Adding a unit test (`test_async_mamba_align_accepted_counts_race`) in `tests/v1/worker/test_gpu_model_runner.py` verifying that shifted row positions preserve true historical counts. ### Test Plan - Ran unit test `pytest tests/v1/worker/test_gpu_model_runner.py -k test_async_mamba_align_accepted_counts_race` (PASSED). - Verified `ruff check` passes cleanly.", "url": "https://github.com/vllm-project/vllm/pull/51599", "createdAt": "2026-08-10T02:31:18Z", "updatedAt": "2026-08-13T02:19:32Z", "timestamp": "2026-08-13T02:19:32Z", "metrics": { "reactions": 2, "comments": 19 }, "labels": [ "ready", "mrv1-only" ], "author": "bandham-manikanta", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51605", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Size batched Triton MoE tiles from per-expert rows", "text": "## Purpose `get_default_config` divides `M` by `E` to estimate rows per expert, but the expert-batched layout passes `max_num_tokens`, which is already per-expert — so the fallback over-sizes tiles for the E small GEMMs `batched_triton_kernel` launches. Platform-agnostic as it is a layout mismatch. Only the heuristic fallback changes; override and tuned-file paths are untouched. Also makes the tile index the fastest-varying grid axis, matching `fused_moe`. ## Test Plan `pytest tests/kernels/moe/test_batched_moe.py`, plus kernel A/B and e2e base-vs-patched from paired worktrees at the same commit with identical settings. Measured on XPU only so far; NVIDIA 2x H200 numbers to follow before this leaves draft. ## Test Result *XPU Intel Arc B70, 2x, TP=2* Kernel A/B, 40 cells: geomean 1.075x, min 0.97x, max 1.43x. E2E, 24 cells, all passing. `rows_per_expert = max_num_batched_tokens * topk / E` decides whether the heuristic engages: | regime | geomean | min | max | n | |---|---|---|---|---| | heuristic active | 1.026x | 1.008x | 1.042x | 6 | | defers to shared config | 0.999x | 0.967x | 1.040x | 9 | E2E pending *NVIDIA H200, 2x, TP=2* Pending — same kernel A/B and e2e pair, to be filled in before this leaves draft.
Full numbers and limitations - E2E coverage is bf16 only: WNA16 rejects `moe_backend=batched_triton`, and fp8 only offers BATCHED_TRITON via auto-selection under a batched activation format. Quantized guards are unit-tested. - `benchmarks/kernels/benchmark_moe.py` covers dense shapes only and is CUDA-only (CUDA graphs, Ray `num_gpus`), so the kernel A/B above was run out-of-tree.
The grid reorder comes from [intel-xpu-backend-for-triton#6200](https://github.com/intel/intel-xpu-backend-for-triton/pull/6200) (commit `afa5150`) by @Egor-Krivov. A follow-up will add `@triton.autotune` to this kernel for both CUDA and XPU, which needs the reorder as a prerequisite: once the autotuner picks the block sizes, the host can no longer derive the grid from the config dict. The default config stays the fallback for `VLLM_TRITON_FORCE_FIRST_CONFIG`, `VLLM_BATCH_INVARIANT`, and wna16. Claude was used for the analysis, benchmarking, and edits; all changes were reviewed by me.", "url": "https://github.com/vllm-project/vllm/pull/51605", "createdAt": "2026-08-10T03:16:08Z", "updatedAt": "2026-08-13T12:00:58Z", "timestamp": "2026-08-13T12:00:58Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "oonyshch", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51611", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Fix stale rejection_sample_method and synthetic_acceptance_rate", "text": "## Purpose Fixes #51609 Sync the `--speculative-config` table in `docs/features/speculative_decoding/README.md` with the current code: 1. `rejection_sample_method` (line 87): `strict, probabilistic, synthetic` (default `strict`) -> `standard, synthetic, block` (default `standard`); `probabilistic` now belongs to `draft_sample_method`. See #40651. 2. `synthetic_acceptance_rate` (line 88): split into `synthetic_acceptance_rates` (list) and `synthetic_acceptance_length` (scalar, mutually exclusive). See #40662. ## Why not duplicating an existing PR - #37383: open but stale (needs-rebase since 2026-05, no activity since 2026-03); its new section documents the same outdated values plus a non-existent `speculative_token_tree` key. This PR takes the minimal fix of the existing table. - #51500: typo-only fixes in the same README (touches adjacent lines 89, 200, 220). It does not change the values on lines 87-88. If it merges first, I will rebase manually to resolve any adjacent-line conflicts. ## Test Plan - pre-commit run markdownlint-cli2 --files docs/features/speculative_decoding/README.md ## Test Result Markdown-only change; local environment has no Python, so markdownlint was not run locally — relying on CI check. --- ## AI Assistance This PR was drafted with AI assistance; the human submitter reviewed every changed line. Co-authored-by: deepseek-v4-flash", "url": "https://github.com/vllm-project/vllm/pull/51611", "createdAt": "2026-08-10T04:19:58Z", "updatedAt": "2026-08-13T10:56:28Z", "timestamp": "2026-08-13T10:56:28Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "ready", "build-docs" ], "author": "qwerqwerqwe8688-jpg", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51614", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KV Offload] Emit self-describing CPU events at KV-group block granularity", "text": "## Purpose Fix self-describing CPU KV events for hybrid KV-cache layouts where request hashes are computed more frequently than the full-attention group's block size. For example, DeepSeek V4 may use: - tokens_per_hash = 4, derived from the GCD of its KV-group block sizes - tokens_per_block = 256 for the full MLA group The offloading event tracker currently publishes every raw 4-token hash and sets BlockStored.block_size = 4. For one 256-token CPU chunk, this produces 64 hashes and describes them as independently stored 4-token blocks, even though the full-attention group is stored and indexed at 256-token block granularity. Consumers configured for the 256-token vLLM block size may reject or misinterpret these CPU events. ## Fix Resolve each chunk's raw request hashes from tokens_per_hash granularity to the KV group's tokens_per_block granularity using the same resolve_block_hashes() conversion used by GPU KV events. The event now: - emits the tail hash of each constituent KV-group block - sets block_size to GroupOffloadConfig.tokens_per_block - retains whole-chunk token_ids - preserves multiple constituent block hashes when blocks_per_chunk > 1 Examples with 4-token raw hashes and 256-token KV-group blocks: ``` CPU chunk size Emitted raw-hash indices Event block size ━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ 256 [63] 256 ──────────────── ────────────────────────── ────────────────── 512 [63, 127] 256 ``` This does not change GPU prefix-cache events, Mamba partial-hash hits, CPU offload keys, or CPU chunk lookup granularity. Sliding-window and SSM groups continue using their existing placeholder event path. ## Test coverage Added a parameterized regression test covering: - one 256-token block per CPU chunk - two 256-token blocks in a 512-token CPU chunk - a 4-token request-hash granularity in both cases", "url": "https://github.com/vllm-project/vllm/pull/51614", "createdAt": "2026-08-10T04:34:00Z", "updatedAt": "2026-08-13T03:28:16Z", "timestamp": "2026-08-13T03:28:16Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "bug", "ready", "kv-connector" ], "author": "ziqifan617", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51624", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][Power] Unqualized MoE Backend for Power (VSX)", "text": "This PR adds PowerPC specific unquantized backend support for fused MoE using Power10 VSX MMA instructions. ## Purpose Currently, grouped GEMM is not supported for Power architecture in vLLM. This PR introduces a Power/VSX specific unquantized CPU backend for Fused MoE. Key features include: - Addition of `csrc/cpu/micro_gemm/cpu_micro_gemm_vsx.hpp` to support grouped GEMM on Power architectures. - Use of Power10 MMA instructions for optimized BF16 matrix multiplication. ## Test Plan - Verified mathematical correctness across all unaligned and aligned MoE kernel test cases using: ```bash pytest tests/kernels/moe/test_cpu_fused_moe.py -v ``` - Performance benchmarking using `vllm bench` with `google/gemma-4-26B-A4B-it` to compare against the `main` branch. ## Test Result The unquantized Power10 MMA implementation yields massive speedups in the prefill phase (TTFT), and provides strong throughput gains at higher batch sizes compared to the `main` branch. | Concurrency | TTFT Mean (Prefill) vs Main | Total Tok/s (Throughput) vs Main | TPOT Mean (Decode) vs Main | |-------------|-----------------------------|----------------------------------|----------------------------| | **C=1** | 35.5% faster | 1.1% faster | same as main | | **C=4** | 35.6% faster | same as main | same as main | | **C=8** | 37.0% faster | same as main | same as main | | **C=64** | 35.4% faster | 15.7% faster | 9.9% faster | | **C=128** | 35.1% faster | 13.9% faster | 10.2% faster | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51624", "createdAt": "2026-08-10T06:48:10Z", "updatedAt": "2026-08-13T02:56:10Z", "timestamp": "2026-08-13T02:56:10Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "cpu" ], "author": "Akashcodes732", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51633", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Platform] Add check_runner_kv_caches_multi_layer", "text": "## Purpose Add `check_runner_kv_caches_multi_layer` interface to avoid platform hardcode in bind_kv_cache. So that oot platform can override it to avoid error raising. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51633", "createdAt": "2026-08-10T07:26:32Z", "updatedAt": "2026-08-13T15:36:51Z", "timestamp": "2026-08-13T15:36:51Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "rocm", "intel-gpu", "ready", "cpu", "nvidia" ], "author": "wangxiyuan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51647", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Pad non-aligned AITER MLA heads", "text": "## Summary - pad non-16-aligned ROCm AITER MLA query heads to the next supported multiple of 16 and slice padding from the output - size dense and sparse persistent metadata for the padded launch shape - enable Kimi-K3 TP4's 24 heads/rank to use AITER MLA instead of falling back to Triton MLA ## Performance 8x MI355X, Kimi-K3 TP4/DP2/EP8, 100,000 input tokens (95,911 shared prefix + 4,089 suffix), OSL 1024, one API server, three runs per cohort. | Cohort | Triton MLA TPOT | AITER padded TPOT | TPOT change | Triton output tok/s | AITER output tok/s | Throughput change | |---|---:|---:|---:|---:|---:|---:| | C16 | 62.696 ms | 41.013 ms | -34.6% | 205.40 | 286.48 | +39.5% | | C24 | 76.061 ms | 46.816 ms | -38.5% | 256.32 | 373.99 | +45.9% | Source base: vLLM `61c1dd0966fd8d91954003cf6b7ab746448a46f8`, AITER `ca68b4f3501762c15c550cb920a5516e9710cf89`. ## Test plan - [x] targeted ROCm AITER MLA tests: 32 passed, 2 skipped - [x] Ruff check and format check on all changed files - [x] GSM8K-100: 99/100, 0 malformed outputs, 0 invalid numeric answers - [x] TP4/DP2/EP8 full-graph server startup and 100K C16/C24 decode benchmark Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/51647", "createdAt": "2026-08-10T09:20:08Z", "updatedAt": "2026-08-13T17:02:23Z", "timestamp": "2026-08-13T17:02:23Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "rocm", "verified" ], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51649", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix extreme case in pythonic parser argument json safety", "text": "# [Bugfix] Pythonic tool parsers: argument JSON safety and per-call salvage ## Purpose Follow-up to #48171. Nine defects in the pythonic tool-call family, all found by running real LFM2 agent traces through the parsers. | Model output | Before | After | |---|---|---| | `calc(x=1e999)` | `{\"x\": Infinity}` — invalid JSON | call rejected | | `search(**{'query': 'cats'})` | `{\"null\": {\"query\": \"cats\"}}` | `{\"query\": \"cats\"}` | | `get_weather('Paris', unit='celsius')` | `{\"unit\": \"celsius\"}` — `'Paris'` gone | call rejected | | bad call next to a good one | good call dropped too | only the bad one skipped | | broken quote next to a good call | whole block dropped | good call survives | | two calls, each with a nested quote | one call, argument = the other call's text | both calls, intended values | | a call still arriving mid-stream | an exception with a traceback logged on **every chunk** | quiet wait for more text | | `[foo(x=1])]` — brackets the model got wrong | same, and no log names the offending text | one warning carrying the block | | olmo3 `get_weather(...)` — a **correct** call | one traceback per token of the name | nothing logged | **Provenance**, since the nine are not equally my own: - Rows 1-3 predate #48171 in the shared `handle_single_tool` — they behave identically on the commit before that PR merged. They were deferred from it because fixing them changes all four of `pythonic`, `llama4_pythonic`, `olmo3` and `lfm2`. - Rows 4-5 are long-standing conservative behavior: the block yields no calls at all. Nothing wrong is emitted, so these are robustness, not correctness. - **Row 6 is a further fix for the problems in #48171.** Before that PR the ambiguous-quote block yielded no calls; the nested-quote recovery turned that into a call with corrupted arguments. - Row 7 is log hygiene, and also my own doing: the per-call salvage in row 4 treated \"nothing convertible yet\" as a failure. Nothing was emitted wrongly and no call was lost — it is purely a server-side noise fix. - Rows 8-9 are log hygiene too, and both predate #48171. Row 8 is the same block re-parsed on every chunk, raising at the same bracket every time; skipping it is correct and unchanged, only the reporting is. Row 9 needs no malformed input at all — it fires on every well-formed olmo3 tool call. The five groups are separable — say the word and I will split the PR. ### Rows 5-9 in detail Rows 5 and 6 are block-level, and LFM2 emits several calls in one block often enough for that to matter: in a pinchbench run with LFM2.5-2.6B, 11% of tool-emitting messages carry more than one call (72 of 639). **Row 5** — one broken call takes the block down with it: ``` [read(path='/etc/hosts'), bash(command='grep 'e' a.log', timeout='30s')] ^^^^^^^^^^^^^^^^^^^^^^^ perfectly valid ^^^^^^^ unescaped quotes ``` `ast.parse` sees one syntactic unit, so the whole block fails and `read` is dropped with it — the agent loop gets no tool result at all. The per-call salvage cannot help: it needs a parsed call list, which does not exist. Now the block is split at top-level commas and each segment is parsed on its own, so `read` survives and only the broken call is lost. Splitting runs both a string-aware and a bracket-only scan. A broken quote desynchronizes the former and hides every later separator; string arguments always sit at bracket depth >= 1, so the latter never splits inside them (`exec(command='echo a, b')` stays one segment). A wrongly split segment simply fails to parse and is dropped, so this can only under-recover, never attribute arguments to the wrong call. **Row 6** — the further recovery for #48171 picks a reading that corrupts: ``` [bash(command='sed -n '1,5p' a.py'), bash(command='sed -n '9,12p' b.py')] ^ ^ the only reading that parses closes here, absorbing the entire second call into the first one's command value ``` The client received one call whose `command` was `sed -n '1,5p' a.py'), bash(command='sed -n '9,12p' b.py`, which the agent then executes verbatim, and the second `sed` never happened. That is the defect: not a call that failed to parse, but a call reported as successful carrying an argument the model never wrote. Counting brackets is immune to the broken quote, so the block's call count is now the invariant: a reading that loses calls is rejected as a swallow. That alone restores the pre-#48171 outcome — the block yields no calls, and nothing wrong is emitted. The block then falls through to segment splitting, and each segment is exactly the single-call shape whose nested quote #48171 already recovers (`bash(command='sed -n '1,5p' a.py')`), so both calls come back with their intended values through the existing path rather than a new guess. **Row 7** — the salvage in row 4 mistook a normal streaming state for a failure: ``` [fetch(url='http://x/?a=1'), bash(cm <- all that has arrived ^^ parsed as bash(cm): a positional argument, because 'cmd=' is only half streamed ``` `make_valid_python` withholds a completion while a parameter name is still arriving, but it recognizes that state by counting `=` against `,` in the text before the open paren. An earlier string value that itself contains `=` — a URL query, a `sed` script, an env assignment — defeats the count, and the half-typed name completes to a bare `Name`. Three chunks of the block above land in that state; it parses perfectly the moment `=` arrives. A call the model genuinely wrote with a positional argument (rejected as of row 3) is in that state on *every* one of its chunks — 13 of them for `[read('/etc/hosts')]`. Pre-fix both raised, the streaming handler caught it and logged a full traceback: over a thousand in one production run, while not a single tool call was lost. `salvage_tool_calls` no longer decides policy — it returns what it managed to convert, and the caller decides what an empty result means: - complete block (non-streaming): raise and fall back to content, as before - mid-stream: return quietly and wait for the next chunk **Row 8** — the same shape of noise from a different failure, and older than #48171. A block whose brackets are structurally wrong cannot be completed at any prefix, and streaming re-parses from the start of the block on every chunk, so `make_valid_python` raises at the same offset until the block ends: ``` [foo(x=1])] 3 tracebacks [bash(cmd='ls -la'], timeout=30)] 6 [bash(cmd='find . -name \"*.py\" ...'], timeout=30, retries=3, ...) 25 ``` The count follows how much of the call arrives after the bad bracket, so a long argument list is the worst case, and none of the tracebacks named the text responsible. The failure is now handled where it happens: silence while the block is partial (its end has not arrived, so the requote recovery has not run yet), and one warning carrying the block once the end token is present. Which blocks are skipped does not change. The three sibling parsers report at a different moment: they have no reliable terminator (`pythonic` has none, and the tags the other two strip are optional), so they log the first text that failed — which ends at the bad bracket — once per parser instance, and a parser is constructed per request. **Row 9** — found while writing row 8's tests, and the only row that needs no malformed input. olmo3 calls are newline separated rather than bracketed, so its streaming path wraps the completed text in `[...]` itself: ``` get_weather <- name here, \"(\" not yet ^^^^^^^^^^^ wrapped into [get_weather]: parses, but the element is a Name, not a Call -> rejected ``` Every tool call passes through the state where part of its name has arrived and its `(` has not, so **every healthy olmo3 tool call logged a traceback per token of its name**: 1 for `f`, 2 for `get_weather`, 7 for `search_the_internet_for_recent_news`. The call itself was always emitted correctly, which is why this went unnoticed. Mid-stream the state is now a wait; the non-streaming path, where a bare name really is malformed output, still rejects it. Not duplicating existing work: no open PR touches `handle_single_tool` argument handling (checked via search); #43984 fixed the same non-finite class on the JSON side (`coerce_to_schema_type`) and is the precedent for rejecting non-finite arguments. ## Test Plan ```bash pytest tests/tool_parsers/ ``` Every fix has utils-level regression tests plus end-to-end cases through the lfm2 parser, and the three sibling parsers gain sibling-survival regressions. Each new test was verified to fail without its fix. ## Test Result **38 new regression cases**, each verified to fail without its fix: | File | Tests | |---|---| | `test_utils.py` | 195 | | `test_lfm2_tool_parser.py` | 57 | | `test_pythonic_tool_parser.py` | 24 | | `test_llama4_pythonic_tool_parser.py` | 26 | | `test_olmo3_tool_parser.py` | 25 | All pass, and the rest of `tests/tool_parsers/` is unaffected: the whole directory runs with **zero failures** (1013 passed; the 3 skips and 34 xfails are pre-existing markers in other parsers' suites, unchanged by this PR). pre-commit (ruff check/format, typos, mypy 3.10) and the pinned mypy 1.20.2 with the repo config: clean on all changed files. ## Notes for reviewers Block splitting is deliberately non-streaming: a partial streaming block is legitimately unparsable and must keep waiting for more text rather than be split into calls that are not finished yet. The recovery in rows 4-6 changes behavior rather than only fixing wrong output: a block that loses one of several calls can execute a dependent sequence partially (`cd` then `ls`). I judge that better than the block-level drop it replaces — the model gets an environment response and can correct — but it is a judgement call, not a pure bugfix, and I am happy to drop it if you disagree. The swallow guard in row 6 is not part of that judgement call: it stops a corrupted argument from being executed, and without the recovery it simply restores the pre-#48171 behavior of yielding no calls. Skipped calls are logged server-side with the offending AST, so the rate of malformed model output stays measurable rather than being silently absorbed. ---", "url": "https://github.com/vllm-project/vllm/pull/51649", "createdAt": "2026-08-10T09:32:18Z", "updatedAt": "2026-08-13T00:42:28Z", "timestamp": "2026-08-13T00:42:28Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "tool-calling", "llama" ], "author": "fatday", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51653", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Enable V2 model runner for Kimi-K3 on ROCm", "text": "## Purpose Kimi-K3 on ROCm was gated from using V2 model runner. After validation using the up-to-date upstream. V2 model runner is working as expected. ## Test Plan Command to start Kimi-K3 on mi355x ``` export VLLM_ROCM_USE_AITER=1 export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # enable/disable V2 as A/B test export VLLM_USE_V2_MODEL_RUNNER=0/1 vllm serve moonshotai/Kimi-K3 \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --load-format auto \\ --gpu-memory-utilization 0.95 \\ --mm-encoder-tp-mode data \\ --max-num-batched-tokens 4096 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --max-num-seqs 128 ``` ## Test Result Tested with lm-eval: ``` lm_eval --model local-chat-completions --model_args model=moonshotai/Kimi-K3,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=256,max_retries=10,max_gen_toks=4096,max_length=16384,timeout=60000 --batch_size auto --tasks gsm8k --num_fewshot 20 --apply_chat_template ``` | runner | run 1 | run 2 | |--------|-------|-------| | V1 | 96.66% | 97.04% | | V2 | 97.57% | 96.89% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51653", "createdAt": "2026-08-10T10:03:11Z", "updatedAt": "2026-08-13T14:34:54Z", "timestamp": "2026-08-13T14:34:54Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "ready", "kimi", "k3" ], "author": "vllmellm", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51655", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add Muse Glimmer model support", "text": "Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context. Adds the model, its config and processor, channel-scoped reasoning and ATEM tool-call parsers, and DFlash speculative decoding support for its draft head. The model does not emit JSON tool calls and does not wrap reasoning in tags. Every turn is a sequence of channel-scoped messages, and both parsers key off that framing, so --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer are used together. The reasoning parser forces skip_special_tokens=False; without it the markers are stripped before parsing and both channels collapse into content. tool_choice=\"required\" and named tool_choice set supports_required_and_named=False so vLLM does not apply JSON guided decoding to them -- that path assumes JSON tool calls, and forcing it here either trapped the call in the reasoning channel or leaked the raw framing into content. The DFlash draft head (MuseGlimmerAssistantModel) reuses the existing qwen3_dflash implementation: same architecture, same tensors. It reads the target's residual stream at layers [1, 13, 25, 37, 49] and predicts a 16-slot block per forward. Squashed from the onyx-support integration branch (30 commits). ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51655", "createdAt": "2026-08-10T10:20:30Z", "updatedAt": "2026-08-13T11:46:25Z", "timestamp": "2026-08-13T11:46:25Z", "metrics": { "reactions": 16, "comments": 34 }, "labels": [ "documentation", "new-model", "speculative-decoding", "ready", "tool-calling", "qwen", "mrv2" ], "author": "xianbaoqian", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51664", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Helm] Fix chart resource references", "text": "Assisted-by: Codex ## Purpose Fix inconsistent Helm chart resource references when custom labels and autoscaling are enabled. Before this change: - The Service selector used configured labels, while the Deployment selector and Pod labels were hard-coded to `test/test`. - The HPA targeted a non-existent Deployment named `vllm`. After this change, the Deployment selector, Pod labels, Service selector, and HPA target consistently reference the release-specific resources and configured labels. This does not duplicate an existing PR: [fill in after running the duplicate checks; e.g. “No overlapping open PR was found.”] ## Test Plan ```bash helm lint . helm unittest . helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/service.yaml helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/deployment.yaml helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/hpa.yaml ``` ## Test Result helm lint . 1 chart(s) linted, 0 chart(s) failed helm unittest . Charts: 1 passed, 1 total Test Suites: 5 passed, 5 total Tests: 10 passed, 10 total Before rendering with custom labels showed: - Service selector: production/qwen-serving - Deployment selector and Pod labels: test/test - HPA target: vllm After rendering with the same values showed: - Service selector, Deployment selector, and Pod labels: production/qwen-serving - HPA target: vllm-test-deployment-vllm ### Rendered manifest comparison #### Before **Service selector — already uses configured labels** \"스크린샷 **Deployment selector and Pod labels — incorrectly hard-coded to test/test** \"스크린샷 **HPA target — incorrectly points to vllm** \"스크린샷 #### After **Service selector — configured labels** \"스크린샷 **Deployment selector and Pod labels — now match the Service** \"스크린샷 **HPA target — now points to the release-specific Deployment** \"스크린샷 No model evaluation was run because this Helm-template-only change does not affect model output, accuracy, or serving behavior. - [x] The purpose of the PR is described. - [x] Test commands are provided. - [x] Test results and before/after comparison are provided. - [ ] Documentation update is not needed. This does not duplicate an existing PR: no overlapping open PR was found.", "url": "https://github.com/vllm-project/vllm/pull/51664", "createdAt": "2026-08-10T11:35:20Z", "updatedAt": "2026-08-13T05:03:03Z", "timestamp": "2026-08-13T05:03:03Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "documentation", "performance", "rocm", "ci/build", "multi-modality", "tool-calling", "qwen", "cpu", "kv-connector", "nvidia", "quantization" ], "author": "iwannagotobed", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51665", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix weight tying", "text": "Fixes #51063. vLLM decides whether to tie `lm_head` purely from `tie_word_embeddings` and never looks at the checkpoint, so a config claiming tied weights against a checkpoint with a real `lm_head` silently discards it and generates gibberish. `ModelConfig.maybe_untie_word_embeddings` now checks the safetensors metadata and builds the model untied so the head loads; after loading, `torch.equal` re-ties it if it was just a redundant copy (as in `Qwen/Qwen3-0.6B`) or warns and keeps it if it genuinely differs. This matches `PreTrainedModel.tie_weights`. The reverse case, untied config with no head in the checkpoint, now ties and warns instead of leaving the head uninitialised. That makes the per-model workarounds redundant. `AutoWeightsLoader` now skips weights whose target parameter aliases one already loaded, which removes ~50 hand-rolled `lm_head` skips and also covers the ~20 models that tie with no skip at all. Tying itself is now expressed one way. `self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)` is the only form that dispatches through `quant_method.tie_weights`, so the other two are converted to it: - `self.lm_head.weight = self.model.embed_tokens.weight` (33 sites) bypassed the quant method entirely, so it was wrong for quant methods that repack. The `ParallelLMHead` already exists at that point, so this is a one-line swap with no structural change. - `self.lm_head = self.model.embed_tokens` (23 sites) built no `ParallelLMHead` at all in the tied branch. These now always build one and tie it. Two consequences worth review: a transient vocab x hidden allocation, and the quant config is now consulted for the `lm_head` prefix where it previously never was. - Five models hardcoded tying with `assert config.tie_word_embeddings` (`gemma`, `gemma2`, `commandr`, `cohere2_moe`, `mpt`). Untying at config time would trip these, so the asserts are dropped and an untied config keeps using the embeddings exactly as it does today. Those five still have no real `lm_head`, so they cannot honour one in a checkpoint; left for a follow-up. So `AutoWeightsLoader` now skips weights whose target parameter aliases one already loaded. That removes ~50 hand-rolled `lm_head` skips and also covers the ~20 models that tie with no skip at all.", "url": "https://github.com/vllm-project/vllm/pull/51665", "createdAt": "2026-08-10T11:38:44Z", "updatedAt": "2026-08-13T16:59:14Z", "timestamp": "2026-08-13T16:59:14Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "speculative-decoding", "llama", "qwen", "gpt-oss", "mistral", "kimi", "k3", "cohere" ], "author": "hmellor", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51674", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN", "text": "## Purpose Speed up Qwen3.5 (GDN linear attention) MTP speculative decode on Blackwell. During MTP decode, the Triton path launches a chain of small kernels per step (gating, delta-rule recurrence, state rewind/update, gated RMSNorm), which leaves the GPU latency-bound at decode batch sizes. This PR adds a single fused CUDA kernel, `fused_gdn_decode_post_conv_mtp`, that consumes the post-convolution `mixed_qkv` and performs the entire GDN MTP decode step in one launch: - SiLU activation on q/k/v and gating (`softplus`/`sigmoid` on `a`, `b`, `A_log`, `dt_bias`) - the gated delta-rule recurrence over up to 8 speculative tokens per request, with rewind to the last accepted token via `num_accepted_tokens` - recurrent state update in place (BF16 or FP32 state) - gated RMSNorm epilogue, with output precision matched to the Triton path ## Test Plan New/extended tests (all require SM100): ```bash pytest tests/kernels/mamba/test_gdn_fused_mtp.py -v pytest tests/kernels/test_fused_gdn_post_conv.py -v pytest tests/v1/attention/test_gdn_metadata_builder.py -v pytest tests/test_envs.py -k \"gdn\" pytest tests/compile/test_config.py ``` `test_gdn_fused_mtp.py` covers the model-path dispatch for pure spec-decode, mixed, prefill, and regular-decode batches against the Triton reference. `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` covers the kernel directly for BF16/FP32 state, ragged acceptance patterns, and TP head counts (tp4/tp16). ## Test Result On 1x B200 (SM100), CUDA 13.1, torch 2.13.0+cu130: - `tests/kernels/mamba/test_gdn_fused_mtp.py`: **6 passed** - `tests/kernels/test_fused_gdn_post_conv.py`: **70 passed** - `tests/v1/attention/test_gdn_metadata_builder.py`: **10 passed** - `tests/test_envs.py -k gdn`: **1 passed** - `tests/compile/test_config.py`: **44 passed** ## Microbenchmark 1x B200, CUDA 13.1, torch 2.13.0+cu130. Both variants measured over the **same kernel boundary**: post-conv `mixed_qkv` in, gated+normed `core_attn_out` out, recurrent state updated in place. - Triton path = `fused_sigmoid_gating_delta_rule_update` + gated RMSNorm (`rmsnorm_fn`), exactly as dispatched by `_forward_core_fused_norm` today. - Fused path = single `fused_gdn_decode_post_conv_mtp` launch. - Timed by capturing 20 iterations in a CUDA graph and replaying 50x with CUDA events (production MTP decode runs under full CUDA graphs, so graph-replay GPU time is the relevant metric; CPU launch overhead excluded for both). - **cold** rotates input/state buffer sets so the working set exceeds 4x L2 (126 MB), i.e. state comes from HBM as in real serving where other layers evict L2 between calls; **warm** reuses one buffer set (L2-hot upper bound). - K=V=128, 4 MTP tokens/request, `num_accepted_tokens=2`. Context length does not affect either variant (fixed-size recurrent state), so results apply to any seqlen (e.g. 8192). Cold-cache speedup is **1.17x-2.20x**, largest in the latency-sensitive BS=4-32 decode regime. **TP=4 (H=4, HV=32), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 20.31 | 12.19 | 1.67x | 16.49 | 11.57 | 1.42x | | 4 | 27.91 | 12.99 | 2.15x | 20.65 | 11.73 | 1.76x | | 8 | 35.69 | 17.89 | 2.00x | 27.26 | 16.59 | 1.64x | | 16 | 46.66 | 32.32 | 1.44x | 41.44 | 30.60 | 1.35x | | 32 | 71.85 | 51.75 | 1.39x | 71.99 | 51.62 | 1.39x | | 64 | 120.65 | 97.12 | 1.24x | 120.61 | 96.93 | 1.24x | | 128 | 221.13 | 188.16 | 1.18x | 221.11 | 188.02 | 1.18x | **TP=4 (H=4, HV=32), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 23.50 | 13.73 | 1.71x | 17.74 | 12.70 | 1.40x | | 4 | 32.04 | 14.57 | 2.20x | 24.12 | 12.82 | 1.88x | | 8 | 42.69 | 20.78 | 2.05x | 33.78 | 18.68 | 1.81x | | 16 | 60.27 | 38.14 | 1.58x | 60.01 | 37.70 | 1.59x | | 32 | 96.82 | 65.27 | 1.48x | 97.08 | 65.15 | 1.49x | | 64 | 167.63 | 130.97 | 1.28x | 167.27 | 126.25 | 1.32x | | 128 | 311.52 | 267.23 | 1.17x | 314.21 | 266.75 | 1.18x | **TP=16 (H=1, HV=8), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 15.87 | 11.52 | 1.38x | 15.92 | 11.57 | 1.38x | | 4 | 19.64 | 12.32 | 1.59x | 16.46 | 11.57 | 1.42x | | 8 | 23.52 | 12.67 | 1.86x | 17.92 | 11.61 | 1.54x | | 16 | 27.56 | 12.92 | 2.13x | 20.64 | 11.69 | 1.77x | | 32 | 35.39 | 17.82 | 1.99x | 27.16 | 16.60 | 1.64x | | 64 | 46.45 | 32.27 | 1.44x | 40.72 | 30.57 | 1.33x | | 128 | 73.57 | 51.77 | 1.42x | 73.69 | 51.61 | 1.43x | **TP=16 (H=1, HV=8), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 17.44 | 12.59 | 1.38x | 17.71 | 12.61 | 1.41x | | 4 | 23.30 | 13.71 | 1.70x | 17.92 | 12.68 | 1.41x | | 8 | 26.18 | 13.93 | 1.88x | 20.08 | 12.71 | 1.58x | | 16 | 31.77 | 14.50 | 2.19x | 23.99 | 12.81 | 1.87x | | 32 | 42.81 | 20.84 | 2.05x | 34.81 | 19.05 | 1.83x | | 64 | 61.15 | 38.45 | 1.59x | 60.99 | 37.89 | 1.61x | | 128 | 98.89 | 66.26 | 1.49x | 99.21 | 66.21 | 1.50x | --- ## Model Evaluation An lm_eval score comparison was not run: the fused path requires the `num_v_heads == 8 * num_k_heads` GDN head layout, and no public Qwen3.5 checkpoint ships that config (0.8B is 16/16, 35B-A3B is 16/32, 397B-A17B is 16/64) — on those models the dispatch check never selects the fused kernel, so default behavior is unchanged. Correctness of the fused kernel against the Triton path is covered by the numerical-equivalence tests above: `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` checks kernel outputs and updated recurrent state against `fused_sigmoid_gating_delta_rule_update` + `rmsnorm_fn` across BF16/FP32 state, ragged acceptance patterns, and TP head counts, and `test_gdn_fused_mtp.py` checks the model-path dispatch (pure / mixed / prefill / decode batches) end-to-end at the layer level.
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51674", "createdAt": "2026-08-10T12:49:25Z", "updatedAt": "2026-08-13T14:50:17Z", "timestamp": "2026-08-13T14:50:17Z", "metrics": { "reactions": 5, "comments": 9 }, "labels": [ "ready", "ci/build", "qwen", "nvidia" ], "author": "Jie-Fang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51692", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM", "text": "## Purpose Activated when shapes allow for it and configs are tuned. **Implications (DSv3 1k/1K):** - TP8+DPA: +4-8% QPS - TP8+EP: +0-4% QPS ## Test Plan Bench serve & accuracy validation with DSv3 on (1) TP8+DPA (2) TP8+EP on 8xMI350. (note need to run with `VLLM_ROCM_USE_AITER_FP8BMM=0` until https://github.com/vllm-project/vllm/issues/51957 is resolved) ```bash VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_FP8BMM=0 \\ vllm serve deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ ``` Sweep concurrencies 1-128, 1k/1k ```shell vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 --num-warmups 256' ``` ```shell pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k ``` ## Test Result ### Profiling This branch: Using DPA, the MLA decode o-proj is using preshuffled weights. No other GEMM is changed. 21us in trace. \"image\" Main branch: ck-tile kernel for o-proj. 52us in trace, so **more than 2x slower that of the bpreshuffled version!** \"image\" ### TP8+DPA | Concurrency | Branch | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | |-------------|-------------|---------------|---------------|--------|----------------| | 1 | This branch | 156.9816 | 18.2976 | 0.0529 | +7.30 | | 1 | Nightly | 158.3445 | 19.7720 | 0.0493 | | | 2 | This branch | 161.0075 | 18.5311 | 0.1049 | +8.14 | | 2 | Nightly | 198.6506 | 19.9552 | 0.0970 | | | 4 | This branch | 220.7625 | 18.7688 | 0.2054 | +7.48 | | 4 | Nightly | 284.2663 | 20.1512 | 0.1911 | | | 8 | This branch | 557.7459 | 22.6129 | 0.3479 | +7.61 | | 8 | Nightly | 317.1527 | 23.8614 | 0.3233 | | | 16 | This branch | 581.6531 | 23.5224 | 0.6528 | +5.96 | | 16 | Nightly | 609.4239 | 24.8873 | 0.6161 | | | 32 | This branch | 767.6995 | 23.7180 | 1.2794 | +4.61 | | 32 | Nightly | 787.4866 | 24.8746 | 1.2230 | | | 64 | This branch | 1449.0858 | 24.3342 | 2.4326 | +5.50 | | 64 | Nightly | 1394.4616 | 25.7091 | 2.3057 | | | 128 | This branch | 2018.3187 | 25.9321 | 4.4837 | +4.90 | | 128 | Nightly | 2201.5545 | 27.2313 | 4.2742 | | ### TP8+EP | Concurrency | Branch | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | |-------------|-------------|---------------|---------------|--------|----------------| | 1 | This branch | 108.3783 | 12.6642 | 0.0765 | -0.39 | | 1 | Nightly | 113.4111 | 12.6244 | 0.0768 | | | 2 | This branch | 172.7154 | 13.6376 | 0.1416 | +0.14 | | 2 | Nightly | 178.4821 | 13.6537 | 0.1414 | | | 4 | This branch | 280.6493 | 13.3944 | 0.2861 | +0.42 | | 4 | Nightly | 287.1797 | 13.446 | 0.2849 | | | 8 | This branch | 424.7564 | 13.5723 | 0.5592 | +0.81 | | 8 | Nightly | 427.5746 | 13.6773 | 0.5547 | | | 16 | This branch | 479.1289 | 14.4931 | 1.0434 | +0.69 | | 16 | Nightly | 484.925 | 14.6155 | 1.0363 | | | 32 | This branch | 817.9601 | 15.6236 | 1.9015 | +3.95 | | 32 | Nightly | 821.0856 | 15.8782 | 1.8292 | | | 64 | This branch | 938.6096 | 21.6289 | 2.7676 | +9.01 | | 64 | Nightly | 932.3192 | 23.9109 | 2.5388 | | | 128 | This branch | 1064.6245 | 27.3408 | 4.4065 | +4.04 | | 128 | Nightly | 1079.7563 | 28.087 | 4.2355 | | ### GSM8k This branch (TP8+DPA) |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9477|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9484|± |0.006 Nightly |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9439|± |0.0063| | | |strict-match | 5|exact_match|↑ |0.9431|± |0.0064| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51692", "createdAt": "2026-08-10T14:54:38Z", "updatedAt": "2026-08-13T12:19:17Z", "timestamp": "2026-08-13T12:19:17Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "simondanielsson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51695", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MOE] Standardize and abstract fused shared expert optimization selection", "text": "## Disclosure AI assistance was used. The changes were reviewed and tested manually. ## Purpose Standardize fused shared-expert (FSE) detection so model construction and checkpoint loading use the same quantization-compatible decision **throughout all models implementing FSE**. This is e.g. useful for `shared_expert` quantization through various quantization methods (e.g. online quantization in https://github.com/vllm-project/vllm/pull/51285 / https://github.com/vllm-project/vllm/pull/51392), so that FSE compatibility can be detected more largely based on arbitrary quantization config across all model archs, and not simply based on https://github.com/vllm-project/vllm/blob/fac808b36f502d0d992509a187dca94c68b0360a/vllm/model_executor/models/qwen3_next.py#L81-L98 (qwen3 specific, quark specific) This is also useful e.g. for https://github.com/vllm-project/vllm/issues/50963 that is NOT rocm/aiter specific. The shared resolver validates FSE against the quantization configuration, rejects an incompatible requested FSE configuration, and ensures all MoE layers in a model agree on the resolved state. The mixed-state error reports the counts of enabled and disabled layers. Behavior change compared with `main`: - When `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS` is enabled but quantization is incompatible, model construction now raises `ValueError` instead of falling back to the non-fused shared-expert path. - Models with mixed resolved FSE states now raise `NotImplementedError`. FSE is only supported/tested for all (or none) layers currently. Left to do: - [ ] Maybe left to an other PR: Should `not get_current_vllm_config().parallel_config.enable_expert_parallel` be promoted globally (from `vllm/models/minimax_m3/amd/model.py`) - [ ] Maybe left to an other PR: Should `on_gfx950()` from minimax_m3 be promoted (// backend specific?)? - [x] update `deepseek_v4/amd/model.py` - [ ] Reliance on `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS` for non-AITER backends to turn on FSE in MiniMax-M3 on AMD https://github.com/vllm-project/vllm/blob/0e2d78028c473bb04fc74bf945eb32da29239b0c/vllm/models/minimax_m3/amd/model.py#L112-L127 ==> does not require `VLLM_USE_AITER=1`, does not use `rocm_aiter_ops.is_fusion_moe_shared_experts_enabled()` - [ ] Check on `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` once https://github.com/vllm-project/vllm/pull/48223 is merged ## Test Plan - `pytest tests/model_executor/layers/test_fused_shared_expert.py -v` ## Test Result Tests pass.", "url": "https://github.com/vllm-project/vllm/pull/51695", "createdAt": "2026-08-10T15:04:56Z", "updatedAt": "2026-08-13T12:51:46Z", "timestamp": "2026-08-13T12:51:46Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "qwen", "deepseek", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51700", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO)", "text": " DBO for Model Runner V2 (RFC: #50738 ) is two PRs: #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) -> #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) DBO for Model Runner V2 ([RFC #50738](https://github.com/vllm-project/vllm/issues/50738)) is two PRs: Stacked on top of #50945; the diff below contains that PR's commits too, so review only the last three (`[Feat][WIP] FULL cudagraph capture…` onward). @RyanJHamby wrote the first of them ([draft](https://github.com/RyanJHamby/vllm/commit/327520123d7a0903b28685a4282080a01dafcc2a)); the other two are the fixes needed to make it run on a GPU. ## Summary #50945 runs microbatched steps eager. This PR captures and replays them as FULL CUDA graphs: at replay there are no microbatch threads, no `threading.Event` handoffs and no per-microbatch Python, so the CPU launch path that dominates eager decode disappears and the interleaving the handoff protocol produces during capture is frozen into the graph. Scope is FULL only. `deepep_high_throughput` + DP>1 already forces `cudagraph_mode=NONE` in `compilation.py`, so capturing microbatched steps only ever applies to the decode-side backends (`deepep_low_latency` / `nixl_ep` / `allgather_reducescatter`). PIECEWISE + DBO stays unsupported, as in the V1 runner. ## Design **Candidate generation.** `_init_candidates` emits a `num_ubatches=2` twin for every FULL capture candidate, in both the uniform-decode and the mixed branch (`_maybe_ubatch_twin`). The decode branch matters: backends that only support uniform batches (MLA) get downgraded to `FULL_AND_PIECEWISE`, so their FULL graphs *are* the uniform-decode ones — generating twins only in the mixed branch captures nothing at all on those models, which is what the first GPU run of the draft showed. The gate is `UBatchRunner.wants_ubatch`, the same predicate the DP handshake votes with, so captured shapes cannot drift from dispatchable ones. It has to stay a pure function of config: ranks dispatch independently after the all-reduce, and a rank that replays while another runs eager would hang the expert all-to-all. **Dispatch.** The microbatching branch of `sync_cudagraph_and_dp_padding` stops hardcoding `cg_mode=NONE` and goes through `cudagraph_manager.dispatch(..., num_ubatches=...)`, asked with the synced uniform token count so uniform-decode graphs can match. Its existing fallback covers \"no graph for this shape\", so microbatched steps run either way. Every rank then pads to `desc.num_tokens` rather than the raw cross-rank maximum, because dispatch rounds up to the captured size. **Capture.** `UBatchRunner.run()` splits into `begin_capturable_run()` (spawn threads, wait the ready barrier — pure CPU/event work, outside the graph) and a `finish()` callback (handoff and join, inside the graph), mirroring V1's `gpu_ubatch_wrapper.py::_capture_ubatches`. Two things V1 does that a naive port misses, both of which stopped the draft from capturing anything: - the graph must be captured **on the stream the microbatch threads launch onto**. `torch.cuda.graph()` opens its own side stream unless told otherwise, so the threads' work would land outside the graph. `UBatchRunner` now owns a `capture_stream` that `CudaGraphManager` passes as `torch.cuda.graph(..., stream=...)`. - each microbatch thread initializes its CUDA and cuBLAS state **before** the capture opens — cuBLAS allocates its workspace on first use per stream and allocating mid-capture is illegal — and sets its device index, which a fresh thread does not inherit. **Attention metadata.** `UBatchRunner.prepare()` takes the descriptor's `cg_mode` instead of hardcoding `NONE`, so FULL builds per-microbatch metadata against the padded sizes; that is what keeps shapes and buffer addresses identical between capture and replay. `for_capture` is threaded through for the worst-case `max_seq_len`, as the non-ubatched path does. **Replay.** `execute_model` replays through `run_fullgraph(batch_desc)` with no threads and no `ubatch_runner.forward_context`, matching the non-ubatched FULL branch. `prepare()` still runs: it is what writes this step's `query_start_loc` / `seq_lens` into the persistent per-microbatch buffers the graph reads from. Everything else from #50945 is unchanged: `ubatching.py` and the `dbo_yield*` hooks, the slicing, the persistent buffers, the per-microbatch metadata builders, the DP handshake. That was the P0 bet — the buffers have been capture-safe from the start, so this is capture plumbing rather than a re-architecture. ## Testing DeepSeek-V2-Lite and Qwen3-30B-A3B-Thinking-2507-FP8, `DP2 / TP1 / EP2`, `nixl_ep`, 2× L20X, `VLLM_USE_V2_MODEL_RUNNER=1`. ```bash pre-commit run --files # ruff, ruff-format, mypy-3.10: pass .venv/bin/python -m pytest tests/v1/worker/test_gpu_ubatch_slicing.py -q # 19 passed (GPU), 15 passed / 4 skipped (CPU-only) ``` Capture happens: with DBO enabled, on Qwen3-30B-A3B, 0.46 → 0.96 GiB. Both DP ranks identical. Replay happens: an instrumented build logging the branch taken in `execute_model` shows microbatched graphs being replayed under real traffic on both models, and mixed batches correctly falling back to eager microbatching on MLA, which has no mixed FULL graph: ``` replay cg_mode=FULL, num_tokens=32..64, uniform_token_count=1, num_ubatches=2 plain-full cg_mode=FULL, num_tokens=1..32, uniform_token_count=1, num_ubatches=1 (below the DBO threshold) eager-ubatch cg_mode=NONE, num_tokens=609, uniform_token_count=None, num_ubatches=2 (mixed batch, MLA) ``` ## Accuracy GSM8K, 1319 questions, 5-shot, greedy. | build | DeepSeek-V2-Lite | Qwen3-30B-A3B | |---|---|---| | main (`11ba93f3`), no DBO | 0.3457 | — | | #50945, no DBO | 0.3495 | — | | #50945, `--enable-dbo` (eager) | 0.3563 | — | | this PR, no DBO | — | 0.5451 | | this PR, `--enable-dbo` + FULL CG | 0.3503 | 0.5383 | Run-to-run spread is ~1pp on this harness, so microbatching with captured graphs is accuracy-neutral on both models. For reference, an earlier broken revision scored 0.135–0.149 with a 5–6% invalid rate on DeepSeek-V2-Lite, so a regression here is unmistakable. ## Known limits - Perf validation is deferred to P4 and needs cross-node EP; the single-node numbers I can produce here are not representative of what DBO is for. - PIECEWISE + DBO remains unsupported, matching the V1 runner. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51700", "createdAt": "2026-08-10T15:44:41Z", "updatedAt": "2026-08-13T02:23:55Z", "timestamp": "2026-08-13T02:23:55Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "nvidia", "mrv2" ], "author": "specture724", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec", "text": "## Purpose Part of the KV-cache layout standardization series (RFC #42082). **Stacked on #51612** — the diff shown includes it until that lands and this retargets `main`. Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants in MLA spec overrides. This PR makes specs plain data and moves each packed format to the backend/component that owns it: - `AttentionSpec` gains two optional packing fields `num_head_slots` (H in standard layout) and`state_content_bytes` (C in standard layout). Page sizes derive uniformly as `num_heads × storage_block_size × content`. > NOTE: i dont love the num_head_slots name but this should hopefully be temporary until we can [fully refactor the backends](https://github.com/vllm-project/vllm/issues/42449); i.e. separate allocation from connector and kv-cache manager concerns - `AttentionBackend.customize_spec(spec)` is a temporary hook until we can move to having the backend return the spec instead of the layer (see: https://github.com/vllm-project/vllm/issues/42449) Prepares the ground for the layout-standardization PR ([[6/N]](https://github.com/vllm-project/vllm/pull/51718)), where these fields describe the standardized `[B, H, N, C]` page and the AITER backends publish separate K/V head groups through the same hook. --- AI-assisted (Claude); reviewed by submitter", "url": "https://github.com/vllm-project/vllm/pull/51704", "createdAt": "2026-08-10T16:20:40Z", "updatedAt": "2026-08-13T17:08:18Z", "timestamp": "2026-08-13T17:08:18Z", "metrics": { "reactions": 2, "comments": 23 }, "labels": [ "rocm", "ready", "nvidia", "ready-run-all-tests", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51705", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][DSpark][DCP] Support decode context parallelism for Kimi-K3 DSpark", "text": "## Purpose Enable decode context parallelism (DCP) for Kimi-K3 with the DSpark drafter on ROCm. Two things stood in the way: 1. **Hybrid KV cache grouping was not drafter-invariant.** The DSpark draft adds an MLA-only attention type with far fewer layers than the target. Since group size was `min(bucket_sizes)`, that small bucket shrank `group_size` and over-split the target's mamba layers. It also made an engine holding the drafter group its mamba layers differently from one without it, which breaks KV transfer in PD-disaggregated serving. 2. **The draft KV cache group cannot be DCP-sharded.** The DSpark draft attends over the whole sequence and discards its decode LSE, so it needs replicated KV while the target stays sharded. ### Changes **Grouping** — exclude draft-only buckets, identified by the existing `non_causal_multi_token_decode` spec flag, when choosing the group size. The draft bucket is then padded up to the target's group size instead of setting it; for K3 DSpark that is the 19 padding layers vLLM already logs, which is the deliberate trade for keeping the target's grouping stable. Models without a drafter are unaffected because no spec carries the flag. **Replicated draft group** — whole-sequence block tables, an unsharded slot mapping so every rank writes the full sequence, `dcp_world_size=1` metadata, and an undivided capacity estimate. `MultiHeadLatentAttention` accepts DCP for these groups and pins their impl to `dcp_world_size=1`; the config-time guard that rejected the combination is removed. **Target decode LSE** — single-token decode runs through aiter gluon with `return_lse=True` so per-rank partials can be merged. `AiterMLAImpl` declares `can_return_lse_for_decode` and keeps queries in bf16 on the gluon path. Multi-token decode (`q_len > 1`) under DCP has no gluon kernel yet and raises `NotImplementedError` rather than returning partials without an LSE. Also included: two K3 GDN decode-kernel fixes that surface under spec decode — a strided `decode_conv_indices` view needs compaction, and `state_indices` only needs unit stride when `numel > 1`. ## Test Plan Single node, 8×MI355X, `Kimi-K3` (mxfp4) + `Kimi-K3-DSpark` draft, TP=8, spec decode on (`num_speculative_tokens=7`, `method=dspark`, draft `attention_backend=TRITON_MLA`, `draft_sample_method=probabilistic`, `rejection_sample_method=block`). Baseline is TP=8 without DCP; the DCP runs add `--decode-context-parallel-size N` and are otherwise byte-identical. 1. **GSM8K**, 200 items from the `openai/gsm8k` test split, `temperature=0`, `max_tokens=1024`, concurrency 16, scoring the last integer in `content`. 2. **Configuration coverage**: `decode_context_parallel_size` 1, 2 and 8, each with spec decode on and off, at native (1M) and capped context. ## Test Result TP8 and DCP=8 below are a matched pair from the same build. ### Accuracy and draft acceptance — GSM8K, 200 items | Config | Score | Accuracy | Mean acceptance length | |---|---|---|---| | TP8 (dcp=1) | 192/200 | 96.0% | 4.88 | | DCP=8 | 193/200 | 96.5% | 4.55 | Repeat runs land at 192–195/200 for TP8 and 193–196/200 for DCP=8, so the overlapping ranges are run-to-run variation from concurrent batching (`temperature=0` fixes a single request, not the batch composition it lands in). Acceptance length varies over 4.55–4.96 across the same runs for both configurations. ### Throughput — GSM8K phase, 200 items @ concurrency 16 | Config | Wall time | Mean generation throughput | Peak | |---|---|---|---| | TP8 (dcp=1) | 107 s | 270.0 tok/s | 394.5 tok/s | | DCP=8 | 117 s | 261.2 tok/s | 395.3 tok/s | Within run-to-run spread (a repeat DCP=8 run finished in 90 s at 310.6 tok/s), so these are on par rather than a measured difference in either direction. ### KV cache capacity Replicating only the draft group preserves the target's sharding benefit: | Config | spec on | spec off | |---|---|---| | TP8 (dcp=1) | 956,229 tokens | 2,319,404 tokens | | DCP=8 | 1,506,430 tokens | 12,426,353 tokens | ### Grouping The reported KV cache sizes and the padding-layer warnings are identical to a build without the grouping change, for both TP8 and DCP=8, so grouping is unchanged for models this does not target. ### Configuration coverage `decode_context_parallel_size` 1, 2 and 8 all start and serve, with spec decode on and off, at both native 1M context and capped context. `dcp=2` also scores 8/8 on a short GSM8K gate at 128K and at native context. Before this PR, `decode_context_parallel_size > 1` with the DSpark drafter was rejected at config time.", "url": "https://github.com/vllm-project/vllm/pull/51705", "createdAt": "2026-08-10T16:40:18Z", "updatedAt": "2026-08-13T10:22:34Z", "timestamp": "2026-08-13T10:22:34Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "needs-rebase", "mrv2", "kimi", "k3" ], "author": "YukioZzz", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51707", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Humming] Fuse Kimi SiTU activation + per-token FP8 quant on w2 path", "text": "Draft. Stacked on #51332 (Humming MXFP4 w + block-FP8 activation) and #51673 (SiTU `valid_rows` early-exit) — only the top commit is this PR's contribution; the rest are those two open PRs. Do not merge until both land. Fuses the Kimi SiTU (SituGLU) activation and the following per-token dynamic FP8 (e4m3) quantization of the Humming down-projection (w2) input into one CUDA kernel (`situ_and_mul_quant`), replacing the separate `situ_and_mul` + `quant_input` passes and skipping the intermediate bf16 activation buffer. - Bit-for-bit identical to the unfused path (activation rounded through `scalar_t` before abs-max/quant; per-token scale = absmax/448, dequant = q*scale, matching humming `calc_scale`). - Honors the DeepEP v2 contiguous-layout `valid_rows` count from #51673: padding rows are skipped and get a benign scale. - Off by default; behind `VLLM_HUMMING_FUSE_ACT_QUANT`, gated on SiTU + per-token FP8/float32-scale w2. Two-step path preserved as fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51707", "createdAt": "2026-08-10T16:54:51Z", "updatedAt": "2026-08-13T10:20:12Z", "timestamp": "2026-08-13T10:20:12Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization", "kimi" ], "author": "elvircrn", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51714", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][DSV4] Opt-in AITER gluon kernel for sparse-MLA decode on gfx950", "text": "## Purpose Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair (`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). ## Changes Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): - `VLLM_ROCM_DSV4_SPARSE_GLUON` (**default off**). Read once at import — this sits on the per-step decode path, so re-reading the environment per call would be pure overhead. With the flag unset the decode path is byte-for-byte unchanged. - The gluon entrypoint is resolved lazily behind `functools.cache`, so builds whose AITER lacks the kernel are unaffected. - A failed launch latches a process-wide flag and degrades to Triton for the rest of the process, rather than taking the server down mid-request. - The first successful dispatch logs once, so enabling the flag is observable rather than silent. - Additionally gated on `_ON_GFX950`. ## Requires ROCm/aiter#4673 The kernel needs [ROCm/aiter#4673](https://github.com/ROCm/aiter/pull/4673) to be correct on large KV pools. Without it the gluon driver decides `use_buffer_load` from `nelement() * element_size()`, which undercounts a strided pool whose addressable span exceeds a 32-bit offset; past 2 GiB the `buffer_load` offsets wrap and the gather silently returns zeros. An earlier revision of this PR refused to run under DP-attention, citing [ROCm/aiter#4667](https://github.com/ROCm/aiter/issues/4667) (gsm8k 1.00 → 0.30 on EP8 + DP-attention). That guard has been removed: #4667 is the overflow above, and DP-attention was only ever the thing that grew the pool far enough to cross the boundary — not a property of the kernel. Gating on `data_parallel_size` therefore disabled the kernel on exactly the configuration it was written for. With #4673 applied, DP-attention runs clean. ## Test Plan - DeepSeek-V4 on MI355X (gfx950) with `VLLM_ROCM_DSV4_SPARSE_GLUON=0` and `=1`, comparing decode outputs for parity and measuring decode throughput. - Cover both `swa_only` (single-segment) and two-loop layers. - Confirm the fallback latch by forcing a launch failure. - Cover DP-attention (DEP8) with aiter#4673 applied. ## Test Result | Concurrency | Output tok/s (base → +PR) | Δ | Median TPOT (base → +PR) | Δ | |---|---|---|---|---| | 8 | 284.79 → 291.33 | **+2.30%** | 24.67 → 24.16 ms | −2.04% | | 16 | 483.69 → 488.59 | **+1.01%** | 29.32 → 29.02 ms | −1.03% | | 32 | 708.51 → 712.89 | **+0.62%** | 40.78 → 40.52 ms | −0.65% | | 64 | 955.23 → 958.04 | **+0.29%** | 62.55 → 62.36 ms | −0.30% | ### AgentX agentic replay (fast mode, TP=8, concurrency 32, MTP on) | Metric | base (Triton) | +PR (gluon) | Δ | |---|---|---|---| | Output tok/s | 430.76 | 453.23 | **+5.22%** | | Output tok/s/GPU | 53.84 | 56.65 | **+5.22%** | | Per-user tok/s | 32.72 | 39.96 | **+22.12%** | | ITL avg | 39.83 ms | 34.60 ms | **−13.11%** | | ITL p50 | 35.87 ms | 26.44 ms | **−26.30%** | | Time-to-2nd-token p50 | 38.19 ms | 21.79 ms | **−42.95%** | | TTFT p50 | 656.7 ms | 700.6 ms | +6.68% (regression) | ### DP-attention, with aiter#4673 applied DEP8 (DP-attention + EP8) + MegaMoE + gluon, MI355X, agentic replay at concurrency 96: **12,244 tok/s/GPU**. All eight DP ranks log the gluon path engaging. This is the configuration the removed guard used to refuse.", "url": "https://github.com/vllm-project/vllm/pull/51714", "createdAt": "2026-08-10T18:00:26Z", "updatedAt": "2026-08-13T16:40:41Z", "timestamp": "2026-08-13T16:40:41Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "jiacao-amd", "state": "open", "assignees": [ "shen-shanshan" ] }, { "id": "github:vllm-project/vllm:pull_request:51718", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[6/N][KV-Cache Layout Refactor] Standardize KV cache layout", "text": "## Purpose The core of the KV-cache layout standardization series (RFC #42082). **Stacked on #51704 → #51612** — the diff shown includes them until they land and this retargets. Standardizes every KV cache allocation on the logical `[L, B, H, N, C]` vocabulary: - `KVCacheLayout` enumerates the physical stride permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`, `BLNHC`, `BHLNC`); `reshape_kv_cache` views each flat allocation as dense `[num_blocks, block_bytes]` tiles; per-layer caches are always logical `[B, H, N, C]` views with the physical layout expressed via strides (MLA binds the squeezed `[B, N, C]`). - Layout resolution has a single writer: attention-backend selection publishes the layout into `CacheConfig` (test override > backend-required > `VLLM_KV_CACHE_LAYOUT` > connector preference > `LBNHC`); every consumer reads it from there. - The AITER backends opt into separate K/V head groups (`[B, 2, N, H*hs]`, `LBHNC` required) through `customize_spec`, keeping the fused QK-norm+RoPE+cache kernel dispatched with the exact block-interior contract aiter v0.1.19 asserts — fusion never regresses at any commit in the series. - The KV block zeroer matches main's per-segment form (block dim is always 0 in standardized views); block copies are a plain `unflatten` + index copy; connectors (NIXL, mooncake, moriio, hf3fs, offloading, example) and the attention benchmark runner consume the standardized views and published layout. - `indexes_kv_by_block_stride` and per-backend cache-shape/stride-order hooks are removed — the standardized views make them constant. Replaces #44458 (same content, restacked). ## Test Plan / Result Validated on B200 (full history in #44458): backend-correctness suites (FlashAttention/FlashInfer/Triton/Flex 127P), full MLA suite and `test_prefix_prefill` at exact main failure-set parity, connector unit battery at parity, zeroer/attn-utils/model-runner suites green, e2e smokes across FLASH_ATTN / TRITON(+int8 per-token-head) / FLASHINFER / FLEX / fp8 / CUTLASS_MLA (DeepSeek-V2-Lite). ROCm/AITER validation in progress on AMD. --- AI-assisted (Claude); under review by submitter ## Validation CPU suites (kv-cache-utils, packing, zeroer, attn-utils, gpu-model-runner, nixl / mooncake / moriio / offloading / simple-offload): green. The only failures on this host are environmental and reproduce identically on the base branch (same 5 test ids): GPU 0 is held by another process, and NIXL is not installed. `tests/v1/attention/test_mla_backends.py`: 1729 passed. 20 more OOM'd against a GPU another process had filled; rerun on a reserved GPU they pass (290 passed, 0 failed). E2E on this branch, all deterministic on re-run: | model | exercises | result | |---|---|---| | `Qwen/Qwen3-0.6B` | dense attention | pass | | `deepseek-ai/DeepSeek-V2-Lite-Chat` | MLA specs, compressed KV | pass under LBNHC, LBHNC and BLHNC | | `ibm-granite/granite-4.0-h-tiny` | hybrid attention + Mamba | pass | Not run here (needs a bigger box): DeepSeek-V4 and MiniMax-M3 themselves, so the DSV4 multi-group overlay, the `fp8_ds_mla` 584/656-byte paths, the compact MXFP4 indexer and M3's sparse/indexer backends have unit coverage but no e2e. Those paths are covered by `tests/v1/core/test_contiguous_kv_packing.py` and `tests/v1/worker/test_dsv4_packed_zeroer_geometry.py` using the real DSV4 page sizes. To reproduce e2e: ```bash vllm serve deepseek-ai/DeepSeek-V4 --kv-cache-dtype fp8_ds_mla vllm serve MiniMaxAI/MiniMax-M3 # and repeat each with VLLM_KV_CACHE_LAYOUT=BLHNC ``` --- AI-assisted (Claude); every changed line reviewed and tests run by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/51718", "createdAt": "2026-08-10T18:30:58Z", "updatedAt": "2026-08-13T06:26:04Z", "timestamp": "2026-08-13T06:26:04Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "documentation", "performance", "rocm", "intel-gpu", "ci/build", "deepseek", "cpu", "kv-connector", "nvidia", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51723", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Clamp block table indices in align mode to prevent OOB gather", "text": "In mamba_cache_mode='align' with speculative decoding, sequences with seq_len near max_model_len caused indices_to_gather to exceed the block table column count, triggering a CUDA device-side assert in torch.gather. Fixes #42084 ## Purpose Fix an out-of-bounds `torch.gather` in `mamba_get_block_table_tensor` when `mamba_cache_mode='align'` is used together with speculative decoding and a large `max_model_len`. In `align` mode the Mamba block table has shape `(#requests, cdiv(max_model_len, block_size))` — one column per block over the full context window, with **no** extra speculative columns appended. When computing the gather indices: ```python start_indices = (seq_lens - 1) // kv_cache_spec.block_size offsets = torch.arange(1 + kv_cache_spec.num_speculative_blocks, ...) indices_to_gather = (start_indices.unsqueeze(1) + offsets).to(torch.int64) ``` for a sequence with `seq_len` near `max_model_len`, `start_indices` is at or near the last valid column. Adding `num_speculative_blocks` (which is `num_speculative_tokens` when using a draft model) then pushes `indices_to_gather` past the end of the block table, triggering: ``` torch.AcceleratorError: CUDA error: device-side assert triggered ``` This crashes the entire EngineCore process, killing all in-flight requests. **Fix:** add a single upper-bound clamp on `indices_to_gather` before the gather call, mirroring the identical `.clip(max=block_table.shape[1] - 1)` pattern already used in `make_local_attention_virtual_batches` (line 452 in the same file): ```python indices_to_gather.clamp_(max=block_table.shape[1] - 1) ``` Fixes #42084 ## Test Plan New CPU-only regression test added to `tests/v1/attention/test_mamba_update_block_table.py`: ```bash pytest tests/v1/attention/test_mamba_update_block_table.py::test_mamba_get_block_table_tensor_align_mode_oob_clamp -v ``` The test constructs a block table sized for `align` mode (`cdiv(max_model_len, block_size)` columns, no speculative tail), then calls `mamba_get_block_table_tensor` with `seq_lens` near `max_model_len` and `num_speculative_blocks=2`. Without the fix the test raises an index error; with the fix it passes and verifies the output shape and value bounds. ## Test Result **Before fix:** ``` IndexError: index out of bounds ``` **After fix:** ``` PASSED tests/v1/attention/test_mamba_update_block_table.py::test_mamba_get_block_table_tensor_align_mode_oob_clamp ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51723", "createdAt": "2026-08-10T19:13:48Z", "updatedAt": "2026-08-12T18:48:11Z", "timestamp": "2026-08-12T18:48:11Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "shernshiou", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51734", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "replace batch_norm to numerically identical without cudnn", "text": "Fixes #51717 ## Purpose With `--mm-device-do-normalize` (added in #50411), image normalization runs on device through `FusedInputNorm`. That module implements the per-channel affine `output = (input * rescale_factor - mean) / std` by calling `F.batch_norm` with `running_mean=0`, `running_var=1`, `eps=0`, `weight=1/std`, `bias=-mean/std`. On CUDA, `F.batch_norm` dispatches to cuDNN's batch-norm kernels, whose batch dimension is bounded by the CUDA grid limit (~65535). In `FusedInputNorm` the batch dimension is the number of patches, which grows unbounded with image resolution and with the number of images batched into a forward pass. On large, image-heavy requests (e.g. 2560x1440 inputs on Qwen2.5-VL) the patch count exceeds the limit and cuDNN raises: ``` RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR ``` ### Fix The operation is a plain per-channel affine map — there is no statistical behavior to preserve. Replace the `F.batch_norm` call with the equivalent broadcasted multiply-add: ## Test Plan # Just the new tests: .venv/bin/python -m pytest tests/models/test_vision.py -k fused_input_norm -v # The whole vision suite : .venv/bin/python -m pytest tests/models/test_vision.py -v ## Test Result New one : $ pytest tests/models/test_vision.py -k fused_input_norm -v tests/models/test_vision.py::test_fused_input_norm_matches_reference[1] PASSED tests/models/test_vision.py::test_fused_input_norm_matches_reference[37] PASSED tests/models/test_vision.py::test_fused_input_norm_matches_reference[70000] PASSED tests/models/test_vision.py::test_fused_input_norm_identity_passthrough PASSED ================= 4 passed, 19 deselected, 16 warnings in 2.96s ================= Whole vision suite: platform darwin -- Python 3.10.20, pytest-9.0.3, pluggy-1.6.0 -- /opt/anaconda3/envs/vllm-dev/bin/python3.10 cachedir: .pytest_cache rootdir: /Users/khushalidesai/Desktop/vllm configfile: pyproject.toml plugins: anyio-4.13.0 collected 23 items tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers0-10-10-expected_features0] PASSED [ 4%] tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers1-10-10-expected_features1] PASSED [ 8%] tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers2-10-20-expected_features2] PASSED [ 13%] tests/models/test_vision.py::test_resolve_visual_encoder_outputs[select_layers3-10-20-expected_features3] PASSED [ 17%] tests/models/test_vision.py::test_run_dp_sharded_vision_model[1] SKIPPED [ 21%] tests/models/test_vision.py::test_run_dp_sharded_vision_model[4] SKIPPED [ 26%] tests/models/test_vision.py::test_run_dp_sharded_vision_model[5] SKIPPED [ 30%] tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes0-2-expected_shuffle_indices0-expected_gpu_sample_counts0-expected_grouped_sizes_per_gpu0-empty input] PASSED [ 34%] tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes1-4-expected_shuffle_indices1-expected_gpu_sample_counts1-expected_grouped_sizes_per_gpu1-fewer samples than GPUs] PASSED [ 39%] tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes2-1-expected_shuffle_indices2-expected_gpu_sample_counts2-expected_grouped_sizes_per_gpu2-single GPU] PASSED [ 43%] tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes3-2-expected_shuffle_indices3-expected_gpu_sample_counts3-expected_grouped_sizes_per_gpu3-balanced assignment] PASSED [ 47%] tests/models/test_vision.py::test_get_load_balance_assignment_cases[sizes4-2-expected_shuffle_indices4-expected_gpu_sample_counts4-expected_grouped_sizes_per_gpu4-unbalanced sizes] PASSED [ 52%] tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[1] SKIPPED [ 56%] tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[3] SKIPPED [ 60%] tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model[5] SKIPPED [ 65%] tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model_empty_input SKIPPED [ 69%] tests/models/test_vision.py::test_run_dp_sharded_mrope_vision_model_uneven_load SKIPPED [ 73%] tests/models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[2] PASSED [ 78%] tests/models/test_vision.py::test_simple_mrope_vision_model_spatial_merge[4] PASSED [ 82%] tests/models/test_vision.py::test_fused_input_norm_matches_reference[1] PASSED [ 86%] tests/models/test_vision.py::test_fused_input_norm_matches_reference[37] PASSED [ 91%] tests/models/test_vision.py::test_fused_input_norm_matches_reference[70000] PASSED [ 95%] tests/models/test_vision.py::test_fused_input_norm_identity_passthrough PASSED [100%] ================== 15 passed, 8 skipped, 16 warnings in 7.55s =================== Calude was used to develop this fix, each line is read and reviewed by me. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51734", "timestamp": "2026-08-12T16:48:21Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "ready", "multi-modality", "verified" ], "author": "khushali9", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:51742", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Fix KV cache group explosion for hybrid models", "text": "## Purpose When profiling Qwen/Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash, we found the KV cache group_size become 1. This increases build metadata overhead. We found the target model Qwen/Qwen3.6-35B-A3B layer buckets are [30 mamba, 10 full]. The DFlash drafter z-lab/Qwen3.6-35B-A3B-DFlash adds [5 sliding, 1 full], the added 1 full attention layer that can't merge with the target's 10 full attention layers. The current logic used `min` as the group size [here](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/core/kv_cache_utils.py#L1245-L1246). `group_size = min([30, 10, 5, 1]) = 1` makes it to split into 46 groups of 1 layer each. So this PR update `_get_kv_cache_groups_uniform_page_size` to exclude buckets with 1 layer from the `min` computation. Result: layers [30 mamba, 10 full, 5 sliding, 1 full] become 10 groups instead of 46 groups. e2e output throughput improves 102% at batch size 1 on H200. ``` Group 0: MambaSpec, layers=5 Group 1: MambaSpec, layers=5 Group 2: MambaSpec, layers=5 Group 3: MambaSpec, layers=5 Group 4: MambaSpec, layers=5 Group 5: MambaSpec, layers=5 Group 6: FullAttentionSpec, layers=5 Group 7: FullAttentionSpec, layers=5 Group 8: SlidingWindowSpec, layers=5 Group 9: FullAttentionSpec, layers=1 ``` Note: This PR does not cause regression for other models. ``` ┌─────────────────────────────────────────┬───────────────────────────┬────────────────────────────┐ │ Model │ Before │ After │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B │ 4 groups (10 layers each) │ 4 groups (unchanged) │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B + DFlash │ 46 groups (1 layer each) │ 10 groups │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B + Eagle │ 41 groups (1 layer each) │ 5 groups │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ gemma-4-26B-A4B-it (5:1 sw/full) │ 6 groups │ 6 groups (unchanged) │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ gpt-oss-20b (12 sw + 13 full) │ 2 groups │ 2 groups (unchanged) │ └─────────────────────────────────────────┴───────────────────────────┴────────────────────────────┘ ``` ## Test Plan ``` pytest -s -v tests/v1/core/test_kv_cache_utils.py ``` ## Test Result Unit tests passed. ## Profiling Profiled Qwen/Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash on H200. Main: \"Screenshot PR: \"Screenshot Main: `build_attn_metadata` takes around 31ms, because [attn_metadata_builder.build](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/worker/gpu/attn_utils.py#L583) are called 46 times. PR: `build_attn_metadata` takes around 6ms, because [attn_metadata_builder.build](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/worker/gpu/attn_utils.py#L583) are called 10 times. ## Benchmark * **DFlash** ``` vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"z-lab/Qwen3.6-35B-A3B-DFlash\",\"method\":\"dflash\",\"num_speculative_tokens\":8}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 97.79 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.61 Output token throughput (tok/s): 184.07 Peak output token throughput (tok/s): 60.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 351.88 ---------------Time to First Token---------------- Mean TTFT (ms): 138.20 Median TTFT (ms): 132.91 P99 TTFT (ms): 723.12 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.99 Median TPOT (ms): 4.71 P99 TPOT (ms): 16.71 ---------------Inter-token Latency---------------- Mean ITL (ms): 17.07 Median ITL (ms): 17.06 P99 ITL (ms): 17.78 ---------------Speculative Decoding--------------- Acceptance rate (%): 30.62 Acceptance length: 3.45 Drafts: 5243 Draft tokens: 41944 Accepted tokens: 12844 Per-position acceptance (%): Position 0: 66.83 Position 1: 49.11 Position 2: 36.62 Position 3: 27.90 Position 4: 22.09 Position 5: 17.17 Position 6: 13.98 Position 7: 11.27 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 208.46 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.61 Output token throughput (tok/s): 1381.58 Peak output token throughput (tok/s): 636.00 Peak concurrent requests: 24.00 Total token throughput (tok/s): 2473.16 ---------------Time to First Token---------------- Mean TTFT (ms): 205.39 Median TTFT (ms): 192.83 P99 TTFT (ms): 367.00 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.87 Median TPOT (ms): 10.59 P99 TPOT (ms): 36.67 ---------------Inter-token Latency---------------- Mean ITL (ms): 37.74 Median ITL (ms): 22.07 P99 ITL (ms): 155.20 ---------------Speculative Decoding--------------- Acceptance rate (%): 31.23 Acceptance length: 3.50 Drafts: 82664 Draft tokens: 661312 Accepted tokens: 206538 Per-position acceptance (%): Position 0: 70.86 Position 1: 49.80 Position 2: 36.20 Position 3: 27.68 Position 4: 21.82 Position 5: 17.57 Position 6: 14.30 Position 7: 11.62 ================================================== ``` PR: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 48.40 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 1.24 Output token throughput (tok/s): 371.88 Peak output token throughput (tok/s): 122.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 710.91 ---------------Time to First Token---------------- Mean TTFT (ms): 95.85 Median TTFT (ms): 117.33 P99 TTFT (ms): 126.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.38 Median TPOT (ms): 2.19 P99 TPOT (ms): 7.99 ---------------Inter-token Latency---------------- Mean ITL (ms): 8.23 Median ITL (ms): 8.26 P99 ITL (ms): 9.22 ---------------Speculative Decoding--------------- Acceptance rate (%): 31.16 Acceptance length: 3.49 Drafts: 5181 Draft tokens: 41448 Accepted tokens: 12916 Per-position acceptance (%): Position 0: 68.06 Position 1: 49.53 Position 2: 36.98 Position 3: 28.37 Position 4: 22.33 Position 5: 17.76 Position 6: 14.32 Position 7: 11.95 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 175.64 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 5.47 Output token throughput (tok/s): 1639.68 Peak output token throughput (tok/s): 652.00 Peak concurrent requests: 26.00 Total token throughput (tok/s): 2935.18 ---------------Time to First Token---------------- Mean TTFT (ms): 154.56 Median TTFT (ms): 153.48 P99 TTFT (ms): 299.94 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.23 Median TPOT (ms): 8.79 P99 TPOT (ms): 30.92 ---------------Inter-token Latency---------------- Mean ITL (ms): 31.33 Median ITL (ms): 21.56 P99 ITL (ms): 128.12 ---------------Speculative Decoding--------------- Acceptance rate (%): 30.26 Acceptance length: 3.42 Drafts: 84594 Draft tokens: 676752 Accepted tokens: 204774 Per-position acceptance (%): Position 0: 69.47 Position 1: 48.27 Position 2: 34.92 Position 3: 26.63 Position 4: 21.04 Position 5: 16.87 Position 6: 13.70 Position 7: 11.16 ================================================== ``` e2e output throughput improves 102% at batch size 1, improves 18% at batch size 16. * **Eagle** ``` vllm serve Qwen/Qwen3.5-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"jiapingW/Qwen3.5-35B-A3B-Eagle3-Specforge\",\"method\":\"eagle3\",\"num_speculative_tokens\":4}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.5-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 146.45 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.41 Output token throughput (tok/s): 122.91 Peak output token throughput (tok/s): 48.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 234.96 ---------------Time to First Token---------------- Mean TTFT (ms): 115.48 Median TTFT (ms): 137.76 P99 TTFT (ms): 150.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.78 Median TPOT (ms): 7.51 P99 TPOT (ms): 14.46 ---------------Inter-token Latency---------------- Mean ITL (ms): 21.12 Median ITL (ms): 21.16 P99 ITL (ms): 21.72 ---------------Speculative Decoding--------------- Acceptance rate (%): 43.24 Acceptance length: 2.73 Drafts: 6605 Draft tokens: 26420 Accepted tokens: 11424 Per-position acceptance (%): Position 0: 67.51 Position 1: 46.98 Position 2: 33.72 Position 3: 24.75 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 305.64 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 3.14 Output token throughput (tok/s): 942.27 Peak output token throughput (tok/s): 672.00 Peak concurrent requests: 23.00 Total token throughput (tok/s): 1686.76 ---------------Time to First Token---------------- Mean TTFT (ms): 201.53 Median TTFT (ms): 202.58 P99 TTFT (ms): 345.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.27 Median TPOT (ms): 15.30 P99 TPOT (ms): 29.23 ---------------Inter-token Latency---------------- Mean ITL (ms): 34.26 Median ITL (ms): 24.06 P99 ITL (ms): 157.68 ---------------Speculative Decoding--------------- Acceptance rate (%): 27.84 Acceptance length: 2.11 Drafts: 136316 Draft tokens: 545264 Accepted tokens: 151791 Per-position acceptance (%): Position 0: 50.02 Position 1: 29.58 Position 2: 18.92 Position 3: 12.84 ================================================== ``` PR: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 54.89 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 1.09 Output token throughput (tok/s): 327.94 Peak output token throughput (tok/s): 130.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 626.91 ---------------Time to First Token---------------- Mean TTFT (ms): 88.68 Median TTFT (ms): 109.96 P99 TTFT (ms): 123.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.76 Median TPOT (ms): 2.75 P99 TPOT (ms): 4.20 ---------------Inter-token Latency---------------- Mean ITL (ms): 7.63 Median ITL (ms): 7.65 P99 ITL (ms): 8.03 ---------------Speculative Decoding--------------- Acceptance rate (%): 44.37 Acceptance length: 2.77 Drafts: 6495 Draft tokens: 25980 Accepted tokens: 11528 Per-position acceptance (%): Position 0: 69.75 Position 1: 48.21 Position 2: 34.26 Position 3: 25.28 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 188.81 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 5.08 Output token throughput (tok/s): 1525.38 Peak output token throughput (tok/s): 763.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2730.56 ---------------Time to First Token---------------- Mean TTFT (ms): 139.44 Median TTFT (ms): 142.08 P99 TTFT (ms): 295.77 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.02 Median TPOT (ms): 9.90 P99 TPOT (ms): 18.75 ---------------Inter-token Latency---------------- Mean ITL (ms): 26.48 Median ITL (ms): 18.78 P99 ITL (ms): 119.41 ---------------Speculative Decoding--------------- Acceptance rate (%): 41.36 Acceptance length: 2.65 Drafts: 108609 Draft tokens: 434436 Accepted tokens: 179686 Per-position acceptance (%): Position 0: 66.34 Position 1: 45.16 Position 2: 31.29 Position 3: 22.65 ================================================== ``` e2e output throughput improves 166% at batch size 1, improves 61% at batch size 16. ## Accuracy Testing * **DFlash** ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.6-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3472|± |0.0131| | | |strict-match | 5|exact_match|↑ |0.3268|± |0.0129| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3594|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3389|± |0.0130| ``` * **Eagle** ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.5-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.0121|± | 0.003| | | |strict-match | 5|exact_match|↑ |0.0000|± | 0.000| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8605|± |0.0095| | | |strict-match | 5|exact_match|↑ |0.8491|± |0.0099| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51742", "createdAt": "2026-08-11T00:16:11Z", "updatedAt": "2026-08-12T22:39:41Z", "timestamp": "2026-08-12T22:39:41Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "speculative-decoding" ], "author": "xyang16", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51746", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add explicit model info cache preparation", "text": "## Purpose PR #23558 by @manoelmarques added the source-hash-validated runtime ModelInfo cache and reported model-class inspection at 10.1034 seconds uncached versus 0.6127 seconds cached, n=1 per state. During review, @hmellor objected to mirroring common runtime requirements into the build. The merged design therefore populated the cache at runtime. Without a matching entry, first activation still pays the subprocess inspection. This PR adds an explicit preparation command for the existing cache format: ```bash vllm prepare-model-info Qwen3ForCausalLM ``` The command runs in the final installed environment and ensures a readable entry exists under `VLLM_CACHE_ROOT/modelinfos`. It does not download model weights. It accepts only an unchanged built-in lazy registration, then verifies that the entry can be read after it is written. Unknown, plugin, eager, and overridden registrations fail. Normal startup behavior is unchanged: missing or stale entries still use the existing subprocess inspection and safe replacement path, and ordinary cache writes remain best-effort. This deliberately does not generate entries during a generic wheel or source build. Model inspection can depend on runtime packages, platform modules, and compiled extensions, so the entry must be prepared in the target runtime or final deployment image. The tradeoff is time shifting, not free work. In the hardware gate, the preparation command took 9.7860 seconds, n=1, and produced an 870-byte entry. It is useful when that prepared cache is baked or distributed to matching fresh deployments with the same installed environment, or when readiness latency matters more than image preparation time. A writable deployment that starts once will populate the same cache on its own. ## Test Plan Focused registry and public CLI coverage: ```bash pytest -q \\ tests/entrypoints/unit_tests/test_launch_cli.py \\ tests/models/test_registry.py::test_prepare_model_info_inspects_builtin_lazy_registration \\ tests/models/test_registry.py::test_lazy_prepare_model_info_requires_readable_cache \\ tests/models/test_registry.py::test_inspect_model_info_keeps_best_effort_cache_write \\ tests/models/test_registry.py::test_lazy_prepare_model_info_accepts_readable_cache \\ tests/models/test_registry.py::test_prepare_model_info_rejects_unsupported_registration ``` All applicable changed-file pre-commit hooks were also run from exact base `b2506d62aec7e6bccc5959b829221a7ae217abf3` to exact head `13f5279875da8dd5e90c9a5a3f779fe839689e54`. The end-to-end gate used Qwen3-0.6B FP16 TP1 on one RTX 4090 Laptop GPU, maximum model length 256, warm model pages, and identical non-ModelInfo generated state. The interleaved sequence was empty, valid, valid, empty, empty, valid. ## Test Result Focused public CLI and registry coverage passed 20 tests. All applicable changed-file pre-commit hooks passed, including Ruff, formatting, markdownlint, Python 3.10 mypy, SPDX, lazy-import, forbidden-import, and configuration checks. | Starting state | Process launch to HTTP ready | Process launch to first correct token | | --- | ---: | ---: | | Empty ModelInfo cache | 20.2773 s median | 20.4360 s median | | Valid prepared entry | 16.0033 s median | 16.1558 s median | | Reduction | 4.2739 s, 21.1% | 4.2802 s, 20.9% | Each arm used n=3. All 18 exact-token requests passed, the deliberately wrong-token control failed, and all six cells passed GPU engagement, cleanup, source-origin, page-cache, and generated-state controls. This hardware result used exact source `8a90f1ab7565b6b6ad824b5643c203bb40bddcac` on base `8bcc916a98a90822882455ab30aba02c409da2a6`. The same feature patch was then rebased without conflicts onto base `b2506d62aec7e6bccc5959b829221a7ae217abf3` at current head `13f5279875da8dd5e90c9a5a3f779fe839689e54`; its stable patch ID is unchanged, and the intervening upstream commits overlap none of the six feature files. It is a Qwen3-0.6B installed-runtime integration result, not an exact-current-head, dense-model, cross-platform, or wheel-build performance claim. The measured source used a recorded compatible precompiled runtime payload. AI assistance was used. I reviewed the changes and can defend the implementation, constraints, and measurements.", "url": "https://github.com/vllm-project/vllm/pull/51746", "createdAt": "2026-08-11T01:35:33Z", "updatedAt": "2026-08-13T17:45:30Z", "timestamp": "2026-08-13T17:45:30Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "new-model", "frontend" ], "author": "matteso1", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51757", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Fix] Support heterogeneous models (Gemma 4 / Qwen) failing with AmbiguousGlobalPerLayerAttributeError", "text": "### What this PR does This PR fixes severe crashes that occur when loading heterogeneous models (such as **Gemma 4** and **Qwen MTP**) with recent versions of `transformers` (>=5.15.0). These models possess varying `head_dim` and `num_key_value_heads` across layers. The new `transformers` library raises an `AmbiguousGlobalPerLayerAttributeError` (which does **not** inherit from `AttributeError`) when attempting to access per-layer attributes on the global config object. ### The Fix This PR addresses all 3 crash points during model initialization and weight loading: 1. **Config Initialization**: Adds a `_safe_getattr()` helper to `ModelArchConfigConvertorBase` that catches the `AmbiguousGlobalPerLayerAttributeError` exception when `get_head_size()` and others are evaluated globally, falling back safely to `per_layer_config[0]`. 2. **Layer Initialization**: `gemma4_mtp.py` now consults `config.per_layer_config[layer_idx]` during `__init__` to instantiate attention modules with the correct dimensional parameters rather than the potentially incorrect global defaults. 3. **Parameter Weight Loading**: Introduces `_safe_narrow()` into `BasevLLMParameter` (with a fast-path for homogeneous layouts) and adds 1D clipping fallback in `weight_utils.py` to prevent tensor slicing bounds overflow when merging parameters of disparate layer topologies. ### Validation Tested extensively in a clustered environment on H200 and A100 setups provisioning Gemma 4 31B and Qwen MTP models. *(Reference implementations and our runtime patcher used to validate this in production can be found at [quivent/vllm-heterogeneous](https://github.com/quivent/vllm-heterogeneous))* ### Checkboxes - [x] Tested locally - [x] Does not break standard/homogeneous models", "url": "https://github.com/vllm-project/vllm/pull/51757", "createdAt": "2026-08-11T03:33:25Z", "updatedAt": "2026-08-13T12:12:04Z", "timestamp": "2026-08-13T12:12:04Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen" ], "author": "quivent", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51767", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[FS-Offloading] : Simplify fs manager", "text": "## Purpose The tracking of failed keys on `main` is handled in the fs/manager. This makes the manager more complicated than necessary. This PR makes failed_keys a first class citizen in the DualQueueThreadPool. The DualQueueThreadPool already maintains a JobState to track a Job's progress and reasons about failure/success based on results directly from the threadpool. With this change, the manager simply relies on the finished job results from the DualQueueThreadPool. This PR also adds batching infrasturcture to DualQueueThreadPool, but exercises it naively (same policy as main), i.e. all keys from a single request are processed by a single thread. ## Test Plan `pytest -s tests/v1/kv_offload/tiering/test_fs_tier.py` vllm serve command: ``` ++ vllm serve openai/gpt-oss-120b --tensor-parallel-size=2 --kv-transfer-config '{ \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"blocks_per_chunk\" : 1, \"spec_name\": \"TieringOffloadingSpec\", \"cpu_bytes_to_use\": 150323855360, \"eviction_policy\": \"lru\", \"secondary_tiers\": [{ \"type\": \"fs\", \"root_dir\": \"/mnt/nvme-storage/\", \"n_read_threads\": 16, \"n_write_threads\": 16 }] ,\"enable_cross_layers_blocks\" : \"True\" } }' --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --port 8000 ``` lm eval command: ``` TARGET_URL=\"http://127.0.0.1:8000\" MODEL=\"openai/gpt-oss-120b\" LM_EVAL_NUM_CONCURRENT=1000 LM_EVAL_TASKS=\"gsm8k\" lm_eval \\ --model local-completions \\ --model_args \"base_url=${TARGET_URL}/v1/completions,model=${MODEL},tokenized_requests=False,num_concurrent=${LM_EVAL_NUM_CONCURRENT},trust_remote_code=True\" \\ --tasks ${LM_EVAL_TASKS} \\ --seed 42 \\ --num_fewshot 25 \\ --gen_kwargs temperature=0.0 ``` ## Test Result Unit tests pass lm eval outputs : `Run 1 : cold cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6247|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4223|± |0.0136| ``` `Run 2 : warm cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6270|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4223|± |0.0136| ``` `Run 3 : warm cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6156|± |0.0134| | | |strict-match | 25|exact_match|↑ |0.4155|± |0.0136| ``` ## Performance `guidellm command` ``` BENCH_RATE=\"64\" BENCH_RATE_TYPE=\"concurrent\" BENCH_MAX_SECONDS=\"700\" BENCH_RANDOM_SEED=\"889\" BENCH_TURNS=5 BENCH_PROMPT_TOKENS=\"4096\" BENCH_OUTPUT_TOKENS=\"512\" BENCH_PREFIX_TOKENS=\"10000\" PREFIX_COUNT=$((4 * BENCH_RATE)) #PREFIX_COUNT=$BENCH_RATE DATA=\"{\\\"kind\\\":\\\"synthetic_text\\\",\\\"prompt_tokens\\\":${BENCH_PROMPT_TOKENS},\\\"output_tokens\\\":${BENCH_OUTPUT_TOKENS},\\\"turns\\\":${BENCH_TURNS},\\\"prefix_buckets\\\": [{\\\"bucket_weight\\\": 100, \\\"prefix_count\\\": ${PREFIX_COUNT}, \\\"prefix_tokens\\\": ${BENCH_PREFIX_TOKENS}}]}\" guidellm run \\ --backend \"kind=openai_http,target=http://127.0.0.1:8000,request_format=/v1/completions\" \\ --profile \"kind=concurrent,streams=${BENCH_RATE}\" \\ --constraint \"kind=max_duration,seconds=${BENCH_MAX_SECONDS}\" \\ --seed \"kind=static,value=${BENCH_RANDOM_SEED}\" \\ --data \"$DATA\" ``` `main commit 419b51b38517bd446c6e` ``` ℹ Server Throughput Statistics (All Requests) |============|=======|======|=========|==============|===============|==============| | Benchmark | Requests ||| Input Tokens | Output Tokens | Total Tokens | | Strategy | Concurrency || Per Sec | Per Sec | Per Sec | Per Sec | | | Mdn | Mean | Mean |||| |------------|-------|------|---------|--------------|---------------|--------------| | concurrent | 64.0 | 63.7 | 3.2 | 75537.7 | 1661.7 | 77083.7 | |============|=======|======|=========|==============|===============|==============| ``` `PR` ``` ℹ Server Throughput Statistics (All Requests) |============|=======|======|=========|==============|===============|==============| | Benchmark | Requests ||| Input Tokens | Output Tokens | Total Tokens | | Strategy | Concurrency || Per Sec | Per Sec | Per Sec | Per Sec | | | Mdn | Mean | Mean |||| |------------|-------|------|---------|--------------|---------------|--------------| | concurrent | 64.0 | 63.7 | 3.2 | 75408.5 | 1647.8 | 77056.2 | |============|=======|======|=========|==============|===============|==============| ``` No regression with PR.", "url": "https://github.com/vllm-project/vllm/pull/51767", "createdAt": "2026-08-11T06:14:33Z", "updatedAt": "2026-08-13T02:34:39Z", "timestamp": "2026-08-13T02:34:39Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "varun-sundar-rabindranath", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51772", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][MLA] Fuse Kimi-K3 chunked-context K/V packing", "text": "## Purpose The K3 MLA layer delegated chunked-context prefill to `impl._compute_prefill_context`, which per chunk casts `kv_nope` to fp8, casts `k_pe`, concatenates `[k_nope | k_pe]`, and re-quantizes a query the fused new-token epilogue had already quantized. This gives the layer its own context loop so that tail collapses into one kernel per chunk: `fused_kimi_k3_mla_kv_concat{,_quant_fp8}` reads the strided `kv_b_proj` output and the gathered `k_pe` in place -- the latter still in its fp8 cache layout -- and writes a contiguous key plus, on the fp8 path, a contiguous fp8 V. Only the gather and `kv_b_proj` remain. Casts to E4M3 use the native pairwise converters, so the result is bit-identical to `.to(torch.float8_e4m3fn)`. It also wires an `out` tensor through `run_prefill_context_chunk`, reusing the existing `supports_out()` capability. A backend honoring it writes each chunk's partial straight into the accumulating context partial, removing the per-chunk output copy for every non-continuation chunk; a continuation still has to merge with the partial already in place, so it keeps its own buffer. Decode context parallelism is unchanged and still uses `impl._context_parallel_compute_prefill_context`. The second commit fixes a bug in the first, found by serving a stock checkpoint before posting: the fused loop originally replaced the impl's per-chunk cast of the gathered latent with a load-time check that `kv_b_proj` consumes fp8 directly. Cache dtype and `kv_b_proj` dtype are independent -- a plain fp8 cache leaves the gather in fp8, but only an fp8-quantized `kv_b_proj` can take that, and a stock Kimi-K3 checkpoint's is bf16. Every rank died at weight load under `--kv-cache-dtype fp8`: ``` AssertionError: Kimi-K3 with a plain fp8 KV cache needs a kv_b_proj that consumes the fp8 gathered latent directly; this checkpoint's kv_b_proj wants torch.bfloat16. ``` The cast is restored in `run_chunk` and the check dropped. It is a no-op `.to` when `kv_b_proj` already consumes the fp8 latent, so the fused path keeps its win; the fusion itself is untouched. ## Not a duplicate Per the contribution policy, checked before opening: ``` gh pr list --repo vllm-project/vllm --state open --search \"Kimi-K3 chunked context MLA\" gh pr list --repo vllm-project/vllm --state open --search \"kv_b_proj\" gh pr list --repo vllm-project/vllm --state open --search \"fuse chunked context K/V packing\" gh pr list --repo vllm-project/vllm --state all --head agent/kimi-k3-fused-context-kv-pack ``` No open PR touches the K3 fused chunked-context prefill path. The nearest neighbours are different work: #37245 fixes an int8 dtype cast for `kv_c_normed` in the *generic* MLA prefill, and #48051 stops re-quantizing a bf16 MLA `kv_b_proj` on ROCm for GLM MoE DSA. Neither touches `vllm/models/kimi_k3/nvidia/mla.py` or the fused pack kernels. ## Test Plan ``` pytest tests/models/kimi_k3/test_mla_prefill_context.py \\ tests/v1/attention/test_mla_prefill_registry.py \\ tests/v1/attention/test_mla_context_chunks.py \\ tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py \\ tests/kernels/attention/test_kimi_k3_mla_key_concat_kv_cache.py ``` `tests/models/kimi_k3/test_mla_prefill_context.py` is new here. It asserts the fused loop hands the prefill backend the same `(q, k, v)` per chunk, and returns the same merged partial, as the generic impl -- across bf16/fp8 caches, both `kv_b_proj` weight dtypes, and both `supports_out()` modes. ## Test Result **67 passed** (B300, aarch64, CUDA 13). The two `bf16_kv_b_proj` + `fp8` cache cases fail on the first commit alone and pass with the second: ``` FAILED ...test_fused_context_matches_generic_impl[bf16_kv_b_proj-fp8-copy_out] FAILED ...test_fused_context_matches_generic_impl[bf16_kv_b_proj-fp8-writes_out] 2 failed, 7 passed ``` `pre-commit run --files vllm/models/kimi_k3/nvidia/mla.py tests/models/kimi_k3/test_mla_prefill_context.py` passes (ruff check/format, mypy, SPDX, forbidden-imports). ## Model Evaluation Kimi-K3 served end to end in 2P1D PD disaggregation on 8 x 4 B300 -- prefill 2 x TEP8 (`--enforce-eager`, mns 16), decode DEP16/TP1 (mns 12, hybrid LB), NIXL connector, EAGLE3-3 speculative decoding with `rejection_sample_method` at its default `standard`, `--kv-cache-dtype fp8`. That is exactly the configuration that could not load before the second commit. | Eval | Score | Samples | |---|---|---| | GSM8K, 5-shot, `/v1/completions`, temp 0.6 / top_p 0.95 | **0.9575** flexible-extract / **0.9575** strict-match | 1319, 0 request errors | | OCRBench, thinking at effort high, temp 1.0 / top_p 1.0 | **0.892** +- 0.010 | 1000 | I do not have a pre-change baseline for the same configuration, because a stock checkpoint with an fp8 KV cache could not load at all on the first commit alone. The fp8 packing itself is verified exact against torch's cast in the kernel tests, and the fused loop is verified equal to the generic impl chunk-for-chunk in the new test. ## Note AI assistance was used for the second commit (the `kv_b_proj` cast fix, its test, and this description) and for running the evaluation above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51772", "createdAt": "2026-08-11T06:28:19Z", "updatedAt": "2026-08-13T05:42:58Z", "timestamp": "2026-08-13T05:42:58Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "ready", "nvidia", "kimi", "k3" ], "author": "zyongye", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51777", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docker] Update to nixl-1.3.2", "text": "## Purpose The KV-connectors layer installs `nixl` from requirements/kv_connectors.txt, then force-reinstalls the CUDA-matched backend wheel so the correct nixl_ep_cpp.so is present. That second install passes `--no-deps` and no version, which bypasses the `nixl` meta package's `nixl-cu*==` constraint and resolves whatever nixl-cu${CUDA_MAJOR} is newest on PyPI. The image therefore silently ships a different NIXL than it pins: with `nixl == 1.3.1` in kv_connectors.txt, the CUDA 13 image ends up with `nixl 1.3.1` / `nixl-cu12 1.3.1` / `nixl-cu13 1.3.2`. Because the `nixl` meta package dispatches on torch.version.cuda and re-exports nixl_cu13's bindings, 1.3.2 is what actually loads while the pin reads 1.3.1. 1.3.2 fixes a telemetry-gating bug in 1.3.1 where an agent created with telemetry disabled leaves getXferTelemetry returning NIXL_ERR_NO_TELEMETRY, which breaks callers that request transfer telemetry. [Documented in this Dynamo issue](https://github.com/ai-dynamo/dynamo/issues/12952) This PR attempts to fix the silently drifting nixl backend package version, and capture fix for above issue by moving to latest nixl version. ## Fix - ~~Pin the version in the command to update nixl-cu${CUDA_MAJOR}~~ (changed to below after discussion with stakeholders) - Remove the redundant nixl-cu reinstallation. - Bump the pin to nixl 1.3.2, the version CUDA 13 images have been shipping in practice. - Bump the nixl version used by XPU build as well to fix the test that showed this issue. ## Test Plan - Run the test in https://github.com/ai-dynamo/dynamo/issues/12952 ## Test Result Image built with this PR: ``` ============ Serving Benchmark Result ============ Successful requests: 100 Failed requests: 0 Maximum request concurrency: 8 Benchmark duration (s): 55.70 Total input tokens: 102399 Total generated tokens: 25600 Request throughput (req/s): 1.80 Output token throughput (tok/s): 459.61 Peak output token throughput (tok/s): 594.00 Peak concurrent requests: 16.00 Total token throughput (tok/s): 2298.01 ``` Image built without this PR: ``` ### Benchmark result: Traceback (most recent call last): File \"/usr/local/bin/vllm\", line 10, in sys.exit(main()) ^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py\", line 95, in main args.dispatch_function(args) ...... File \"/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py\", line 2095, in main_async benchmark_result = await benchmark( ^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py\", line 865, in benchmark raise ValueError( ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Never received a valid chunk to calculate TTFT.This response will be marked as failed! ### Error in Decode upon first request E0730 15:51:33.763552 436 nixl_agent.cpp:1291] getXferTelemetry: cannot return values when telemetry is not enabled. (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] NIXL transfer failure: transfer_exception. Marking blocks as invalid | Context: {'failure_type': 'transfer_exception', 'request_id': 'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd', 'engine_id': '4c7d2274-6256-41df-82c2-adad8eb2c5b4', 'remote_engine_id': '85d325a3-20c1-469a-a18c-ca82378bac38', 'remote_request_id': 'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-8b2a6ba3', 'remote_host': '192.168.200.198', 'remote_port': 20098, 'num_local_blocks': 16, 'num_remote_blocks': 16, 'local_block_ids_sample': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]} (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] Traceback (most recent call last): (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] File \"/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py\", line 2108, in _pop_done_transfers (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] res = self.nixl_wrapper.get_xfer_telemetry(handle) (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] File \"/opt/venv/lib/python3.12/site-packages/nixl_cu12/_api.py\", line 670, in get_xfer_telemetry (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] return self.agent.getXferTelemetry(handle._handle) (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] nixl_cu12._bindings.nixlNoTelemetryError: NIXL_ERR_NO_TELEMETRY (EngineCore pid=436) WARNING 07-30 15:51:33 [base_worker.py:1989] Skipping KV post-processing for failed request ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd (EngineCore pid=436) ERROR 07-30 15:51:33 [scheduler.py:2804] Failing 1 request(s) due to KV load failure (failure_policy=fail, 1024 tokens affected). Request IDs: {'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd'} ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51777", "createdAt": "2026-08-11T07:17:48Z", "updatedAt": "2026-08-13T06:26:40Z", "timestamp": "2026-08-13T06:26:40Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "intel-gpu", "ci/build", "kv-connector", "nvidia" ], "author": "sandeep-maddipatla", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51778", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add prediction token usage details", "text": "## Summary - expose per-request speculative decoding acceptance and rejection counts through engine outputs - populate `completion_tokens_details.accepted_prediction_tokens` and `rejected_prediction_tokens` for streaming and non-streaming Chat Completions and Completions responses - preserve per-choice accounting for parallel sampling and add focused scheduler, output processor, and serving tests This PR intentionally does not implement `reasoning_tokens`. Existing PRs, including #35037, already cover that work. `CompletionTokenUsageInfo` includes the field for protocol completeness, but this change leaves it unset. ## Why this is not a duplicate I checked issue #50895 and searched open PRs by the issue number and by `accepted_prediction_tokens rejected_prediction_tokens`. No open PR implements these two fields. The existing related PRs focus on `reasoning_tokens`, while this PR handles the speculative-decoding fields listed as untracked in #50895. ## Tests Passed: ```text .venv/bin/ruff check --ignore ISC004 .venv/bin/ruff format --check .venv/bin/python -m compileall -q git diff --check pre-commit hooks run by `git commit` ``` Focused pytest execution: ```text .venv/bin/python -m pytest \\ tests/v1/core/test_scheduler.py::test_stop_via_update_from_output \\ tests/v1/engine/test_output_processor.py::test_speculative_token_counts_accumulate_per_request \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_prediction_token_details \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_omits_prediction_details_without_spec_decode \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_sums_prediction_details_across_choices \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_streaming_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_streaming_prediction_token_details -q ``` Result: `8 passed, 15 warnings in 2.89s`. ## Model evaluation Not applicable. This change reports existing speculative-decoding accounting and does not affect model output, accuracy, or serving decisions. ## AI assistance AI assistance was used to implement this change. I have reviewed every changed line and understand the implementation end to end. Fixes #50895", "url": "https://github.com/vllm-project/vllm/pull/51778", "createdAt": "2026-08-11T07:22:23Z", "updatedAt": "2026-08-13T16:56:42Z", "timestamp": "2026-08-13T16:56:42Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend", "needs-rebase" ], "author": "luyixiao95", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51784", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix][EPLB] Log correct avg/max token counts and balancedness", "text": "## Purpose Here is a fix for a small but confusing logging bug that I ran into: the EPLB balancedness log line reduced `avg_tokens`/`max_tokens` over the layer axis instead of the rank axis, so logged values were wrong whenever `num_moe_layers != num_ranks` leading to wrong conclusions about expert balance. Now both reduce the trailing rank axis. Log-only, actual rebalancing unaffected. AI-assisted, every line reviewed and tested by the submitter. ## Test Plan `pytest tests/distributed/test_eplb_state.py -v` (new CPU regression test, registered in the EPLB CI lane incl. AMD mirror). I added the test to pin down the problem. Happy to remove it if you don't see any value in it. ## Test Result Fails on `main` (`balancedness=0.517` vs correct `0.876`), passes with fix. No model evals: log-only change.", "url": "https://github.com/vllm-project/vllm/pull/51784", "createdAt": "2026-08-11T08:27:15Z", "updatedAt": "2026-08-13T16:02:29Z", "timestamp": "2026-08-13T16:02:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "fjosw", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Remove dead `QuantizationConfig.is_mxfp4_quant`", "text": "## Purpose This is dead code following https://github.com/vllm-project/vllm/pull/37128. This was originally added in https://github.com/vllm-project/vllm/pull/29008 that supported padding for gpt-oss / certain MXFP4 backends, see: https://github.com/xuebwang-amd/vllm/blob/c62f664e97977ee54ab1d1c77604ebb45081bc06/vllm/model_executor/layers/fused_moe/layer.py#L259-L277 This is now handled in: https://github.com/vllm-project/vllm/blob/490259c1f63faf025b8050504db63d81c817d781/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L640 ## Test Plan N/A ## Test Result N/A", "url": "https://github.com/vllm-project/vllm/pull/51793", "createdAt": "2026-08-11T09:46:17Z", "updatedAt": "2026-08-13T17:58:31Z", "timestamp": "2026-08-13T17:58:31Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "quantization" ], "author": "fxmarty-amd", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51794", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4", "text": "## Purpose > Part of https://github.com/vllm-project/vllm/issues/41820. > Continue working on https://github.com/vllm-project/vllm/pull/43718. > Based on https://github.com/vllm-project/vllm/pull/50866. > Added https://github.com/vllm-project/vllm/pull/52093 and https://github.com/vllm-project/vllm/pull/52094. To be completed... ## Test Plan - Benchmark with SA InferenceX `8k1k` and `agentic` workload. - Acc test with `gsm8k` dataset. ## Test Result ### Benchmark (workload=8k1k dp8+ep) | Concurrency | Metric | Disable CSA multi-stream | Enable CSA multi-stream | Comparison | | :---------: | :------------------------------ | :----------------------- | :---------------------- | :--------- | | 1 | Output token throughput (tok/s) | 29.67 | 23.71 | -20.09% ↓ | | 1 | Mean TTFT (ms) | 1457.20 | 1451.69 | -0.38% ↓ | | 1 | Mean TPOT (ms) | 32.18 | 40.67 | +26.38% ↑ | | 8 | Output token throughput (tok/s) | 168.20 | 146.34 | -13.00% ↓ | | 8 | Mean TTFT (ms) | 1768.56 | 1708.51 | -3.40% ↓ | | 8 | Mean TPOT (ms) | 44.85 | 51.90 | +15.72% ↑ | | 64 | Output token throughput (tok/s) | 477.18 | 451.11 | -5.46% ↓ | | 64 | Mean TTFT (ms) | 2966.52 | 3052.56 | +2.90% ↑ | | 64 | Mean TPOT (ms) | 129.00 | 136.34 | +5.69% ↑ | ### Benchmark (workload=agentx) To be completed... ### Acc Test To be completed... ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51794", "createdAt": "2026-08-11T09:48:40Z", "updatedAt": "2026-08-13T13:52:16Z", "timestamp": "2026-08-13T13:52:16Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm", "deepseek", "nvidia" ], "author": "shen-shanshan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51795", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Reject negative token ids as out-of-vocabulary", "text": "## Purpose `InputProcessor._validate_model_input` validates that caller-supplied token ids are within the vocabulary, but it only checks the upper bound, so a negative token id passes validation. A token id is used as an index downstream, and a negative index is never valid input. This PR adds a symmetric lower-bound check beside the existing upper-bound check. A negative id is now rejected with the same \"out of vocabulary\" error (HTTP 400). The check sits on the shared input-validation path, so it covers generate, embedding, and pooling requests. ## Test Plan ```bash pytest -q tests/entrypoints/openai/completion/test_prompt_validation.py::test_negative_token_ids ``` ## Test Result The new test passes: a `/v1/completions` request with `prompt=[-1]` returns HTTP 400 with an \"out of vocabulary\" message (previously it passed validation). The existing `test_out_of_vocab_token_ids` (upper-bound) continues to pass. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51795", "createdAt": "2026-08-11T09:48:47Z", "updatedAt": "2026-08-13T04:34:56Z", "timestamp": "2026-08-13T04:34:56Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "bug", "ready" ], "author": "ECMGit", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51797", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Match Qwen3.5 GDN BF16 semantics", "text": "## Summary - preserve activation-dtype Q/K normalization and beta semantics for Qwen3.5 BF16 activations - cover fused prefill, generic/speculative decode, and packed decode with the same model-scoped gate - propagate the gate through Triton warmup so compiled variants match runtime selection - add sensitivity tests for prefill and both incremental-decode paths Fixes #51779. ## Root cause The Qwen3.5 GDN CUDA kernels widened BF16 Q/K and beta inputs to FP32 before normalization or sigmoid and only rounded when storing the final result. The official Transformers formula returns BF16-rounded intermediate tensors, so the fused kernels implemented a different numerical function. This change introduces explicit BF16 rounding boundaries around the square, reduction result, epsilon addition, inverse square root, final Q/K product, and beta sigmoid. The gate is enabled only by Qwen3.5 and only for BF16 inputs; existing behavior remains the default for other callers and FP32 inputs. ## Standalone reproducer The reproducer is intentionally **not included in this PR's repository diff**. It is available as a separate downloadable script: - [View or download `reproduce_github_issue_51779.py`](https://gist.github.com/mrjoema/18b0005b93e6fdeaad8f52be723b289d) ```bash curl -L \\ https://gist.githubusercontent.com/mrjoema/18b0005b93e6fdeaad8f52be723b289d/raw/042da542f90f32c8746c4083354358426b7a4517/reproduce_github_issue_51779.py \\ -o /tmp/reproduce_github_issue_51779.py # On upstream main: .venv/bin/python /tmp/reproduce_github_issue_51779.py --expect buggy # On this PR branch: .venv/bin/python /tmp/reproduce_github_issue_51779.py --expect fixed ``` The fixed expectation requires exact zero mismatch for Q, K, and beta against the Transformers BF16 formula. ## Validation Passed locally: - repository pre-commit hooks on all nine changed files - Ruff lint and formatting checks - Python bytecode compilation for all changed Python files - `git diff --check` Passed on an NVIDIA A100-SXM4-40GB: ```console $ .venv/bin/python scripts/reproduce_github_issue_51779.py --expect fixed GPU: NVIDIA A100-SXM4-40GB seed=51779, tokens=64, dtype=torch.bfloat16 relative L2 error exact mismatch rate BF16 reference FP32 promoted BF16 ref FP32 promoted q: 0.000000e+00 2.915142e-03 0.00% 27.87% k: 0.000000e+00 2.925246e-03 0.00% 28.09% beta: 0.000000e+00 1.559851e-03 0.00% 100.00% FIXED: all outputs exactly match BF16 Transformers semantics. ``` ## Duplicate-work check No open PR references #51779. PR #49953 is an unrelated ROCm/AITER long-prefill split-QKV optimization and does not address Qwen3.5 CUDA BF16 semantics. ## AI assistance AI assistance was used to develop this change. Before marking the PR ready, the human submitter will review every changed line and run the pending GPU and model evaluations.", "url": "https://github.com/vllm-project/vllm/pull/51797", "createdAt": "2026-08-11T10:01:04Z", "updatedAt": "2026-08-13T06:50:19Z", "timestamp": "2026-08-13T06:50:19Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen", "verified" ], "author": "mrjoema", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51809", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Enable Kimi K3 KDA kernel tests on XPU", "text": "## Purpose `gather_initial_states()` asserted `state.is_cuda`, which is False for XPU tensors, so it blew up before reaching the Triton kernel; widened to `state.is_cuda or state.is_xpu`. `test_kda.py` hardcoded `DEVICE = \"cuda\"`, now taken from `current_platform.device_type` - the CUDA-only cases already self-gate. This gives XPU coverage of the KDA Triton kernels: chunked prefill (`chunk_kda`, plus the fused gate/cumsum variant) and the fused-recurrent packed-decode and spec-decode paths. ## Test Plan ```bash python -m pytest tests/models/kimi_k3/test_kda.py -v ``` ## Test Result 45 passed, 6 skipped on Intel Arc Pro B70. Tested also on H200 - 51 passed", "url": "https://github.com/vllm-project/vllm/pull/51809", "createdAt": "2026-08-11T11:12:28Z", "updatedAt": "2026-08-13T12:16:56Z", "timestamp": "2026-08-13T12:16:56Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "intel-gpu", "ready", "kimi", "k3" ], "author": "pmanczak", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51813", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix and test EPLB balancedness calculation", "text": "## Purpose Fix EPLB balancedness logging to aggregate rank load within each MoE layer. The current reduction uses the layer axis, so it can report perfect balance when one EP rank receives all tokens in every layer. ## Test Plan ```bash pytest -q tests/distributed/test_eplb_algo.py ``` ## Test Result The new regression case has equal token totals per layer with all tokens routed to one rank. Before the fix, it produced `avg_tokens=100` and `max_tokens=100`; after the fix, it produces the expected `avg_tokens=50` and `max_tokens=200`. ```text 15 passed, 1 skipped ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] Documentation updates are not required for this metrics-only fix.
", "url": "https://github.com/vllm-project/vllm/pull/51813", "createdAt": "2026-08-11T12:03:34Z", "updatedAt": "2026-08-13T03:27:22Z", "timestamp": "2026-08-13T03:27:22Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "jdebache", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51821", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][CI] Restore the DeepSeek-V4 input GEMM override point", "text": "## Purpose The GSM8K accuracy job for `amd/DeepSeek-V4-Flash-NVFP4` on gfx950 reports 0.0000 against a threshold of 0.92. It is not a near miss: the server comes up clean and then answers every one of the questions with an unparsable run of repeated tokens, so the failure is in the numerics rather than in the harness. A bisect over the window in which the job turned red lands on 79c865b838e34f7a98a936771284773819d79c8f, \"[Perf] Narrow DeepSeek V4 eager CUDA graph region\" ([#51430](https://github.com/vllm-project/vllm/pull/51430)). Alongside narrowing the graph region, that commit deleted `DeepseekV4Attention._fused_wqa_wkv_gemm` and inlined `self.fused_wqa_wkv(hidden_states)[0]` at its only call site. The deleted method is the override point the ROCm attention layer implements. On ROCm the `fused_wqa_wkv` weight is preshuffled in place when the model loads and its block scale is kept to the side, so the only thing that can multiply it correctly afterwards is AITER's preshuffled-B GEMM. With the call inlined, the input projection of every attention layer runs the plain linear over a shuffled weight, and the model produces garbage from the first layer on. The same removal leaves the ROCm override's own fallback calling a `super()` method that no longer exists, so any path that took the fallback would raise `AttributeError`. Nothing about this is visible on CUDA, and the ROCm override still reads as if it were connected, which is why the removal went through. ## Relationship to the other #51430 follow-ups Two open PRs address #51430, both for a different symptom on NVIDIA, and neither covers this one. [#51750](https://github.com/vllm-project/vllm/pull/51750) is an auto-generated full revert opened after the B200 nightly saw GSM8K fall to 0.0432 with MTP acceptance collapsing to 4.1%; it would undo this bug as a side effect, but it is still a draft and the author of #51430 has proposed a narrower fix-forward instead. That fix-forward, [#51768](https://github.com/vllm-project/vllm/pull/51768), attributes the NVIDIA failure to the legacy V1 model runner's breakable PIECEWISE CUDA graph path and guards that combination in `vllm/config/vllm.py`, leaving the attention implementation exactly as #51430 left it. If #51768 is the one that lands, ROCm stays broken. The two failures are independent. This one does not need a model runner, a graph mode or MTP to show up: it reproduces at compilation mode 0 with no graph captured, and at `FULL_DECODE_ONLY`, which #51768 explicitly keeps allowed. It also cannot appear on CUDA, since nothing preshuffles the weight there. This is not an alternative to either of them. On CUDA the restored method computes exactly what the inlined call computed, so this change is a no-op there and does nothing for the B200 failure — #51768 is still needed for that. In the other direction, if #51750 lands after all, this one becomes redundant and I will close it. Searches for open PRs mentioning `fused_wqa_wkv`, and for open PRs referencing #51430, turned up nothing else that restores this method. ## The change Restore `_fused_wqa_wkv_gemm` on the base class as a plain call into `fused_wqa_wkv`, and send the call site back through it. That is the whole change, eight lines. The narrowed CUDA graph region and everything else from #51430 are untouched. ## Test Plan gfx950 (MI355X), with the model, server arguments and environment taken verbatim from `tests/evals/gsm8k/configs/DeepSeek-V4-Flash-NVFP4.yaml`: tensor parallel 8, fp8 KV cache, compilation mode 3 with `FULL_DECODE_ONLY`, AITER enabled. GSM8K at 8 shots, shortened to 400 questions to keep the turnaround reasonable. The same evaluation was run before the change on the unmodified checkout and after it. To rule out the parts of #51430 that the title points at, the failure was also reproduced at tensor parallel 1 and at compilation mode 0, where no graph is captured at all. ## Test Result Before the change, on current main, accuracy is 0.0000 with an invalid rate of 1.000 — no answer is parsable. It stays 0.0000 at tensor parallel 1 and at compilation mode 0, so neither sharding nor graph capture is what breaks it. After the change, accuracy is 0.953 with an invalid rate of 0.000, comfortably above the 0.84 floor the threshold and its tolerance imply. Reverting #51430 in full gives an equivalent result, which confirms the method removal is the only part of that commit involved. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51821", "createdAt": "2026-08-11T13:05:48Z", "updatedAt": "2026-08-13T01:33:53Z", "timestamp": "2026-08-13T01:33:53Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "rocm", "deepseek" ], "author": "stefankoncarevic", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51823", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(pooling): validate BGE-M3 combined task ownership", "text": "## Purpose Fixes #51801. BGE-M3 can resolve to the internal `embed&token_classify` task. That combined task belongs to the sparse IO-processor plugin, which also converts the combined model output into the public dense/sparse response. Registering a generic pass-through processor would avoid the original `KeyError`, but it would not perform that conversion. This revision keeps the combined task plugin-owned: - fail at startup with an actionable error when `embed&token_classify` is selected without an IO processor plugin; - register only the plugin processor when the plugin is available; - pin the documented sparse server to `token_classify` and require a restart with `token_embed` for ColBERT. This does not restore dynamic per-request task switching or change model weights, kernels, pooling math, or output values. I checked the issue Development section and searched open PRs for `#51801`, `embed&token_classify`, and BGE-M3 pooling processor changes. No overlapping PR was found; #44530 is about asymmetric ColBERT encoding and is unrelated. AI assistance was used for investigation, code drafting, test scaffolding, and review. I reviewed every changed line, reproduced the factory behavior, and understand the change well enough to maintain it. ## Test Plan - `.venv\\Scripts\\python.exe -m pytest tests/entrypoints/pooling/test_factories.py -q --confcutdir=tests/entrypoints/pooling` - `.venv\\Scripts\\python.exe -m ruff check vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `.venv\\Scripts\\python.exe -m ruff format --check vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `.venv\\Scripts\\python.exe -m py_compile vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `git diff --check` Model evaluation is not applicable because the revision changes factory validation and documentation only; it does not affect model execution or numerical outputs. ## Test Result - Focused factory regression: `2 passed` - Ruff check: passed - Ruff format check: passed - Python compile check: passed - Diff check: passed ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose and linked issue are described. - [x] The test plan includes the commands used. - [x] Test results are included. - [x] Documentation is updated for the fixed-task serving contract.
", "url": "https://github.com/vllm-project/vllm/pull/51823", "createdAt": "2026-08-11T13:22:26Z", "updatedAt": "2026-08-13T02:00:49Z", "timestamp": "2026-08-13T02:00:49Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "frontend" ], "author": "030611", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51833", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add DeepGrove Maple (MapleForCausalLM)", "text": "## Purpose Adds support for [`deepgrove/maple-preview`](https://huggingface.co/deepgrove/maple-preview) (`MapleForCausalLM`), a 20B-A1B MoE reasoning model (MIT). Architecture specifics, none of which an existing model file combines: - `layer_types` interleaves sliding-window and global attention 3:1. The checkpoint runs FlashAttention with `window_size=(sliding_window, 0)`, i.e. an inclusive boundary spanning 513 positions, while vLLM maps `sliding_window=N` to `(N - 1, 0)`; the model therefore passes `config.sliding_window + 1`, as `modernbert.py` already does for the same reason. The window is applied per layer via `per_layer_sliding_window`, and `MapleModel.__init__` clears `cache_config.sliding_window` so the global layers do not inherit it. - `nope_on_global_attention`: the `full_attention` layers get no positional encoding, so `rotary_emb` is built only for the sliding layers. RoPE is half-width (`partial_rotary_factor: 0.5` → `rotary_dim=64` of `head_dim=128`). - Per-head QK RMSNorm over the full head dim, before RoPE. - Experts clamp gate/up to ±7, i.e. `SiluAndMulWithClamp` → `FusedMoEFactory(activation=\"silu\", swiglu_limit=...)`, not `swigluoai` (that adds `up + 1` and `alpha=1.702`). - `router_dtype: \"fp32\"` → gate is `ReplicatedLinear(params_dtype=torch.float32)` with `router_logits_dtype=torch.float32`. ### Why `transformers_utils/config.py` is touched The checkpoint declares `auto_map` but **never serializes `model_type`** (it only exists on the class in `configuration_maple.py`). `_CONFIG_REGISTRY` is keyed by `model_type`, so the lookup is skipped entirely and `AutoConfig` demands `trust_remote_code=True`. Registering the class with Transformers does not help either — `has_local_code` is `\"model_type\" in config_dict and ... in CONFIG_MAPPING`, and the first clause short-circuits on the missing key. So the model type is recovered from `architectures`, and the registered class is built directly by reusing the branch that already exists for `_SPECULATIVE_DECODING_CONFIGS`. This keeps `--trust-remote-code` unnecessary and lets vLLM own defaults the checkpoint omits (notably `swiglu_limit`). ### No new parsers The chat template is Qwen3-shaped, so existing parsers work as-is: `--reasoning-parser qwen3 --tool-call-parser hermes --enable-auto-tool-choice`. ## Test Plan ```bash pytest tests/models/test_maple.py tests/transformers_utils/test_missing_model_type.py pytest \"tests/models/test_initialization.py::test_can_initialize_large_subset[MapleForCausalLM]\" pytest tests/models/test_registry.py # 2x H20, bf16 vllm serve deepgrove/maple-preview --max-model-len 8192 [--tensor-parallel-size 2] [--enable-expert-parallel] python tests/evals/gsm8k/gsm8k_eval.py --port 8000 --num-questions 200 --num-shots 5 ``` GSM8K was also run in the model's intended chat/thinking mode (`temperature=0`, answer taken after ``). Numeric parity against the checkpoint's own modeling code, with only its FA3 kernel swapped for an SDPA mask that reproduces `window_size=(w, 0)` exactly, on a 900-token natural-text prompt (`prompt_logprobs=20` vs. `log_softmax` of the reference logits, aligned so that vLLM position `i` is compared against reference position `i - 1`). ## Test Result Lint: all hooks pass; mypy passes on 3.10–3.13. `actionlint` skipped (downloads a Go toolchain, and the diff has no workflow files); `--all-files` unavailable because `pre-commit` needs `git ls-files --deduplicate` (git ≥ 2.31, this box has 2.29.2), hence the diff range. | Test | Result | | --- | --- | | `tests/models/test_maple.py` | 8 passed | | `tests/transformers_utils/test_missing_model_type.py` | 1 passed (verified red when the `architectures` recovery is reverted) | | `test_can_initialize_large_subset[MapleForCausalLM]` | passed | | `tests/models/test_registry.py` | 378 passed, 15 skipped (2 pre-existing failures need `timm`, unrelated) | Weight loading: 9 shards / 37.69 GiB, no missing or unexpected keys. Parity, top-1 agreement in the windowed region (positions > 513) of the 900-token prompt: | | vs reference| | --- | --- | | vLLM (this PR) | **93.78%** (362/386) | | reference vs itself, window shrunk by one token | 93.54% | 93.54% is the ceiling this comparison can reach, so the `+ 1` form is within noise of the reference while the unchanged form is measurably off. Mean |Δ logprob| is 0.091 and next-token top-1 accuracy matches the reference (0.5061 vs 0.5028), confirming there is no systematic divergence. Two findings worth flagging from that harness: - The published modeling code is **silently broken under Transformers 5**: `from_pretrained` leaves the non-persistent `inv_freq` buffer uninitialised (observed values `0.0, 0.0, 2.67e18`), so RoPE degenerates to `cos == 1` everywhere and next-token accuracy collapses from 0.50 to 0.04. vLLM builds its own rotary cache and is unaffected; the numbers above were obtained after restoring `inv_freq` in the reference. - Its RoPE init also fails outright on Transformers 5, since `ROPE_INIT_FUNCTIONS` no longer has a `\"default\"` entry. Another reason to prefer the in-tree `MapleConfig` over `--trust-remote-code`. | GSM8K | chat/thinking, 200 q | 5-shot completion, 200 q | output tok/s | | --- | --- | --- | --- | | TP=1 | **91.0%** | 35.0% | 5020 | | TP=2 | **91.0%** (100 q) | — | — | | TP=2 + EP (128/256 experts per rank) | **90.0%** | 35.5% | 6365 | The low completion-style score is expected rather than a regression: the template opens `` for the assistant, so plain few-shot continuation is not this model's mode. Generations are coherent under both eager and `torch.compile` + full CUDA graphs. Parsers verified end-to-end: `reasoning` and `content` split correctly, and `tool_calls` returns `{\"name\": \"get_weather\", \"arguments\": \"{\\\"city\\\": \\\"Paris\\\"}\"}`. ### Follow-up validation (`a006ca53d6`) This follow-up adds a CPU regression test for the interleaved-cache failure mode seen in earlier sliding-window model fixes: a `full_attention` layer with no per-layer window must not fall back to a model-level cache window. ```bash python -m pytest -v tests/models/test_maple.py \\ tests/transformers_utils/test_missing_model_type.py \\ 'tests/models/test_registry.py::test_registry_imports[MapleForCausalLM]' CUDA_VISIBLE_DEVICES='' VLLM_TARGET_DEVICE=cpu \\ python -m pytest -v tests/transformers_utils/test_missing_model_type.py python -m pytest -v \\ 'tests/models/test_initialization.py::test_can_initialize_large_subset[MapleForCausalLM]' ``` Results: - Maple unit/config/registry selection: **13 passed**. The new test drives `EngineArgs.create_engine_config()` from a config without `model_type` and verifies the recovered 3:1 `layer_types` plus `cache_config.sliding_window is None`. - CPU-only config tests: **2 passed**. - Dummy-weight initialization: passed. - Real `deepgrove/maple-preview` checkpoint on H20: bf16 TP=1 eager serving, prefix caching, default `torch.compile` with full/piecewise CUDA graphs, and repeated identical-prefix requests all generated successfully. The compiled run captured every configured CUDA graph size and produced identical greedy tokens for the repeated request. - Real checkpoint TP=2 with `--enable-expert-parallel`: passed. Each rank held 128 of 256 experts and generated eight tokens successfully. - Spot checks against the original safetensors found exact equality for the embedding, Q/K/V and output projections, QK norms, decoder norms, and LM head. A standalone MoE check against the checkpoint's fp32 router and clamped SwiGLU reference had max/mean absolute errors of 0.015625/0.000626. ## Known limitations - Speculative decoding: `ngram` works (verified). EAGLE3/DFlash/DSpark do not, because `MapleForCausalLM` does not implement `SupportsEagle3` and no draft checkpoint exists for this target; adding the `EagleModelMixin` hooks is left for a follow-up. - `num_shared_experts != 0`, `moe_router_enable_expert_bias` and non-`silu` `hidden_act` raise rather than load something subtly wrong. - `MixtureOfExperts` (EPLB) is not implemented; only `enable_eplb` / `num_redundant_experts` are plumbed through. - Only tested on H20 (Hopper). - Parity is measured at the logits level over one prompt, not per layer. ## Accountability Duplicate-work checks found no existing PR or issue for this model or architecture (`gh pr list`/`gh issue list` over `Maple`, `deepgrove`, `MapleForCausalLM`; the only hits were fuzzy matches on unrelated PRs #42657, #44475, #44633). AI assistance was used to write and test this change. The human submitter has reviewed every changed line, understands the change end-to-end, and ran the tests and evals reported above. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51833", "createdAt": "2026-08-11T14:50:39Z", "updatedAt": "2026-08-13T06:34:37Z", "timestamp": "2026-08-13T06:34:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "new-model" ], "author": "NancyFyong", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51836", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[qwen3] correct linear_attn norm and a_log parameter dtype to F32 for qwen 3.5 series", "text": "## Purpose For Qwen 3.5 series (Gated Delta Net) models, the parameter `language_model.model.layers.{i}.linear_attn.norm.weight` is stored in **fp32** in the [HF checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B/tree/main?show_file_info=model.safetensors.index.json), but vLLM currently allocates it in **bf16** (for bf16 models) and silently casts the loaded values into it. If this is used with offline sharded state, the pre-sharded chedckpoints norm layer will use bf16, so the sharded checkpoint deviates from the original fp32 weight. However, Qwen 3.6 series, the linear_attn.norm weight is stored in `bf16`. We need to differentiates the two series and use the correct dtype. The same for linear attn a log parameter, for Qwen 3.5 series, the a log parameter dtype is `F32`, as well for Qwen 3.6 series, the a log parameter dtype is `BF16`. This PR adds a new text config attribute `real_model_type` demonstrating the real model type is qwen3_5 or qwen3_6. For now, the official qwen 3.5 and qwen 3.6 repo config.json does not contains `real_model_type` in `text_config`, the behavior after this pr is the same with before. But if `text_config` has the `real_model_type` field specified, and its value is, for now, `qwen3_5` or `qwen3_6`, the linear attention a_log and norm parameter will be correctly set with the weight. ## Test Plan Tested manually with offline pre shard command: ``` python examples/features/sharded_state/save_sharded_state_offline.py --model ~/.cache/huggingface/hub/models--Qwen--Qwen3.x-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 --output /home/sharded_weigts --max-num-seqs 512 --tensor-parallel-size 2 --max-file-size 1073741824 ``` for Qwen 3.5 series: 1. the sharded output norm layer parameter is F32 2. the sharded output a log layer parameter is BF16 For Qwen 3.6 series: 1. the sharded output norm layer parameter is BF16 2. the sharded output a log layer parameter is BF16 ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51836", "createdAt": "2026-08-11T15:14:24Z", "updatedAt": "2026-08-13T03:12:25Z", "timestamp": "2026-08-13T03:12:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "qwen" ], "author": "andyxning", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51843", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable fine-grained prefix-cache hits for incompatible hybrid KV layouts", "text": "## Summary Fine-grained prefix-cache hits are enabled for hybrid models containing Mamba `align` groups. However other groups, such as a sliding-window DSpark drafter, may use KV cache managers that only support block-aligned lookups. This previously caused an assertion when prefix caching was enabled. This change: - Enables fine-grained hits only when every KV cache manager supports the required lookup granularity. - Falls back to scheduler-block-aligned prefix hits for incompatible layouts. - Prevents the scheduler from creating partial-tail cache entries after that fallback. - Keeps the drafter’s native sliding-window KV representation, avoiding model-specific overrides. - Logs when fine-grained hits are disabled. ## Kimi K3 results with DSpark SWA - KV capacity is the same with and without: 7,815,323 tokens / 7.45× concurrency. - Repeated 66,007-token prompt reused 57,600 tokens. - The same hit remained after 300 GSM8K requests. - Mean acceptance length: 4.87, 4.83, 4.83. - GSM8K accuracy: 0.30, 0.32, 0.32. - Zero invalid responses. Short and medium prompts may receive no prefix-cache reuse because hits must land on the much larger scheduler alignment. ## Validation - 23 partial-prefix-cache tests passed. - 89 broader prefix-cache tests passed. - Pre-commit passed. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51843", "createdAt": "2026-08-11T16:11:07Z", "updatedAt": "2026-08-12T19:53:11Z", "timestamp": "2026-08-12T19:53:11Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "ready", "kimi" ], "author": "mgoin", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51846", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Frontend] Fix prompt_logprobs=0 bypassing admission guards and chat echo", "text": "## Summary `prompt_logprobs=0` is a valid, meaningful `SamplingParams` value (\"return the prompt token's own logprob, no alternatives\"), but several checks along the request path test it with plain truthiness instead of `is not None`, so a value of `0` silently slips through: - `ChatCompletionRequest.to_sampling_params` inherits `top_logprobs` (which defaults to `0`, not `None`) for *any* `echo=True` chat request, even when the client never asked for logprobs at all. Every echo request therefore does unrequested prompt-logprobs work (extra `compute_logits` pass, `gather_logprobs`, forced device sync per prefill step, and an unrequested `prompt_logprobs` array in the response). `cast_bool(null)`-style truthy checks miss `0` the same way `bool(0)` does anywhere else in Python. - `AsyncLLM.add_request` and `check_logprobs` (both chat and completion protocol) use truthy checks, so `prompt_logprobs=0` bypasses the `--kv-sharing-fast-prefill` incompatibility guard and the documented `stream=True` rejection — while `GPUModelRunner` downstream checks `is not None` and hits a bare `assert`, which aborts the whole `execute_model` step and kills the engine-core process for **every** in-flight and future client, not just the offending request. - `LLMEngine.add_request` (the sync path used by `LLM.generate()`) had no such guard at all, for any value — so the same crash is reachable there too, for any `prompt_logprobs`, not just `0`. ## Fix - Only inherit `top_logprobs` for chat `echo` when `logprobs` was actually requested (`ChatCompletionRequest.logprobs`), mirroring `CompletionRequest`'s existing, already-correct idiom (its source field, `logprobs`, defaults to `None`, so its analogous branch is a no-op unless the client explicitly set it). - Use `is not None` consistently for the `--kv-sharing-fast-prefill` admission guard in both `AsyncLLM.add_request` (async path) and `LLMEngine.add_request` (sync path, guard added — there was none). - `check_logprobs` now rejects any non-`None` `prompt_logprobs` under `stream=True`, in both the chat and completion protocols, instead of only `> 0` / `== -1`. - Fixed a stray \"tokens, tokens,\" typo in the `GPUModelRunner` backstop assert message, touched while already on that exact line. Diff is intentionally scoped to these checks — no changes to the grammar, scheduler, or any GPU/kernel code. ## Not a duplicate of #49622 #49622 (open) adds the sync-path admission check to `LLMEngine.add_request` and fixes the assert typo, but keeps the **truthy** check (`params.prompt_logprobs`), so it does not close the `prompt_logprobs=0` gap, and it doesn't touch the chat `echo` implicit-enablement bug or the `stream=True` rejection gap. This PR's `LLMEngine.add_request` change and typo fix overlap with #49622 at the file level; if #49622 merges first this one becomes a small `is not None` correction on top of it (and drops the typo hunk), rather than a conflicting approach — happy to rebase either way. ## Test Plan Unit tests added: - `tests/entrypoints/openai/chat_completion/test_chat.py` — `test_chat_completion_request_echo_without_logprobs_does_not_set_prompt_logprobs`, `test_chat_completion_request_echo_with_logprobs_inherits_top_logprobs`, `test_chat_completion_request_explicit_prompt_logprobs_zero_is_preserved`, `test_chat_completion_request_rejects_prompt_logprobs_zero_with_stream` - `tests/entrypoints/openai/completion/test_completion.py` — `test_completion_request_rejects_prompt_logprobs_zero_with_stream` - `tests/v1/engine/test_async_llm.py` — `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` - `tests/v1/engine/test_llm_engine.py` — `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` Commands run (Linux, real install, not a container/CI runner): ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=cpu python -m pytest tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_echo_without_logprobs_does_not_set_prompt_logprobs \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_echo_with_logprobs_inherits_top_logprobs \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_explicit_prompt_logprobs_zero_is_preserved \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_rejects_prompt_logprobs_zero_with_stream \\ tests/entrypoints/openai/completion/test_completion.py::test_completion_request_rejects_prompt_logprobs_zero_with_stream \\ -v # 5 passed # Red/green check: reverted only the 5 source files to their pre-fix state # (git checkout HEAD~1 -- ) and reran the same 3 assertion-bearing # tests -> all 3 FAILED (assert 0 is None / DID NOT RAISE VLLMValidationError), # confirming they genuinely exercise the bug. Restored the fix -> green again. ruff check # All checks passed! ruff format --check # 9 files already formatted ``` `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` (both the async and sync variants) require booting a real `LLM`/`AsyncLLM` instance. In my local sandbox this hits `AttributeError: '_OpNamespace' '_C' object has no attribute 'init_cpu_memory_env'` on **any** engine boot, including a completely vanilla one with no `kv_sharing_fast_prefill` involved — a precompiled-wheel/HEAD-source drift specific to my environment, not caused by this change. They should run normally in CI, where the build matches the commit under test. ## Model evaluation Not applicable — this is a request-validation/control-flow fix. It does not change model forward passes, sampling, or output tokens for any request that was already passing validation correctly. ## AI assistance disclosure This change was developed with AI assistance (Claude). I reviewed every changed line, ran the tests and lint listed above myself, and can defend the fix and its scope in review.", "url": "https://github.com/vllm-project/vllm/pull/51846", "createdAt": "2026-08-11T16:35:16Z", "updatedAt": "2026-08-13T15:59:23Z", "timestamp": "2026-08-13T15:59:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "frontend", "needs-rebase" ], "author": "YaseenBashaT", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51851", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump the minor-update group across 1 directory with 174 updates", "text": "Bumps the minor-update group with 174 updates in the / directory: | Package | From | To | | --- | --- | --- | | [regex](https://github.com/mrabarnett/mrab-regex) | `2026.2.28` | `2026.7.19` | | [requests](https://github.com/psf/requests) | `2.32.3` | `2.34.2` | | [tqdm](https://github.com/tqdm/tqdm) | `4.67.3` | `4.70.0` | | [tokenizers](https://github.com/huggingface/tokenizers) | `0.22.2` | `0.23.1` | | [fastapi](https://github.com/fastapi/fastapi) | `0.136.3` | `0.141.1` | | [starlette](https://github.com/Kludex/starlette) | `1.3.1` | `1.6.0` | | [aiohttp](https://github.com/aio-libs/aiohttp) | `3.13.3` | `3.14.3` | | [openai](https://github.com/openai/openai-python) | `2.44.0` | `2.53.0` | | [pydantic](https://github.com/pydantic/pydantic) | `2.12.0` | `2.13.4` | | [prometheus-client](https://github.com/prometheus/client_python) | `0.22.0` | `0.26.0` | | [prometheus-fastapi-instrumentator](https://github.com/trallnag/prometheus-fastapi-instrumentator) | `8.0.2` | `8.1.0` | | [tiktoken](https://github.com/openai/tiktoken) | `0.12.0` | `0.13.0` | | [lark](https://github.com/lark-parser/lark) | `1.2.2` | `1.3.1` | | [typing-extensions](https://github.com/python/typing_extensions) | `4.15.0` | `4.16.0` | | [filelock](https://github.com/tox-dev/py-filelock) | `3.16.1` | `3.32.2` | | [jsonschema](https://github.com/python-jsonschema/jsonschema) | `4.23.0` | `4.26.0` | | [pybase64](https://github.com/mayeut/pybase64) | `1.4.3` | `1.5.0` | | [anthropic](https://github.com/anthropics/anthropic-sdk-python) | `0.112.0` | `0.121.0` | | [opentelemetry-sdk](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-api](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | opentelemetry-semantic-conventions-ai | `0.4.13` | `0.5.1` | | [grpcio](https://github.com/grpc/grpc) | `1.78.0` | `1.83.0` | | [grpcio-reflection](https://grpc.io) | `1.78.0` | `1.83.0` | | [numba](https://github.com/numba/numba) | `0.65.0` | `0.66.0` | | [peft](https://github.com/huggingface/peft) | `0.19.1` | `0.20.0` | | [tensorizer](https://github.com/coreweave/tensorizer) | `2.10.1` | `2.12.1` | | runai-model-streamer | `0.15.7` | `0.16.1` | | [conch-triton-kernels](https://github.com/stackav-oss/conch) | `1.2.1` | `1.3` | | [annotated-types](https://github.com/annotated-types/annotated-types) | `0.7.0` | `0.8.0` | | griffelib | `2.0.2` | `2.1.0` | | [idna](https://github.com/kjd/idna) | `3.10` | `3.18` | | [platformdirs](https://github.com/tox-dev/platformdirs) | `4.3.6` | `4.11.1` | | [pydantic-core](https://github.com/pydantic/pydantic) | `2.41.1` | `2.48.0` | | [pygments](https://github.com/pygments/pygments) | `2.18.0` | `2.20.0` | | [ruff](https://github.com/astral-sh/ruff) | `0.15.12` | `0.16.2` | | [six](https://github.com/benjaminp/six) | `1.16.0` | `1.17.0` | | [urllib3](https://github.com/urllib3/urllib3) | `2.2.3` | `2.7.0` | | [tblib](https://github.com/ionelmc/python-tblib) | `3.1.0` | `3.2.2` | | [absl-py](https://github.com/abseil/abseil-py) | `2.1.0` | `2.5.0` | | [accelerate](https://github.com/huggingface/accelerate) | `1.13.0` | `1.14.0` | | [aiohappyeyeballs](https://github.com/aio-libs/aiohappyeyeballs) | `2.6.1` | `2.7.1` | | [alembic](https://github.com/sqlalchemy/alembic) | `1.16.4` | `1.19.1` | | arctic-inference | `0.1.1` | `0.2.0` | | [argcomplete](https://github.com/kislyuk/argcomplete) | `3.5.1` | `3.7.2` | | [audioread](https://github.com/beetbox/audioread) | `3.0.1` | `3.1.0` | | [azure-core](https://github.com/Azure/azure-sdk-for-python) | `1.38.2` | `1.41.0` | | [azure-storage-blob](https://github.com/Azure/azure-sdk-for-python) | `12.28.0` | `12.30.0` | | [blobfile](https://github.com/blobfile/blobfile) | `3.0.0` | `3.2.0` | | [bm25s](https://github.com/xhluca/bm25s) | `0.2.13` | `0.3.10` | | [boto3](https://github.com/boto/boto3) | `1.35.57` | `1.43.67` | | [botocore](https://github.com/boto/botocore) | `1.35.57` | `1.43.67` | | [cffi](https://github.com/python-cffi/cffi) | `2.0.0` | `2.1.1` | | [cohere-melody](https://github.com/cohere-ai/melody) | `0.11.1` | `0.13.0` | | [colorlog](https://github.com/borntyping/python-colorlog) | `6.10.1` | `6.12.0` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.10.6` | `7.15.4` | | [cramjam](https://github.com/milesgranger/pyrus-cramjam) | `2.9.0` | `2.11.0` | | [cuda-bindings](https://github.com/NVIDIA/cuda-python) | `13.0.3` | `13.3.1` | | [cuda-pathfinder](https://github.com/NVIDIA/cuda-python) | `1.3.3` | `1.6.0` | | [cuda-toolkit](https://developer.nvidia.com/cuda-toolkit) | `13.0.3.0` | `13.3.1` | | [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) | `0.26.3` | `0.72.2` | | [dataproperty](https://github.com/thombashi/DataProperty) | `1.0.1` | `1.1.1` | | [decorator](https://github.com/micheles/decorator) | `5.1.1` | `5.3.1` | | [dill](https://github.com/uqfoundation/dill) | `0.3.8` | `0.4.1` | | [distlib](https://github.com/pypa/distlib) | `0.3.9` | `0.4.3` | | [dnspython](https://github.com/rthalley/dnspython) | `2.7.0` | `2.8.0` | | [docker](https://github.com/docker/docker-py) | `7.1.0` | `7.2.0` | | [einx](https://github.com/fferflo/einx) | `0.3.0` | `0.4.3` | | [email-validator](https://github.com/JoshData/python-email-validator) | `2.2.0` | `2.3.0` | | [fastapi-cloud-cli](https://github.com/fastapilabs/fastapi-cloud-cli) | `0.21.0` | `0.23.0` | | [fonttools](https://github.com/fonttools/fonttools) | `4.55.0` | `4.63.0` | | [frozenlist](https://github.com/aio-libs/frozenlist) | `1.5.0` | `1.8.0` | | [genson](https://github.com/wolverdude/genson) | `1.3.0` | `1.4.0` | | [google-api-core](https://github.com/googleapis/google-cloud-python) | `2.24.2` | `2.34.0` | | [google-auth](https://github.com/googleapis/google-cloud-python) | `2.40.2` | `2.56.3` | | [google-cloud-core](https://github.com/googleapis/google-cloud-python) | `2.4.3` | `2.6.1` | | [google-cloud-storage](https://github.com/googleapis/google-cloud-python) | `3.4.0` | `3.13.1` | | [google-crc32c](https://github.com/googleapis/python-crc32c) | `1.7.1` | `1.8.0` | | [google-resumable-media](https://github.com/googleapis/google-cloud-python) | `2.7.2` | `2.10.1` | | [googleapis-common-protos](https://github.com/googleapis/google-cloud-python) | `1.70.0` | `1.75.1` | | [greenlet](https://github.com/python-greenlet/greenlet) | `3.2.3` | `3.5.4` | | [h11](https://github.com/python-hyper/h11) | `0.14.0` | `0.16.0` | | [h2](https://github.com/python-hyper/h2) | `4.3.0` | `4.4.1` | | [hiredis](https://github.com/redis/hiredis-py) | `3.0.0` | `3.4.1` | | [hpack](https://github.com/python-hyper/hpack) | `4.1.0` | `4.2.0` | | [httpx](https://github.com/encode/httpx) | `0.27.2` | `0.28.1` | | [humanize](https://github.com/python-humanize/humanize) | `4.11.0` | `4.16.0` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.131.0` | `6.165.2` | | [iniconfig](https://github.com/pytest-dev/iniconfig) | `2.0.0` | `2.3.0` | | [jiter](https://github.com/pydantic/jiter) | `0.15.0` | `0.16.0` | | [jmespath](https://github.com/jmespath/jmespath.py) | `1.0.1` | `1.1.0` | | [joblib](https://github.com/joblib/joblib) | `1.4.2` | `1.5.3` | | [jsonschema-rs](https://github.com/Stranger6667/jsonschema) | `0.46.5` | `0.49.9` | | [kiwisolver](https://github.com/nucleic/kiwi) | `1.4.7` | `1.5.0` | | [lazy-loader](https://github.com/scientific-python/lazy-loader) | `0.4` | `0.5` | | [librosa](https://github.com/librosa/librosa) | `0.10.2.post1` | `0.11.0` | | [llvmlite](https://github.com/numba/llvmlite) | `0.47.0` | `0.48.0` | | [mako](https://github.com/sqlalchemy/mako) | `1.3.10` | `1.4.1` | | [matplotlib](https://github.com/matplotlib/matplotlib) | `3.9.2` | `3.10.9` | | [mpmath](https://github.com/mpmath/mpmath) | `1.3.0` | `1.4.1` | | [msal](https://github.com/AzureAD/microsoft-authentication-library-for-python) | `1.34.0` | `1.37.0` | | [msgpack](https://github.com/msgpack/msgpack-python) | `1.1.0` | `1.2.1` | | [mteb](https://github.com/embeddings-benchmark/mteb) | `2.8.3` | `2.18.16` | | [multidict](https://github.com/aio-libs/multidict) | `6.1.0` | `6.7.1` | | [mypy-extensions](https://github.com/python/mypy_extensions) | `1.0.0` | `1.1.0` | | [networkx](https://github.com/networkx/networkx) | `3.2.1` | `3.4.2` | | [nltk](https://github.com/nltk/nltk) | `3.9.1` | `3.10.2` | | [nvidia-cublas](https://developer.nvidia.com/cuda-zone) | `13.1.1.3` | `13.6.1.10` | | [nvidia-cuda-cupti](https://developer.nvidia.com/cuda-zone) | `13.0.85` | `13.3.75` | | [nvidia-cuda-nvrtc](https://developer.nvidia.com/cuda-zone) | `13.0.88` | `13.3.33` | | [nvidia-cuda-runtime](https://developer.nvidia.com/cuda-zone) | `13.0.96` | `13.3.29` | | [nvidia-cudnn-cu13](https://developer.nvidia.com/cuda-zone) | `9.20.0.48` | `9.24.0.43` | | [nvidia-cufft](https://developer.nvidia.com/cuda-zone) | `12.0.0.61` | `12.3.0.29` | | [nvidia-cufile](https://developer.nvidia.com/cuda-zone) | `1.15.1.6` | `1.18.1.6` | | [nvidia-cusolver](https://developer.nvidia.com/cuda-zone) | `12.0.4.66` | `12.2.6.9` | | [nvidia-cusparse](https://developer.nvidia.com/cuda-zone) | `12.6.3.3` | `12.8.2.51` | | [nvidia-cusparselt-cu13](https://developer.nvidia.com/cusparselt) | `0.8.1` | `0.9.1` | | [nvidia-nccl-cu13](https://developer.nvidia.com/cuda-zone) | `2.29.7` | `2.30.7` | | [nvidia-nvjitlink](https://developer.nvidia.com/cuda-zone) | `13.0.88` | `13.3.33` | | [nvidia-nvshmem-cu13](https://developer.nvidia.com/cuda-zone) | `3.4.5` | `3.7.2` | | [nvidia-nvtx](https://developer.nvidia.com/cuda-zone) | `13.0.85` | `13.3.29` | | [opentelemetry-exporter-otlp-proto-common](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp-proto-grpc](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp-proto-http](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-prometheus](https://github.com/open-telemetry/opentelemetry-python) | `0.56b0` | `0.65b0` | | [opentelemetry-proto](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-semantic-conventions](https://github.com/open-telemetry/opentelemetry-python) | `0.56b0` | `0.65b0` | | [pathvalidate](https://github.com/thombashi/pathvalidate) | `3.2.1` | `3.3.1` | | [perceptron](https://github.com/perceptron-ai-inc/perceptron) | `0.1.4` | `0.3.5` | | [pluggy](https://github.com/pytest-dev/pluggy) | `1.5.0` | `1.6.0` | | [polars](https://github.com/pola-rs/polars) | `1.29.0` | `1.43.2` | | [pooch](https://github.com/fatiando/pooch) | `1.8.2` | `1.9.0` | | [propcache](https://github.com/aio-libs/propcache) | `0.2.0` | `0.5.2` | | [proto-plus](https://github.com/googleapis/google-cloud-python) | `1.26.1` | `1.28.3` | | [pycryptodomex](https://github.com/Legrandin/pycryptodome) | `3.22.0` | `3.23.0` | | [pydantic-extra-types](https://github.com/pydantic/pydantic-extra-types) | `2.10.5` | `2.11.1` | | [pydantic-settings](https://github.com/pydantic/pydantic-settings) | `2.14.2` | `2.15.0` | | [pyjwt](https://github.com/jpadilla/pyjwt) | `2.11.0` | `2.13.0` | | [pyparsing](https://github.com/pyparsing/pyparsing) | `3.2.0` | `3.3.2` | | [pystemmer](https://github.com/snowballstem/pystemmer) | `3.0.0` | `3.1.0` | | [pytest-forked](https://github.com/pytest-dev/pytest-forked) | `1.6.0` | `1.7.5` | | [pytest-mock](https://github.com/pytest-dev/pytest-mock) | `3.14.0` | `3.15.1` | | [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) | `2.3.1` | `2.4.0` | | [python-rapidjson](https://github.com/python-rapidjson/python-rapidjson) | `1.20` | `1.23` | | [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) | `3.12.1` | `3.14.5` | | [referencing](https://github.com/python-jsonschema/referencing) | `0.35.1` | `0.37.0` | | [responses](https://github.com/getsentry/responses) | `0.25.3` | `0.26.2` | | rignore | `0.7.6` | `0.8.1` | | runai-model-streamer-azure | `0.15.7` | `0.16.1` | | runai-model-streamer-gcs | `0.15.7` | `0.16.1` | | runai-model-streamer-s3 | `0.15.7` | `0.16.1` | | [s3transfer](https://github.com/boto/s3transfer) | `0.10.3` | `0.19.2` | | [sacrebleu](https://github.com/mjpost/sacrebleu) | `2.4.3` | `2.6.0` | | [schemathesis](https://github.com/schemathesis/schemathesis) | `4.21.6` | `4.24.3` | | [scikit-learn](https://github.com/scikit-learn/scikit-learn) | `1.5.2` | `1.7.2` | | [scipy](https://github.com/scipy/scipy) | `1.13.1` | `1.15.3` | | [sentence-transformers](https://github.com/huggingface/sentence-transformers) | `5.2.0` | `5.7.0` | | [sentry-sdk](https://github.com/getsentry/sentry-python) | `2.63.0` | `2.67.0` | | [soundfile](https://github.com/bastibe/python-soundfile) | `0.12.1` | `0.14.0` | | [sympy](https://github.com/sympy/sympy) | `1.13.3` | `1.14.0` | | [tabulate](https://github.com/astanin/python-tabulate) | `0.9.0` | `0.10.0` | | [termcolor](https://github.com/termcolor/termcolor) | `3.1.0` | `3.3.0` | | [threadpoolctl](https://github.com/joblib/threadpoolctl) | `3.5.0` | `3.6.0` | | [torchcodec](https://github.com/pytorch/torchcodec) | `0.14.0+cu130` | `0.15.0` | | [tritonclient](https://github.com/triton-inference-server/client) | `2.64.0` | `2.71.0` | | [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.1` | | [uvicorn](https://github.com/Kludex/uvicorn) | `0.35.0` | `0.52.1` | | [vector-quantize-pytorch](https://github.com/lucidrains/vector-quantizer-pytorch) | `1.21.2` | `1.31.1` | | [wcwidth](https://github.com/jquast/wcwidth) | `0.2.13` | `0.8.2` | | [xxhash](https://github.com/ifduyue/python-xxhash) | `3.5.0` | `3.8.1` | | [yarl](https://github.com/aio-libs/yarl) | `1.17.1` | `1.24.5` | | pynvvideocodec | `2.0.4` | `2.2.0` | | [nvidia-cutlass-dsl](https://github.com/NVIDIA/cutlass) | `4.6.2` | `4.7.0` | | [tokenspeed-mla](https://github.com/lightseekorg/tokenspeed) | `0.1.8` | `0.2.5` | Updates `regex` from 2026.2.28 to 2026.7.19
Changelog

Sourced from regex's changelog.

Version: 2026.8.12

Support Python 3.15.

Version: 2026.7.19

Git issue 607: SIGSEGV: NULL pointer dereference in `basic_match()` when `(?R)`/`(?0)` recursion is used inside a fuzzy-matching quantifier Git issue 608: SIGSEGV: out-of-bounds read in `bytes1_char_at()` with `DOTALL` + a fuzzy constraint + `(?r)` reverse-direction matching

Version: 2026.7.10

Clarified that the POSIX flag makes it look for only the longest overall match. 
Git issue 606: regex's compiled-pattern cache eviction is not free-threading-safe

Version: 2026.6.28

Git issue 604: regex fails on free-threading python because setlocale is not thread-safe

Version: 2026.5.9

Reverse matching with full unicode casefolding could lead to out-of-range string indexes.

Version: 2026.4.4

A fix for older Python versions before free-threading was supported.

Version: 2026.4.3

More fixes for free-threading.

Version: 2026.3.32

Fixed segfault.

Version: 2026.3.31

Fixed bug again.

Version: 2026.3.30

Fixed bug.

Version: 2026.3.28

Fixed version.

Version: 2026.3.27

Various fixes, including ones to improve free-threading support.

... (truncated)

Commits
  • 0525aff Git issue 608: SIGSEGV: out-of-bounds read in bytes1_char_at() with `DOTALL...
  • 07dfa3b Git issue 607: SIGSEGV: NULL pointer dereference in basic_match() when `(?R...
  • e7716da Python 3.15 not released yet.
  • e204ddb Git issue 606: regex's compiled-pattern cache eviction is not free-threadin...
  • d7e1927 Updated main.yml for Python 3.15.
  • b18daa6 Git issue 604: regex fails on free-threading python because setlocale is not ...
  • e57d185 Reverse matching with full unicode casefolding lead to out-of-range string in...
  • bc57b04 A fix for older Python versions before free-threading was supported.
  • 773e213 More fixes for free-threading.
  • 5d51c75 Fixed segfault.
  • Additional commits viewable in compare view

Updates `requests` from 2.32.3 to 2.34.2
Release notes

Sourced from requests's releases.

v2.34.2

2.34.2 (2026-05-14)

  • Moved headers input type back to Mapping to avoid invariance issues with MutableMapping and inferred dict types. Users calling Request.headers.update() may need to narrow typing in their code. (#7441)

Full Changelog: https://github.com/psf/requests/blob/main/HISTORY.md#2342-2026-05-14

v2.34.1

2.34.1 (2026-05-13)

Bugfixes

  • Widened json input type from dict and list to Mapping and Sequence. (#7436)
  • Changed headers input type to MutableMapping and removed None from Request.headers typing to improve handling for users. (#7431)
  • Response.reason moved from str | None to str to improve handling for users. (#7437)
  • Fixed a bug where some bodies with custom __getattr__ implementations weren't being properly detected as Iterables. (#7433)

New Contributors

Full Changelog: https://github.com/psf/requests/blob/main/HISTORY.md#2341-2026-05-13

v2.34.0

2.34.0 (2026-05-11)

Announcements

  • Requests 2.34.0 introduces inline types, replacing those provided by typeshed. Public API types should be fully compatible with mypy, pyright, and ty. We believe types are comprehensive but if you find issues, please report them to the pinned tracking issue.

    Special thanks to @​bastimeyer, @​cthoyt, @​edgarrmondragon, and @​srittau for helping review and test the types ahead of the release. (#7272)

Improvements

  • Digest Auth hashing algorithms have added usedforsecurity=False to clarify security considerations. (#7310)
  • Requests added support for Python 3.15 based on beta1. Downstream projects should be able to start testing prior to its release in October. (#7422)
  • Requests added support for Python 3.14t. (#7419)

Bugfixes

  • Response.history no longer contains a reference to itself, preventing accidental looping when traversing the history list. (#7328)
  • Requests no longer performs greedy matching on no_proxy domains. The

... (truncated)

Changelog

Sourced from requests's changelog.

2.34.2 (2026-05-14)

  • Moved headers input type back to Mapping to avoid invariance issues with MutableMapping and inferred dict types. Users calling Request.headers.update() may need to narrow typing in their code. (#7441)

2.34.1 (2026-05-13)

Bugfixes

  • Widened json input type from dict and list to Mapping and Sequence. (#7436)
  • Changed headers input type to MutableMapping and removed None from Request.headers typing to improve handling for users. (#7431)
  • Response.reason moved from str | None to str to improve handling for users. (#7437)
  • Fixed a bug where some bodies with custom __getattr__ implementations weren't being properly detected as Iterables. (#7433)

2.34.0 (2026-05-11)

Announcements

  • Requests 2.34.0 introduces inline types, replacing those provided by typeshed. Public API types should be fully compatible with mypy, pyright, and ty. We believe types are comprehensive but if you find issues, please report them to the pinned tracking issue.

    Special thanks to @​bastimeyer, @​cthoyt, @​edgarrmondragon, and @​srittau for helping review and test the types ahead of the release. (#7272)

Improvements

  • Digest Auth hashing algorithms have added usedforsecurity=False to clarify security considerations. (#7310)
  • Requests added support for Python 3.15 based on beta1. Downstream projects should be able to start testing prior to its release in October. (#7422)
  • Requests added support for Python 3.14t. (#7419)

Bugfixes

  • Response.history no longer contains a reference to itself, preventing accidental looping when traversing the history list. (#7328)
  • Requests no longer performs greedy matching on no_proxy domains. The proxy_bypass implementation has been updated with CPython's fix from bpo-39057. (#7427)
  • Requests no longer incorrectly strips duplicate leading slashes in URI paths. This should address user issues with specific presigned URLs. Note the full fix requires urllib3 2.7.0+. (#7315)

... (truncated)

Commits

Updates `tqdm` from 4.67.3 to 4.70.0
Release notes

Sourced from tqdm's releases.

tqdm v4.70.0 stable

  • contrib.concurrent: major improvements
    • support process_map(mp_context, max_tasks_per_child), thread_map(thread_name_prefix) (#1265)
    • fix total based on shortest iterable length (#1473)
    • use default max_workers (#1543 <- #1530, #1518)
    • support timeout, buffersize (#1576)
    • improve ETA (#1708 <- #1161)
    • update as_completed (#1709 <- #1565)
    • add tqdm.concurrent.intepreter_map (#1777)
  • asyncio: support iterables with only __aiter__ (#1714 <- #1686)
  • support reset(float("inf")) (#1783 <- #1781, #651)
  • framework: test & reduce wheel size (#1782)

tqdm v4.69.1 stable

tqdm v4.69.0 stable

  • add tqdm.asyncio.gather(..., return_exceptions=False) (#1776, #1671 <- #1286)
  • misc minor framework updates
    • bump workflow actions & pre-commit hooks

tqdm v4.68.4 stable

tqdm v4.68.3 stable

  • utils: delay os.get_terminal_size (#1763 <- #1760)
  • autonotebook: support QtConsole, Spyder, JupyterLite (#1763, #1628, #1559 <- #1283, #1098, #512)
  • minor docs updates
    • fix typo (#1762)
    • use git-fame
  • misc minor framework updates
    • fix & update CI build
    • pre-commit: add docs & metadata generation
    • move tox.ini -> pyproject.toml, move tox-gh-actions -> tox-gh
    • add Python 3.14, drop 3.7 support

tqdm v4.68.2 stable

  • revert accidental change to ascii default (fixes #1760)
    • UnicodeEncodeError: 'charmap' codec can't encode characters in position 6-7: character maps to <undefined> can be fixed by installing tqdm!=4.68.0,!=4.68.1
  • misc docs updates
    • fix links
    • replace stray rst -> md syntax
    • consistent "progress bar" terminology (#1737)

... (truncated)

Commits
  • 96f2e60 Merge pull request #1777 from shermansiu/feat/interpreter-pool
  • c27393e misc tidy
  • 061c623 Disable tqdm.monitor_interval for subinterpreters because they do not have ...
  • 9fc160b Update how we check for interpreter_map support in the tests
  • b42463a Ensure that subinterpreters can import tqdm while unpickling the initializer
  • 16d5486 Add support for nested progress bars
  • 9f5890f Add initial implementation for interpreter_map
  • 321f920 Merge pull request #1783 from LuShadowX/reset-inf-total
  • 4664b57 minor tidy
  • 426a098 Treat inf total as unknown in reset() too
  • Additional commits viewable in compare view

Updates `tokenizers` from 0.22.2 to 0.23.1
Release notes

Sourced from tokenizers's releases.

Release v0.23.1

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

🚨 Breaking changes

  • Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
  • add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
  • Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
  • 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

⚡ Performance — measured locally on this Mac, not lifted from PRs

Run with cargo bench --bench <name> -- --save-baseline v0_22_2 on v0.22.2, then --baseline v0_22_2 on v0.23.1. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.

Added-vocabulary deserialize — the headline win (#1995, #1999)

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) is now representative of how transformers actually loads tokenizer.json files. The combined effect of daachorse for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:

benchmark v0.22.2 v0.23.1 change
100k tokens, special, no norm ~410 ms 248 ms −40%
100k tokens, non-special, no norm ~7.1 s 273 ms −96%
100k tokens, special, NFKC ~395 ms 235 ms −40%
100k tokens, non-special, NFKC ~7.4 s 290 ms −96%
400k tokens, special, no norm ~15 s 980 ms −94%

Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".

BPE encode

benchmark v0.22.2 v0.23.1 change
BPE GPT2 encode batch, no cache 530 ms 446 ms −16%
BPE GPT2 encode batch (cached) 690 ms 685 ms noise
BPE GPT2 encode (single) 1.95 s 1.94 s noise
BPE Train (small) 32.6 ms 31.5 ms −3%
BPE Train (big) 1.01 s 988 ms −2%

The BPE per-thread cache PR (#2028) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.

Llama-3 encode

... (truncated)

Commits

Updates `fastapi` from 0.136.3 to 0.141.1
Release notes

Sourced from fastapi's releases.

0.141.1

Fixes

  • 🐛 Fix support for background tasks and headers from dependencies in app.frontend(). PR #16105 by @​tiangolo.

Docs

0.141.0

Features

  • ✨ Add app.frontend(check_dir="auto"), to make local development more convenient with fastapi dev. PR #16102 by @​tiangolo.

0.140.13

Fixes

Docs

0.140.12

Fixes

0.140.11

Fixes

  • 🐛 Fix response_model_* params ignored for non-generator endpoints with Iterable[..] return type. PR #15093 by @​YuriiMotov.

0.140.10

Fixes

Internal

0.140.9

Fixes

  • 🐛 Fix exclude_defaults not propagated to dict keys and values in jsonable_encoder. PR #16043 by @​MBGrao.

... (truncated)

Commits
  • 95f8322 🔖 Release version 0.141.1 (#16106)
  • f137944 📝 Update release notes
  • d623544 🐛 Fix support for background tasks and headers from dependencies in `app.fron...
  • 1d211b9 📝 Update release notes
  • 8a1f876 📝 Document FASTAPI_ENV in FastAPI CLI guide (#16104)
  • c7e7b65 🔖 Release version 0.141.0 (#16103)
  • 6bceb84 📝 Update release notes
  • 5429fed ✨ Add app.frontend(check_dir="auto"), to make local development more conven...
  • 628663f 🔖 Release version 0.140.13 (#16096)
  • 0b54fd0 📝 Update release notes
  • Additional commits viewable in compare view

Updates `starlette` from 1.3.1 to 1.6.0
Release notes

Sourced from starlette's releases.

Version 1.6.0

What's Changed

New Contributors

Full Changelog: https://github.com/Kludex/starlette/compare/1.5.1...1.6.0

Version 1.5.1

What's Changed

Full Changelog: https://github.com/encode/starlette/compare/1.5.0...1.5.1

Version 1.5.0

This release is all about giving GZipMiddleware some love. 🗜️

What's Changed

Full Changelog: https://github.com/encode/starlette/compare/1.4.1...1.5.0

Version 1.4.1

What's Changed

Full Changelog: https://github.com/Kludex/starlette/compare/1.4.0...1.4.1

Version 1.4.0

What's Changed

New Contributors

Full Changelog: https://github.com/Kludex/starlette/compare/1.3.1...1.4.0

Changelog

Sourced from starlette's changelog.

1.6.0 (August 8, 2026)

Added

  • Add max_body_size to Starlette and route classes #3431.
  • Expose http.response.debug information via response extensions #3130.

1.5.1 (August 8, 2026)

Fixed

  • Reject inverted single-byte ranges in FileResponse #3389.
  • Limit FileResponse to 100 ranges #3430.

1.5.0 (August 8, 2026)

Added

  • Add exclude_content_types parameter to GZipMiddleware #3418.

Changed

  • Expand default excluded content types in GZipMiddleware #3421.

Fixed

  • Flush GZip output for each streamed chunk #3419.
  • Skip compression of partial responses in GZipMiddleware #3420.

1.4.1 (August 5, 2026)

Fixed

  • Default thread_minimum_size to 128 KiB in GZipResponder, keeping it usable without the new keyword argument #3415.

1.4.0 (August 5, 2026)

Added

  • Offload large GZip compression to a worker thread, keeping the event loop responsive. GZipMiddleware accepts a new thread_minimum_size parameter (default 128 KiB) controlling the minimum body chunk size compressed in a thread #3410.

Changed

  • Use zlib.compressobj instead of GzipFile in GZipMiddleware, reducing memory usage during compression #3411.
  • Lazily allocate GZipMiddleware compression resources, avoiding compressor allocation for responses that are never compressed #3407.
Commits
  • 4f250d6 Version 1.6.0 (#3434)
  • 9eea41a Expose http.response.debug info via response extensions (#3130)
  • 38f8999 Add max_body_size to Starlette and route classes (#3431)
  • c41236c Version 1.5.1 (#3432)
  • 9c500db Limit FileResponse to 100 ranges (#3430)
  • 78ae82c Reject inverted single-byte Range like bytes=5-4 (#3389)
  • c1d6eda chore(deps): bump pymdown-extensions from 11.0 to 11.0.1 (#3429)
  • ee66ca4 chore(deps): bump the python-packages group across 1 directory with 8 updates...
  • 00d1016 fix(tests): skip test_staticfiles_filename_too_long on Windows where os.pathc...
  • d96887e Add Pydantic Logfire banner to the docs (#3428)
  • Additional commits viewable in compare view

Updates `aiohttp` from 3.13.3 to 3.14.3 Updates `openai` from 2.44.0 to 2.53.0
Release notes

Sourced from openai's releases.

v2.53.0

2.53.0 (2026-08-03)

Features

  • api: Add gpt-5.5 and tool name/namespace to Responses types (#3569) (dd1202d)

Bug Fixes

  • ci: avoid NumPy source builds and duplicate HTTPX coverage (#3573) (b58332f)

v2.52.1

2.52.1 (2026-07-31)

Full Changelog: v2.52.0...v2.52.1

Chores

  • ci: pin setup-uv v5 to its underlying commit (#3560) (cbdc98b)

v2.52.0

2.52.0 (2026-07-31)

Full Changelog: v2.51.0...v2.52.0

Features

  • api: content provenance checks (1d6c118)

Bug Fixes

  • client: honor Retry-After delays up to two minutes (#3555) (7fa7946)

Documentation

v2.51.0

2.51.0 (2026-07-30)

Full Changelog: v2.50.0...v2.51.0

Features

... (truncated)

Changelog

Sourced from openai's changelog.

2.53.0 (2026-08-03)

Features

  • api: Add gpt-5.5 and tool name/namespace to Responses types (#3569) (dd1202d)

Bug Fixes

  • ci: avoid NumPy source builds and duplicate HTTPX coverage (#3573) (b58332f)

2.52.1 (2026-07-31)

Full Changelog: v2.52.0...v2.52.1

Chores

  • ci: Reproducer ```python import json, os, tempfile from vllm import LLM, SamplingParams CONFIG = { \"architectures\": [\"LagunaForCausalLM\"], \"model_type\": \"laguna\", \"hidden_size\": 512, \"intermediate_size\": 1024, \"num_hidden_layers\": 4, \"num_attention_heads\": 8, \"num_attention_heads_per_layer\": [8, 16, 8, 16], \"num_key_value_heads\": 2, \"head_dim\": 64, \"max_position_embeddings\": 4096, \"rms_norm_eps\": 1e-6, \"rope_theta\": 500000.0, \"tie_word_embeddings\": False, \"torch_dtype\": \"bfloat16\", \"mlp_only_layers\": [0, 1, 2, 3], \"num_experts\": 4, \"num_experts_per_tok\": 2, \"moe_intermediate_size\": 128, \"shared_expert_intermediate_size\": 128, } model_dir = tempfile.mkdtemp() with open(os.path.join(model_dir, \"config.json\"), \"w\") as f: json.dump(CONFIG, f) llm = LLM(model=model_dir, tokenizer=\"\", load_format=\"dummy\", dtype=\"bfloat16\", max_model_len=2048, enforce_eager=True) out = llm.generate([\"context \" * 64], SamplingParams(max_tokens=512, temperature=0.0, ignore_eos=True)) print(\"SURVIVED\", len(out[0].outputs[0].token_ids)) ```
## Test Result - Unit tests: 6 passed (3 new, plus the 3 existing sliding-window cases). - Reproducer: hangs on current `main` and is killed at the timeout; passes in 1.5s with this change. - Laguna-XS 300-token decode: previously segfaulted or hung, now completes in 47s with coherent output. - GSM8K 5-shot, full 1319 prompts. Before this change the same run died with an `execute_model` timeout. | Filter | exact_match | stderr | | --- | --- | --- | | flexible-extract | 0.9014 | ±0.0082 | | strict-match | 0.8908 | ±0.0086 | Instrumenting the scratchpad bounds shows the overrun directly on unpatched `main`: the region holds 6336 bytes per KV head while the split-KV write path needs 8384, because sizing assumes 6 query heads per KV head (48 heads / 8 KV heads) and the 64-head layers address it with a stride of 8. ``` [SCRATCHPAD-SIZING] split_num=2 max_tile=6 q_heads=24 kv_heads=4 q_per_kv=6 per_kv_head=6336 [SCRATCHPAD-OVERFLOW] per_head_need=8384 per_kv_head=6336 q_head_tile_size=8 ``` Models whose layers share one head count keep the existing single-blob path, so there is no extra allocation or lookup for them. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51852", "createdAt": "2026-08-11T17:18:34Z", "updatedAt": "2026-08-13T04:21:57Z", "timestamp": "2026-08-13T04:21:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "ci/build", "cpu", "verified" ], "author": "ganeshr10", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51855", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[K3] support recoverssm for K3", "text": " ## Purpose This PR adds ReplaySSM speculative decoding for the NVIDIA Kimi-K3 KDA path on Model Runner V2. It is intended for Kimi-K3 serving with DSpark and supports `mamba_cache_mode=align` prefix caching. Without ReplaySSM, KDA speculative decoding materializes a full recurrent state for every speculative position. ReplaySSM keeps one checkpoint and compact per-token records, then reconstructs the state at the accepted position after every verify. Under the tested TP8 Kimi-K3 configuration, this increases the effective cache capacity by 10.97% at the same 46.48 GiB cache-memory budget. ## Design ### Kimi-K3-specific V2 integration The implementation is contained in the Kimi-K3 KDA metadata, layer, and ReplaySSM operator paths. The metadata builder gives speculative rows one KDA checkpoint slot and creates a model-specific post-sampling committer. The V1 integration from the initial implementation was removed because Kimi-K3 does not run on Model Runner V1. ReplaySSM is selected through `--use-replayssm`. The regular KDA prefill and non-speculative decode paths are unchanged. ### Verify records FP32 corrections KDA verify reads the checkpoint without modifying it and computes the normal speculative outputs. For each speculative token it records: - the correction vector `u` in FP32; - the key and raw gate in the activation dtype. Recording `u` removes `v` and `beta` from the commit path and avoids recomputing the state-dependent correction. FP32 is used for `u` to preserve reconstruction accuracy; keys and gates remain BF16 in the evaluated configuration. ### Direct accepted-position reconstruction After verification determines the accepted length, one Triton kernel reconstructs the required checkpoint directly. For the KDA recurrence: ```text S_t = S_{t-1} * exp(g_t) + u_t outer k_t ``` The accepted state is: ```text S_p = S_0 * exp(sum(g_1 ... g_p)) + sum_j u_j outer (k_j * exp(sum(g_{j+1} ... g_p))) ``` The commit kernel evaluates this expression in reverse token order. It loads the original state once and does not recursively update or write intermediate states. The loop bound is the runtime accepted length rather than the complete eight-token DSpark activation window. In align mode, the same launch also reconstructs the next prefix-cache boundary when the accepted range crosses it. Convolution history is compacted to the accepted position and to the boundary using the same commit plan. ### Cache layout Each KDA page contains: ```text (convolution state, KDA checkpoint, FP32 correction records, key/gate records) ``` Replay records are transient and are not included in prefix-cache state copies. The checkpoint and convolution state remain the prefix-cache boundary state. ## Scope - NVIDIA Kimi-K3 KDA - Model Runner V2 - speculative decoding, validated with Kimi-K3-DSpark and seven draft tokens - `mamba_cache_mode=align` prefix caching This PR does not add a generic ReplaySSM interface to shared attention or model state classes, and it does not add Mamba2 support. ## Test Plan and Results ### Unit tests ```bash .venv/bin/python -m pytest tests/models/kimi_k3/test_kda.py -v ``` Result: `69 passed`. The 18 ReplaySSM cases cover: - the actual eight-token DSpark window; - partial and full acceptance lengths; - direct final-state and align-boundary reconstruction; - both convolution-state layouts; - request-index remapping; - gates with and without a lower bound; - cache shape and dtype derivation. ```bash .venv/bin/pre-commit run --files vllm/model_executor/layers/mamba/mamba_utils.py vllm/models/kimi_k3/nvidia/kda.py vllm/models/kimi_k3/nvidia/ops/replayssm.py tests/models/kimi_k3/test_kda.py ``` Result: all hooks passed. ```bash .venv/bin/pre-commit run mypy-3.12 --all-files --hook-stage manual ``` Result: passed for Python 3.12. ### End-to-end accuracy Controlled setup: - Kimi-K3 target and Kimi-K3-DSpark draft model, TP8 on two GB300 nodes; - seven greedy draft tokens and block rejection sampling; - align-mode Mamba cache, prefix caching, FP8 MLA cache, FlashInfer MLA, and fastsafetensors; - identical datasets, prompts, scorers, and serving parameters for ReplaySSM ON and OFF. | GSM8K, 1,319 samples | ReplaySSM ON | OFF | ON - OFF | |---|---:|---:|---:| | Flexible extract | 96.3609% | 96.1334% | +0.2274 pp | | Strict match | 96.0576% | 95.9060% | +0.1516 pp | The paired 95% confidence intervals were `[-0.4549, +0.9098]` pp for flexible extract and `[-0.6065, +0.9098]` pp for strict match. Both runs completed all 1,319 requests with zero request errors. Independently initialized greedy serving runs are not bitwise deterministic, so the paired score comparison is used instead of exact generated-text equality. | MRCR, 40 long prompts run cold and warm | ReplaySSM ON | OFF | |---|---:|---:| | Cold score | 0.74879 | 0.66791 | | Warm score | 0.74879 | 0.66791 | | Cold/warm exact-output consistency | 100% | 97.5% | | Warm prefix-cache token hit rate | 89.13% | 90.50% | The MRCR prompts ranged from 16,757 to 33,772 tokens and contained 991,216 prompt tokens per pass. ReplaySSM completed both passes without request errors. Total DSpark acceptance was 53.3514% with ReplaySSM and 53.4062% without it, a difference of -0.0548 percentage points. GSM8K-only acceptance was 50.0899% with ReplaySSM and 50.0336% without it. ### Memory capacity Both variants used a fixed 46.48 GiB cache-memory budget. | Metric | ReplaySSM ON | OFF | Difference | |---|---:|---:|---:| | Effective cache capacity | 2,856,170 tokens | 2,573,777 tokens | +10.97% | | Max 262k-request concurrency | 10.895x | 9.818x | +10.97% | ### Nsight Systems profile The profile used batch 32, 128 forced output tokens, and 80 captured decode steps per GPU on the same two-node TP8 setup. The aggregate results below include eight initial small-batch steps. | KDA work per decode step | Initial recurrent ReplaySSM commit | This PR | |---|---:|---:| | Verify | 3.115 ms | 3.540 ms | | Commit | 4.793 ms | 1.464 ms | | Commit auxiliaries | 0.081 ms | 0.081 ms | | Total | 7.989 ms | 5.085 ms | The direct reconstruction makes the commit kernel 3.27x faster and reduces total ReplaySSM KDA time by 36.34%. The FP32 correction write increases verify time by 13.66%, but commit time falls by 69.45%. Native non-ReplaySSM KDA took 2.772 ms per step in the corresponding OFF trace. The 72 steady batch-32 steps each verified 256 tokens. Their CUDA-graph model-forward GPU wall time was 65.208 ms with ReplaySSM and 64.873 ms without it, a 0.335 ms or 0.52% difference. Kernel residence exceeds wall time because kernels on concurrent streams overlap. | ReplaySSM ON steady forward kernel residence | Time per step | Share | |---|---:|---:| | DeepGEMM and nvJet GEMM | 38.760 ms | 56.5% | | MoE routing and packing | 8.121 ms | 11.8% | | SiTU activation | 7.682 ms | 11.2% | | TP collectives | 6.136 ms | 8.9% | | ReplaySSM verify | 3.791 ms | 5.5% | | KDA convolution and normalization | 0.938 ms | 1.4% | | MLA attention kernels | 0.519 ms | 0.8% | | Other kernels | 2.658 ms | 3.9% | | Total kernel residence | 68.603 ms | 100% | ReplaySSM verify runs inside model forward and replaces the native KDA recurrent kernel. Its steady kernel residence was 3.791 ms versus 3.009 ms for native KDA, an increase of 0.782 ms. The smaller 0.335 ms forward wall-time difference reflects kernel overlap and run-to-run variation. Commit runs after sampling determines the accepted length, so it is not part of model forward. Across the 72 steady batch-32 steps it took 1.711 ms per step and varied with the accepted length. | Post-sampling ReplaySSM commit | Time per step | Share | |---|---:|---:| | FP32 correction state reconstruction | 1.618 ms | 94.5% | | Convolution-state compaction | 0.081 ms | 4.7% | | Commit-plan preparation | 0.0069 ms | 0.4% | | Align bookkeeping | 0.0061 ms | 0.4% | | Total | 1.711 ms | 100% | Adding the steady model-forward difference and post-sampling commit gives an estimated ReplaySSM GPU critical-path cost of 2.046 ms per batch-32 decode step, before common sampling and postprocessing work. ## Duplicate-work check The following searches were run against open vLLM PRs: ```bash gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM Kimi K3\" gh pr list --repo vllm-project/vllm --state open --search \"KDA speculative decoding\" gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM in:title,body\" ``` No other open PR implements this Kimi-K3 KDA integration. Related work is materially different: - #49847 targets Mamba2; - #49887 targets Gated DeltaNet; - #50140 is a WIP FlashInfer ReplaySSM kernel; - #42792 adds generic Mamba align-prefix speculative support rather than Kimi-K3 KDA ReplaySSM. ## AI assistance AI assistance was used for implementation, tests, profiling analysis, model evaluation analysis, and this PR description. The submitting human must review every changed line, reproduce the relevant tests, and understand and defend the design end-to-end. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and exact commands. - [x] Unit, end-to-end accuracy, memory, acceptance, and performance results. - [x] Duplicate-work check and AI-assistance disclosure.
**BEFORE SUBMITTING, PLEASE READ **", "url": "https://github.com/vllm-project/vllm/pull/51855", "createdAt": "2026-08-11T17:49:54Z", "updatedAt": "2026-08-13T17:03:24Z", "timestamp": "2026-08-13T17:03:24Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "performance", "needs-rebase", "mrv2", "verified", "kimi", "k3" ], "author": "ZJY0516", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51860", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][K3] Dequantize the fp8 decode query for MLA backends without quant-query support - TRITON_MLA", "text": "## Purpose When enabling fp8 kv-cache dtype in DSpark speculative decoding, we got an assert error. THis PR is to put out a minimal fix to unblock this path. On ROCm, the DSpark draft auto-selects TRITON_MLA, the only backend supporting its non-causal multi-token blocks atm. TRITON_MLA dequantizes fp8 KV on load and takes a bf16 query, so Kimi-K3's `_decode_concat_cache` assert on `supports_quant_query_input` makes fp8 KV unusable with DSpark. Run the fused fp8 q-concat + cache insert as usual and dequantize the returned query when the impl does not accept an fp8 query. No-op for backends that do (all current NVIDIA K3 backends and ROCM_AITER_MLA). Verified gfx950 x8 TP8: without the change the serve asserts at startup; with it, DSpark+fp8 serves at 94.4% gsm8k (1319q, greedy), acceptance 3.40. To reproduce: ``` VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_AITER_MLA_ASM_PADDING=asm \\ VLLM_USE_BREAKABLE_CUDAGRAPH=0 \\ VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 SAFETENSORS_FAST_GPU=1 \\ vllm serve /data/Kimi-K3-public \\ --trust-remote-code --tensor-parallel-size 8 --mm-encoder-tp-mode data \\ --gpu-memory-utilization 0.95 --max-model-len 10240 \\ --max-num-batched-tokens 4096 --max-num-seqs 128 \\ --reasoning-parser kimi_k3 --kv-cache-dtype fp8 --no-enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"\",\"num_speculative_tokens\":7}' ``` Startup AssertionError: ``` RuntimeError: Worker failed with error 'Kimi-K3 fp8 KV cache decode requires a backend that accepts an fp8 (quantized) query input.', please check the stack trace above for the root cause ``` Fixed: serves; gsm8k 1319q greedy = 94.4%, 0.1% invalid, 724 tok/s, acceptance 3.40 (logs: serve_minfix_pos.log, k3_gsm8k_minfix_1319q.log) - NVIDIA impact: structurally none where `supports_quant_query_input=True (identical call, new branch skipped); converts assert→working for dequant-on-load backends per the flag's own contract - non-DSpark fp8 + TRITON_MLA impact: It does not need the fallback. The no-spec decode path never reaches _decode_concat_cache; it goes through the generic MLA flow, which already handles the capability gracefully (mla_attention.py quantizes the query only if the impl supports it — no assert, bf16 query otherwise). ## Test Plan test asm-mla + fp8 kv-cache + DSpark: ``` VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_AITER_MLA_ASM_PADDING=asm \\ VLLM_USE_BREAKABLE_CUDAGRAPH=0 \\ VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 SAFETENSORS_FAST_GPU=1 \\ vllm serve /data/Kimi-K3-public \\ --trust-remote-code --tensor-parallel-size 8 --mm-encoder-tp-mode data \\ --gpu-memory-utilization 0.95 --max-model-len 10240 \\ --max-num-batched-tokens 4096 --max-num-seqs 128 \\ --reasoning-parser kimi_k3 --kv-cache-dtype fp8 --no-enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"\",\"num_speculative_tokens\":7}' ``` vllm serve and gsm8k accuracy ## Test Result config | assert version | fallback version -- | -- | -- DSpark + fp8 (TRITON_MLA draft, implicit) | asserts at startup | 94.4% gsm8k, no-spec + fp8 + TRITON_MLA (explicit) | serves (assert unreachable) | serves, 94.7% gsm8k, — fallback provably inert here ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51860", "timestamp": "2026-08-12T16:40:42Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "ready", "quantization", "kimi", "k3" ], "author": "hongxiayang", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:51862", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 Remove prefill pipeline stall in chunk KDA", "text": "## Purpose On ROCm's Kimi-K3 path, each prefill/mixed step has a stall in `prepare_chunk_indices`, caused by two interacting factors: ``` def prepare_chunk_indices(cu_seqlens: torch.Tensor, chunk_size: int) -> torch.Tensor: indices = torch.cat( [ torch.arange(n) for n in triton.cdiv(prepare_lens(cu_seqlens), chunk_size).tolist() ] ) return torch.stack([indices.eq(0).cumsum(0) - 1, indices], 1).to(cu_seqlens) ``` - `tolist()` D2H copy makes the host wait until the device queue is emptied - Host ops executing before the next H2D copy, while the device is sitting idle The data that the host waits on is already sitting on the host. At each step, `GDNAttentionMetadataBuilder` builds `chunk_indices` and `chunk_offsets` from `query_start_loc_cpu` and saves them in attention metadata. However, the ROCm Kimi-K3 path does not use it, and the stall waiting on and rebuilding the data from the device is a duplication of the same work. This PR addresses this and removes the stall by - Making the chunk kda kernels optionally accept `chunk_indices` and `chunk_offsets`. When they're provided, they don't need to be rebuilt through `prepare_chunk_indices`. - Passing these metadata to the kernels to actually remove the stall. - For the `prepare_chunk_offsets` call in metadata builder, building them directly on the device, instead of having a synchronization point waiting on the host-side python loop and host-device copy before the next device workload can be queued. These remove the stall from `prepare_chunk_indices`, and achieves TTFT and performance gain especially in low concurrency scenarios where the stall cannot be hidden by the heavy device workloads. ## Test Plan E2E benchmark and lm_eval with gsm8k on MI355X Server command ``` vllm serve moonshotai/Kimi-K3 \\ --served-model-name moonshotai/Kimi-K3 \\ --trust-remote-code \\ --moe-backend auto \\ --tensor-parallel-size 8 \\ --load-format auto \\ --gpu-memory-utilization 0.95 \\ --mm-encoder-tp-mode data \\ --max-num-seqs 128 \\ --max-num-batched-tokens 4096 \\ --max-model-len 10240 \\ --no-enable-prefix-caching \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --attention-backend TRITON_MLA ``` Benchmark ``` vllm bench serve \\ --model moonshotai/Kimi-K3 --backend vllm --trust-remote-code \\ --dataset-name random \\ --random-input-len 8192 --random-output-len 128 \\ --max-concurrency \"$C\" --num-prompts $((C * 4)) \\ --temperature 0 --ignore-eos --seed 1234 ``` lm_eval ``` lm_eval --model local-chat-completions \\ --model_args \"model=moonshotai/Kimi-K3,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,max_gen_toks=2048,timeout=3600\" \\ --tasks gsm8k --num_fewshot 20 \\ --apply_chat_template --fewshot_as_multiturn \\ --seed 1234 ``` ## Test Result gsm8k on MI355X |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 20|exact_match|↑ |0.9757|± |0.0042| | | |strict-match | 20|exact_match|↑ |0.9765|± |0.0042| Benchmark results | ISL/OSL | Conc | Config | Total tokens/s | Delta (%) | Total reqs/s | Delta (%) | Output tokens/s | Delta (%) | mean TTFT (ms) | Delta (%) | mean TPOT (ms) | Delta (%) | mean ITL (ms) | Delta (%) | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 8192/128 | 8 | main | 8052.42 | -| 0.97 | - | 123.88 | - | 1813.30 | - | 50.47 | - | 50.47 | - | | 8192/128 | 8 | PR | 8316.70 | +3.28% | 1.00 | +3.09% | 127.95 | +3.29% | 1766.05 | -2.61% | 48.78 | -3.35% | 48.78 | -3.35% | | 8192/128 | 16 | main | 10067.08 | - | 1.21 | - | 154.88 | - | 2481.99 | - | 83.84 | - | 83.84 | - | | 8192/128 | 16 | PR | 10358.07 | +2.89% | 1.24 | +2.48% | 159.35 | +2.89% | 2447.98 | -1.37% | 81.17 | -3.18% | 81.17 | -3.18% | | 8192/128 | 32 | main | 11674.19 | - | 1.40 | - | 179.60 | - | 3729.02 | - | 148.50 | - | 148.50 | - | | 8192/128 | 32 | PR | 11766.98 | +0.79% | 1.41 | +0.71% | 181.03 | +0.80% | 3713.48 | -0.42% | 147.20 | -0.88% | 147.20 | -0.88% | | 8192/128 | 64 | main | 12760.26 | - | 1.53 | - | 196.31 | - | 6267.18 | - | 275.10 | - | 275.10 | - | | 8192/128 | 64 | PR | 12822.07 | +0.48% | 1.54 | +0.65% | 197.26 | +0.48% | 6255.47 | -0.19% | 273.57 | -0.56% | 273.57 | -0.56% | ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51862", "createdAt": "2026-08-11T19:00:58Z", "updatedAt": "2026-08-13T09:11:46Z", "timestamp": "2026-08-13T09:11:46Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "rocm", "ready", "kimi", "k3" ], "author": "kliuae", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51864", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Fix MultiConnector returning duplicate finished_recving signals", "text": "## Purpose Resolves #51846 Fixes a scheduler crash (`assert req.status == RequestStatus.WAITING_FOR_REMOTE_KVS`) that occurs in Disaggregated Prefill/Decode (PD) setups when multiple consumer KV connectors are configured (e.g., `NixlConnector` + `LMCacheMPConnector`). ### Root Cause When multiple KV connectors are configured as consumers, `MultiConnector.get_finished()` blindly aggregated the `finished_recving` signal from *all* connectors. Even though `MultiConnector` only assigns the actual load operation to a single chosen connector, the unassigned connectors are given `0` tokens to load in `update_state_after_alloc`. As a result, they immediately consider their load operation \"completed\" and return the `req_id` as finished receiving. This results in the `MultiConnector` yielding the same `req_id` in two separate scheduler steps. The first signal correctly transitions the request out of `WAITING_FOR_REMOTE_KVS` to `RUNNING`, but the second signal crashes the scheduler since the request is no longer waiting for KV. ### The Fix Updated `MultiConnector.get_finished()` to only trust the `finished_recving` signal if the connector reporting it was the one explicitly chosen to load the request's KV. ## Test Plan This is a pure logic fix to `MultiConnector` (fixing an oversight in the loop conditional). **Local linting:** - `pre-commit run ruff-check` - `pre-commit run mypy-3.12 --all-files --hook-stage manual` **Testing:** Due to local environment dependency extraction errors blocking `pytest`, I am relying on the automated GitHub Actions CI pipeline to verify the existing `tests/v1/kv_connector/unit/test_multi_connector.py` suite. ## Test Result - `pre-commit` passed cleanly on the modified file. - No model evaluation is included because this change is confined to the KV connector communication layer and cannot affect model output accuracy or logic. > **Note:** AI assistance was used to root cause. I have reviewed every changed line and verified the logic fix myself. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51864", "createdAt": "2026-08-11T19:59:19Z", "updatedAt": "2026-08-13T10:19:40Z", "timestamp": "2026-08-13T10:19:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "PrakshaaleJain", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51876", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Shard the MoE experts across NUMA nodes", "text": "## Purpose On a rank whose OpenMP threads span several NUMA nodes, the CPU MoE experts read their weights from every node. `parallel_2d` splits the work by thread index and has no notion of where a page lives, so each thread ends up pulling roughly `1 - 1/nodes` of its bytes across the interconnect. The kernel is already shaped for the fix. Both of its GEMM stages parallelise over the **output rows of the weight they read** — stage 1 over the `2N` rows of w1, stage 2 over the `K` rows of w2 — so handing a node a contiguous slice of that axis makes every thread read only pages placed on its own node. The outputs stay disjoint, so there is **no reduction, no barrier and no extra buffer**: the change is which thread gets which block, plus an `mbind` at load time. ### When this helps, and when it does nothing With the default `VLLM_CPU_OMP_THREADS_BIND=auto`, `_get_autobind_cpu_ids()` gives each rank the CPUs of exactly one NUMA node and asserts there are enough nodes to go around, so CPUs and memory already agree and this change is a no-op — the policy detects that and declines. It helps in one configuration: **a single rank whose threads span several NUMA nodes**, which is what a manual `VLLM_CPU_OMP_THREADS_BIND=0-127` on a multi-socket box gives you. ### How the policy decides The kernel maps OpenMP thread `ith` to shard `ith / (nth / shards)`, so sharding is only correct where that map is true of the actual thread binding. The policy therefore reads the binding itself — the ordered CPU list in `KMP_AFFINITY`'s `proclist` or `GOMP_CPU_AFFINITY` — and **verifies** the map rather than assuming it: the first `nth` entries have to form equal contiguous runs, one node each. The run boundaries are then the shard boundaries, and each run's node is where that shard's pages go. It declines, leaving the existing code path byte for byte, whenever that does not hold: threads on a single node (the auto-bind default), `OMP_NUM_THREADS` shorter than the CPU list so the first `nth` threads sit on a subset of the nodes, a list interleaved across nodes rather than grouped by them, a thread count that does not divide, fewer blocks than nodes, or a binding that does not name a CPU per thread at all — which includes the `OMP_PLACES={0,1,...}` + `OMP_PROC_BIND=true` fallback, a single place holding every CPU. Shard index and node id are kept distinct throughout: a rank confined to the second socket of an SNC box is nodes 2 and 3, not 0 and 1. `VLLM_CPU_MOE_NUMA_NODES` overrides the decision with an explicit list of node ids. ## Test Plan ```bash pytest tests/kernels/moe/test_cpu_moe_numa_shard.py pytest tests/kernels/moe/test_cpu_quant_fused_moe.py pytest tests/kernels/moe/test_cpu_fused_moe.py ``` Sharding only changes *which thread* computes which block; the arithmetic of each block is untouched. So the bar is `torch.equal` against the unsharded run, not a tolerance — anything else means a block was computed twice, or not at all, or against the wrong slice of the weights, and those are the three ways this can be wrong. ## Test Result | | | |---|---| | `test_cpu_moe_numa_shard.py` (new) | **19 passed.** 16 geometries × {2, 4} shards, all bit-identical | | `test_cpu_quant_fused_moe.py` | **154 passed** with the default; **58 passed** with sharding forced to 2 and to 4 | | `test_cpu_fused_moe.py` | **220 passed** with 1, 2 and 4 shards | The geometries deliberately include ones that do **not** divide evenly: `N=160` gives 10 blocks of `BLOCK_N`, which across 4 nodes is 3/3/2/2. Slices are whole blocks but need not be equal — requiring equality would have excluded `intermediate_size=2880`, which is the default in `benchmarks/kernels/cpu/benchmark_cpu_fused_moe.py`. ## Performance 2× Xeon Platinum 8592V (Emerald Rapids), SNC2, 4 NUMA domains, gcc 14.2. Geometry taken from this repo's own CPU MoE benchmark (hidden 2880, intermediate 2880, so stage 2 splits 90 blocks as 23/23/22/22). 64 threads bound in node order with `GOMP_CPU_AFFINITY`, one process per row, best of 30 iterations, three runs: | | GB/s of expert weights, six runs | median | spread | |---|---|---|---| | unsharded | 214 / 224 / 223 / 216 / 225 / 228 | 224 | 6.2% | | **4 shards** | **381 / 384 / 387 / 388 / 388 / 388** | **387** | **1.9%** | Paired ratios: 1.78 / 1.71 / 1.73 / 1.79 / 1.73 / 1.70 — **1.70× to 1.79×, median 1.73×**. Measured on an otherwise idle machine (load average 0.50 at the start of the run; an earlier set taken while the box was shared is what the 6.2% on the unsharded row is *not* — that spread is reproducible and is discussed below). Every row carries its thread placement, verified from `/proc/self/task/*/stat` rather than assumed: `64 active (64 parked), {n0=16, n1=16, n2=16, n3=16}`, no collisions. Reading that from `sched_getaffinity` instead would have been useless here — with `GOMP_CPU_AFFINITY` the calling thread is thread 0, pinned to one CPU, and reports `{node0: 1}` no matter how the other 63 landed. The unsharded row is three times more variable than the sharded one (6.2% against 1.9%), consistently. That is what you would expect if it is the one whose performance depends on where pages happen to have landed: with the shards placed, each thread reads its own node every time. Two notes on the measurement, both of which changed the number: * **Experts are rotated between iterations.** Repeating the same 8 experts keeps a 398 MB working set inside the 640 MB of L3 across the two sockets and reports **726 GB/s** — above this machine's 415 GB/s aggregate STREAM ceiling, which is the tell. Rotating puts both rows back under the ceiling and moves the ratio from 1.40× to 1.7×. * **One process per row**, because undoing a pinning restores the affinity mask but does not move threads that are already running, so a later row inherits the placement of an earlier one. ## Scope, and what this does not do * Weights are placed, scales are not. They are a sixteenth of the bytes for MXFP4, and on the FP8 block-quantized path their row axis does not line up with the block split, so a second rule would buy a few percent. * This does **not** change the process memory policy. `init_cpu_memory_env` still binds a rank's allocations to one node, so a model larger than a single node still fails to allocate; that is a separate problem, diagnosed in #51584. * Placement is driven from `CPUWorker.load_model` rather than from each quantization method's `process_weights_after_loading`, because the MXFP4 and FP8 CPU paths do not call the experts' post-load hook at all — hooking per-method would silently miss exactly the backends this helps most. ## Limitations **Measured on a single machine**: 2× Xeon Platinum 8592V, SNC2, 4 NUMA domains. The policy is topology-driven rather than hardcoded, but I have no access to an EPYC box to confirm the gain carries over to NPS4 or to per-CCD domains, and I would rather say so than have a reviewer find out. A 2P NPS4 box brings two changes at once — more domains *and* more shards — and I can only reason about the second: the policy never emits more shards than there are effective nodes, so raising the count past 4 here would mean breaking an invariant the patch itself enforces, and I would rather not quote a number I had to disable a check to get. If it would help the review, I am happy to measure the cost of the sharded structure alone at higher shard counts behind a temporary override. ## Notes for review `csrc/cpu/sgl-kernels/` is vendored from SGLang and re-synced wholesale, so the primitive lives outside it in `csrc/cpu/moe_numa_shard.hpp` and `moe_numa_parallel.hpp`; **the diff inside the vendored directory is three lines per file** — one include and the call names. `BLOCK_N` necessarily exists in three places (kernel, loader, policy). A `static_assert` in `moe_numa_parallel.hpp` ties the C++ copy to `block_size_n()`, so a change there breaks the build rather than leaving the loader placing pages on boundaries the kernel does not split on — which would be slower with no sign of it.", "url": "https://github.com/vllm-project/vllm/pull/51876", "createdAt": "2026-08-11T20:41:09Z", "updatedAt": "2026-08-13T11:03:01Z", "timestamp": "2026-08-13T11:03:01Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "cpu" ], "author": "ccaadaro", "state": "open", "assignees": [ "bigPYJ1151" ] }, { "id": "github:vllm-project/vllm:pull_request:51877", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Speed Up ROCm Skinny GEMM Tests (reduced parameterizations, ", "text": "This PR reduces the `test_rocm_skinny_gemm` suite from a ~2 hour runtime to about 10 seconds by reducing the number of parameterizations from 11040 to 2644, and removing the unnecessary environment cleanup between tests. We now just cleanup the environment one time at the end of the entire module rather than needlessly eating the 0.3s cost to cleanup after each parameterization.", "url": "https://github.com/vllm-project/vllm/pull/51877", "timestamp": "2026-08-12T16:55:08Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "micah-wil", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:51879", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Expose data-parallel topology to offloading backends", "text": "## Purpose Native KV-offloading backends currently receive the engine's `data_parallel_index`, but not the total number of data-parallel replicas or the process-local DP rank. Consequently, `OffloadingParallelConfig` does not contain enough information to describe the DP topology. Add `data_parallel_size` and `data_parallel_rank_local` to `OffloadingParallelConfig` and populate them from `ParallelConfig`. `data_parallel_rank_local` remains `None` when it is not set, such as outside SPMD mode. ## Changes - Add `OffloadingParallelConfig.data_parallel_size`. - Add `OffloadingParallelConfig.data_parallel_rank_local`. - Propagate both values from `build_offloading_config()`. - Update test configurations that construct `OffloadingParallelConfig` directly. - Extend the DP propagation test to verify: - `data_parallel_index = 2` - `data_parallel_size = 4` - `data_parallel_rank_local = 1` This PR only exposes existing configuration values to offloading backends. It does not change DP initialization, offloading policy, or data movement. ## Duplicate-work check Open-PR searches for `data_parallel_size offloading`, `data_parallel_rank_local offloading`, and `KV offload data parallel` found no separate PR implementing this change. The only exact match was this PR. ## Test plan ```bash .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_config.py \\ -q \\ --confcutdir=tests/v1/kv_connector/unit/offloading_connector .venv/bin/python -m pytest \\ tests/v1/kv_offload/test_factory.py \\ tests/v1/kv_offload/test_file_mapper.py \\ -q \\ --confcutdir=tests/v1/kv_offload .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_worker.py \\ -q \\ --confcutdir=tests/v1/kv_connector/unit/offloading_connector \\ -k 'prepare_store_kv or handle_preemptions_non_writer or start_kv_transfers_non_writer or accepts_plugin_spec_default_layout' .venv/bin/python -m pytest \\ tests/v1/kv_offload/tiering/test_fs_tier.py::test_factory_forwards_locality_to_fs_tier \\ tests/v1/kv_offload/tiering/test_obj_tier.py::test_invalid_locality_raises_at_construction \\ tests/v1/kv_offload/tiering/test_obj_tier.py::test_obj_tier_replicated_layout_collapses_mapper_identity \\ -q \\ --confcutdir=tests/v1/kv_offload .venv/bin/pre-commit run ruff-format --files .venv/bin/pre-commit run ruff-check --files git diff --check ``` Results: - Offloading configuration tests: 42 passed - Factory and file-mapper tests: 55 passed - Worker tests: 6 passed, 2 deselected - Tiering tests: 4 passed - Ruff format and check: passed - `git diff --check`: passed", "url": "https://github.com/vllm-project/vllm/pull/51879", "createdAt": "2026-08-11T21:06:22Z", "updatedAt": "2026-08-13T03:27:27Z", "timestamp": "2026-08-13T03:27:27Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "kv-connector" ], "author": "ziqifan617", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51882", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Remove NIXL reinstall step", "text": "## Purpose Remove the NIXL wheel reinstall step. It is not longer needed (NIXL fixed the dependency issues upstream) and it is not working correctly since it does not use the version pin. Currently it causes nixl-cu13 to float to 1.3.2 while nixl/nixl-cu12 stays on the kv_connectors.txt pin (nixl == 1.3.1). ## Test Plan Verified by simulating the Dockerfile install path in a clean venv. ## Test Result **Old path** (`nixl==1.3.1` then unpinned `--force-reinstall --no-deps nixl-cu13`): ``` nixl 1.3.1 nixl-cu12 1.3.1 nixl-cu13 1.3.2 ← mismatch (same bug as Dynamo nightly) ``` **With fix** (pin only, no force-reinstall): ``` nixl 1.3.1 nixl-cu12 1.3.1 nixl-cu13 1.3.1 PASS: all nixl* == 1.3.1 ``` Removing that line is enough; the `kv_connectors.txt` pin keeps all three aligned. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51882", "createdAt": "2026-08-11T22:22:49Z", "updatedAt": "2026-08-13T07:45:31Z", "timestamp": "2026-08-13T07:45:31Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "documentation", "ready", "ci/build", "kv-connector" ], "author": "ovidiusm", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51886", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KVConnector] Add retention interval to OffloadingConnector", "text": "## Purpose Add retention interval support to `OffloadingConnector`. The Mooncake KV connector supports retention interval support via the `envs.VLLM_PREFIX_CACHE_RETENTION_INTERVAL` env variable. It is used to avoid storing/loading blocks that are outside of the sliding window of the attention mechanism. This is implemented via block masks. The implementation in `OffloadingConnector` uses the same block mask mechanism to implement retention interval support. cc @varun-sundar-rabindranath Note: parts of the code generated with claude ## Test Plan - Run kv_offloading unit tests - Add new unit tests - run gptoss w/retention interval enabled. ## Test Result gpt-oss-20b results ``` Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-20b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":4294967296,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-20b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.315 │ 0.185 │ 36.5 │ 1284.8 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.305 │ 0.145 │ 33.4 │ 1434.2 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.325 │ 0.170 │ 25.2 │ 1840.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.325 │ 0.160 │ 29.8 │ 1577.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.345 │ 0.200 │ 22.3 │ 2138.7 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.320 │ 0.165 │ 28.1 │ 1678.6 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` gpt-oss-120b results ``` ● gpt-oss-120b — Retention Interval Eval Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-120b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-120b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.540 │ 0.145 │ 58.9 │ 725.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.540 │ 0.100 │ 51.7 │ 851.9 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.575 │ 0.085 │ 57.0 │ 769.3 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.555 │ 0.080 │ 50.9 │ 870.1 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.570 │ 0.095 │ 40.3 │ 1059.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.570 │ 0.080 │ 50.9 │ 851.3 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` DeepSeekV4 results showing reduction in memory stores for different intervals. ``` DeepSeek-V4-Flash (GSM8K, 200 questions, 8-shot, max_model_len=16384) Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve deepseek-ai/DeepSeek-V4-Flash \\ --enforce-eager \\ --max-model-len 16384 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --block-size 256 Where INTERVAL is unset (baseline), 512 (sparse), or 0 (most sparse). Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 8 \\ --host http://localhost \\ --port 8001 Accuracy: ┌──────────────────────────┬────────┬──────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.960 │ 48.5 │ 371.1 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.970 │ 53.0 │ 345.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ fresh │ 0.965 │ 54.3 │ 331.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ reload │ 0.955 │ 47.9 │ 380.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.970 │ 52.6 │ 343.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.960 │ 46.3 │ 389.6 │ └──────────────────────────┴────────┴──────────┴─────────────┴──────────┘ Prometheus Offload Metrics: ┌──────────────────────────┬─────────────────────────┬───────────────────────┐ │ Configuration │ Store Bytes │ Reduction vs Baseline │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Baseline (no interval) │ 4,610,304,000 (4.61 GB) │ — │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (interval=512) │ 192,430,080 (192 MB) │ 95.8% │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Most sparse (interval=0) │ 104,232,960 (104 MB) │ 97.7% │ └──────────────────────────┴─────────────────────────┴───────────────────────┘ ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51886", "createdAt": "2026-08-11T23:11:05Z", "updatedAt": "2026-08-13T16:42:21Z", "timestamp": "2026-08-13T16:42:21Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "bnellnm", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51896", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Reject oversized media before fully downloading it", "text": "# Reject oversized media before fully downloading it ## What this fixes Several media paths checked size only after a remote response or inline audio value had already been turned into one complete `bytes` object. Batch speech requests could also download several oversized rows concurrently before applying their per-row limit. For example, with a 1 MiB limit, a remote server can return a 2 MiB body. Before this change, `async_get_bytes()` called the response's full-body read and allocated all 2 MiB before any limit ran. After this change, vLLM checks `Content-Length` when present and otherwise reads bounded chunks, stopping as soon as the cumulative total exceeds 1 MiB. Oversized inline base64 audio is rejected from its encoded length before decoding. ## Why it matters A client able to submit media URLs or batch audio could force the API or batch process to consume memory and download bandwidth chosen by the remote response, even though the request would later be rejected. Concurrent rows multiplied that cost and could make the process run out of memory. ## The change The three related sites are combined in commit `4cb3fe0`: | Site | What now happens | |---|---| | PTP-VLLM-037 — chat audio | URL downloads stream under the audio byte limit; oversized base64 is rejected before decoding. | | PTP-VLLM-038 — batch speech | The audio byte limit is passed into each URL download before concurrent row processing. | | PTP-VLLM-094 — shared remote media | `VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB` sets a shared default ceiling; sync and async readers check headers and streamed bytes before caching or decoding. | PTP-VLLM-041 is intentionally not included. Current `main` already has stronger per-modality item validation in the Rust frontend, and applying the older patch would weaken that behavior. ## How it was tested PTP-VLLM-037 changed from 4 failures without the fix to all 4 passing. PTP-VLLM-038 changed from 3 failures and 1 pass to all 4 passing. The PTP-VLLM-094 module could not collect on the old tree because the new shared limit did not exist; with the change, all 4 tests passed. The cases cover a declared oversized `Content-Length`, chunked responses without that header, an exact-limit control, early stop after the first over-limit chunk, base64 pre-checking, and batch handler code not being reached for an oversized row. ## Reference Advisory: GHSA-p6g9-7v3x-m8mv Track B: this is a public PR against `vllm-project/vllm@main`. ## Credit - @KernelClint (Clinton Thomas) and @dhalf (Lucas Bourtoule) - Patch the Planet (Trail of Bits + OpenAI collaboration); discovered using GPT-5.5-Cyber All commits include DCO `Signed-off-by` trailers.", "url": "https://github.com/vllm-project/vllm/pull/51896", "createdAt": "2026-08-11T23:55:04Z", "updatedAt": "2026-08-13T03:40:50Z", "timestamp": "2026-08-13T03:40:50Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "frontend", "multi-modality" ], "author": "KernelClint", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51901", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build] Add warning for unsupported global PTX architecture requests in...", "text": "... CMake configuration. ## Purpose Implements one item from #9129: Warn that PTX builds are not currently supported (post [CI/Build] Per file CUDA Archs (improve wheel size and dev build times) #8845), currently if there is a +PTX in TORCH_CUDA_ARCH_LIST this will be ignored. We should warn when this is the case Users can request PTX through `TORCH_CUDA_ARCH_LIST` values such as `8.0+PTX`. vLLM strips the Torch-provided global `-gencode` flags and rebuilds per-source gencode flags, so the global PTX request is not preserved. This PR surfaces that behavior during CMake configuration instead of silently dropping the request. This branch builds on PR #42375 by: 1. Renaming `clear_cuda_arches` to `clear_cuda_gencode_flags` to clarify that it clears `-gencode` flags from `CMAKE_CUDA_FLAGS`, not the CUDA target architectures list itself. 2. Adding documentation to [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) to explain the CUDA architecture and PTX flags normalization behavior. ## Changes - **Refactor**: Renamed `clear_cuda_arches()` macro to `clear_cuda_gencode_flags()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) for clarity. - **Feat**: Added `warn_if_ptx_arch_requested()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and called it in [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) to warn users when PTX code generation is requested via global CUDA architecture flags. - **Tests**: Added pytest unit test in [tests/test_cmake_utils.py](file:///home/swidanagama/vllm/tests/test_cmake_utils.py) for `clear_cuda_gencode_flags()`. - **Docs**: Added a note in [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) explaining CUDA architecture and PTX flags normalization. ## Test Plan Tested behavior by: 1. Running pytest unit tests in [tests/test_cmake_utils.py](tests/test_cmake_utils.py): ```bash .venv/bin/python -m pytest tests/test_cmake_utils.py -v ``` 2. Running a local install command to verify that the build environment compiles and registers: ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . ``` ## Test Result ### 1. CMake Utility Pytest Verification Output ```text $ .venv/bin/python -m pytest tests/test_cmake_utils.py -v ============================= test session starts ============================== platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/swidanagama/vllm/.venv/bin/python cachedir: .pytest_cache rootdir: /home/swidanagama/vllm configfile: pyproject.toml plugins: asyncio-1.4.0, anyio-4.14.2 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 3 items tests/test_cmake_utils.py::test_exact_family_arch_precedes_generic_family_fallback PASSED [ 33%] tests/test_cmake_utils.py::test_extract_archs_prefers_sass_target_over_corrupted_virtual_arch PASSED [ 66%] tests/test_cmake_utils.py::test_clear_cuda_gencode_flags PASSED [100%] ======================== 3 passed in 0.95s ======================== ``` ### 2. Local Install Result ```text Uninstalled 4 packages in 19ms Installed 164 packages in 939ms ... + vllm==0.26.1rc1.dev652+g066f84253.precompiled (from file:///home/swidanagama/vllm) ``` ## AI assistance disclosure This PR description and code changes were prepared with AI assistance. The submitting contributor has reviewed every changed line and run the relevant validation tests.", "url": "https://github.com/vllm-project/vllm/pull/51901", "createdAt": "2026-08-12T00:11:47Z", "updatedAt": "2026-08-13T14:40:47Z", "timestamp": "2026-08-13T14:40:47Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready", "ci/build", "nvidia" ], "author": "shanewidanagama", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51906", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add routed-experts prompt offset", "text": "## Summary - Add `routed_experts_prompt_start` to OpenAI chat/completion requests and `SamplingParams`, allowing clients to omit an already-known prompt prefix from returned R3. - Centralize NumPy-to-base64 serialization used by existing R3 responses and document the `int32` expert-ID representation. - Keep OpenAI streaming behavior unchanged: R3 remains supported only on existing non-streaming responses. ## Why this is not duplicate work No open PR implements the R3 prompt offset or shared R3 response serialization as a frontend change. #49555 returns sparse-attention indexer top-k, which is a distinct feature; this PR only factors shared R3-facing plumbing. ## Validation - `uvx pre-commit run` — passed. - `PYTHONPATH=. /home/aoshen/vllm/.venv/bin/python -m pytest tests/test_sampling_params.py tests/entrypoints/openai/test_stop_token_ids.py tests/utils_/test_serial_utils.py -q` — 49 passed. - R3 OpenAI integration coverage remains in `tests/entrypoints/openai/test_return_routed_experts.py`. Model evaluation was not run: this changes request/response transport and validation only; it does not change routing computation or generated model outputs. AI assistance was used to implement and validate this PR.", "url": "https://github.com/vllm-project/vllm/pull/51906", "createdAt": "2026-08-12T02:00:29Z", "updatedAt": "2026-08-13T15:09:00Z", "timestamp": "2026-08-13T15:09:00Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "frontend", "ready" ], "author": "aoshen02", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51909", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Cached blocks never hit eviction first", "text": "## Purpose Most cached blocks with block_hash that never hit. Now they are appended to the free list mixed with fewer hit blocks. We should differentiate the both cases because of the non-hit blocks amount that is huge and far greater than hit blocks. So we should append the non-hit blocks first and then append the hit blocks so that non-hit blocks are allocated first in free list and keep alive for hit blocks. ## Test Plan 1. default testcases python -m pytest tests/v1/core/test_kv_cache_utils.py -v python -m pytest tests/v1/core/test_prefix_caching.py -v python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py -v 2. under stress test watching the amount of blocks which are hit ever and reallocated again in free list VLLM_CACHE_ROOT=/data/hf/vllm/ vllm bench throughput --model /data/hf/Qwen3.6-35B-A3B --input-len 1024 --output-len 1024 --dtype float16 --num-prompts 2000 --tensor-parallel-size 4 --enable-prefix-caching ## Test Result 1. default testcases python -m pytest tests/v1/core/test_kv_cache_utils.py -v 76 passed, 14 warnings python -m pytest tests/v1/core/test_prefix_caching.py -v 89 passed, 14 warnings python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py -v 9 passed, 14 warnings 2. under stress test watching the amount of blocks which are hit ever and reallocated again in free list ...... INFO 08-13 09:03:54 [loggers.py:310] Engine 000: Avg prompt throughput: 7629.3 tokens/s, Avg generation throughput: 999.6 tokens/s, Running: 171 reqs, Waiting: 84 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 0.0% ...... **before : 479 blocks after : 406 blocks** the reallocated cached hit blocks decrease **15.24%** ---", "url": "https://github.com/vllm-project/vllm/pull/51909", "createdAt": "2026-08-12T02:15:54Z", "updatedAt": "2026-08-13T09:19:10Z", "timestamp": "2026-08-13T09:19:10Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "shanrow-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51910", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization][CT] fix the mxfp4 inference for MiniMax-M3 with CT format.", "text": "## Description Running **MiniMax-M3** MXFP4 checkpoints quantized with the **compressed-tensors (CT)** `mixed-precision` format currently fails at model load with a `KeyError` on the vision tower. In these checkpoints the **vision module is not quantized** (it is listed in the CT `ignore` list and stored as plain `.weight`), but vLLM wrongly treats the vision layers as quantized and then fails to find the packed weights. This PR contains two small, targeted fixes that make MXFP4 CT inference work, without changing behavior for the bf16 / mxfp8 paths. ## Root cause & fix ### 1. `vllm/model_executor/models/minimax_m3/nvidia/model.py` — vision `hf_to_vllm_mapper` The CT config's `ignore` list is now transformed through the model's `hf_to_vllm_mapper` by `CompressedTensorsConfig.apply_vllm_mapper()` (it calls `_map_name` on every `self.ignore` entry so ignore targets line up with vLLM's internal layer names). The catch: **`ignore` entries are module names without a trailing dot** (e.g. `...encoder.layers.0.mlp.fc1`), whereas the existing substitutions had trailing dots (`.mlp.fc1.` → `.fc1.`). `_map_name` is a substring replace, so: - `.mlp.fc1.` does **not** match the dot-less ignore name `...mlp.fc1` → the ignore target is left untransformed → it never matches vLLM's internal prefix `...fc1` → the (unquantized) vision layer is built as a quantized layer → `KeyError: '...fc1.weight'` at load. The fix drops the trailing dots so the mapping matches both module names and weight names: ```python orig_to_new_substr={ - \".mlp.fc1.\": \".fc1.\", - \".mlp.fc2.\": \".fc2.\", + \".mlp.fc1\": \".fc1\", + \".mlp.fc2\": \".fc2\", }, ``` **Why this is safe for weight loading (bf16/mxfp8 unaffected):** for actual weight names (`...mlp.fc1.weight`), both the old and new substitutions produce the identical result `...fc1.weight`. The change only *additionally* fixes the dot-less ignore-module-name case. It is a strict superset of the previous behavior. *(Evidence that vision is genuinely unquantized in the checkpoint: in the MXFP4 model index, the vision tower has 0 `weight_packed` tensors — all packed weights live under `language_model...block_sparse_moe.experts` — and 258 vision entries appear in the CT `ignore` list.)* ### 2. `vllm/compilation/passes/fusion/allreduce_rms_fusion.py` — flashinfer `weight_bias` version compat MiniMax-M3 uses a GemmaRMSNorm that scales by `(1 + weight)`, carried by the fused all-reduce kernel's `weight_bias=1.0`. The `weight_bias` argument to `flashinfer.comm.trtllm_allreduce_fusion` was added in **flashinfer 0.6.12**; older flashinfer (e.g. 0.6.11) does not accept it, so unconditionally passing it raises `TypeError`. The fix detects support once at import via `inspect.signature` (guarded by `try/except (TypeError, ValueError)` for signature-less C callables) and: - passes `weight_bias=...` when supported, or - folds the bias into `rms_gamma` (`normed * (gamma + bias)`, mathematically equivalent) with a `warning_once` when not. This path is only exercised under TP + flashinfer all-reduce/RMSNorm fusion and is **quantization-agnostic**. It can be removed once the minimum flashinfer version is ≥ 0.6.12. **Prior art:** the same \"probe the installed flashinfer signature and only pass supported kwargs\" approach was used in SGLang [PR #26197](https://github.com/sgl-project/sglang/pull/26197) to keep the Blackwell all-reduce-fusion path enabled across flashinfer API variants.", "url": "https://github.com/vllm-project/vllm/pull/51910", "createdAt": "2026-08-12T02:20:51Z", "updatedAt": "2026-08-13T07:39:15Z", "timestamp": "2026-08-13T07:39:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization" ], "author": "lkk12014402", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51915", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness", "text": "## Purpose Enables GLM-5.2 (`GlmMoeDsaForCausalLM`) end-to-end on `vllm/models/deepseek_v32/amd/` for gfx942/gfx950. Routing is opt-in via `--model-class-overrides`; the registry entry is unchanged, so the default path for GLM-5.2 and DeepSeek-V3.2 is untouched. The following issues were also fixed as a result of this overall enablement since it surfaced dormant bugs from a prior deepseek_v32 porting work: - **`launch_pdl` forwarded to a Triton runtime that doesn't accept it** — the shared `common/kernels.py` passed this CUDA-only kwarg unconditionally; ROCm's Triton rejects it by name even when `False`, so the engine died at startup with `KeyError`. Now passed only when the platform actually supports PDL. - **fp4 bmm invoked with mismatched positional args** — the output tensor landed in the kernel's `dtype` slot (`Expected 'Optional[int]' for 'dtype' but found Tensor`). Routed through the existing `rocm_aiter_ops.batched_gemm_a16wfp4` wrapper rather than calling aiter directly, so the argument contract lives in one place. - **fp4 MLA bmm silently wrong for non-power-of-two K** — `batched_gemm_a16wfp4` only supports power-of-two K, and returns wrong values without erroring otherwise. GLM-5.2's `qk_nope_head_dim=192` lands in the unsupported set; DeepSeek-V3.2's 128 does not. Guarded on both `qk_nope_head_dim` and `kv_lora_rank` to fall back to the fp8 bmm, with a warning. `W_K`/`W_K_scale` are also made contiguous after the transpose, which otherwise produced NaN. - **Indexer-K cache written in the plain layout while its reader expects the shuffled one** — affects DeepSeek-V3.2 identically, not just GLM-5.2. The AMD path writes this cache from the shared `fused_norm_rope`, which only implemented `[n_blocks, blk, head_dim]`, while the reader selects `SHUFFLE` above block size 1: the write kept storing row-major, but the reader expected shuffled, so the indexer scored garbage K and output degenerated once a sequence exceeded `index_topk=2048`, with no error raised. Refactored so the shared kernel stays platform-agnostic — it writes whichever layout the cache reports via a new `uses_shuffled_layout` property — and the AMD-specific shuffled write (`[n_blocks, blk/16, head_dim/16, 16, 16]`, used above block size 1) lives in the AMD backend. The base class returns `False`, so CUDA is unchanged. - **Sparse-MLA block sizes narrowed instead of extended** — the ported backends replaced the inherited `get_supported_kernel_block_sizes()` with `[16, 32]`, which made `select_common_block_size` silently downgrade a requested 64 to 32 via its largest-divisor fallback. Now concatenated with the base list. - **Kernel tests gated on `is_cuda()` and running against dims that matched neither model** — the tests were skipped entirely on CDNA. The gate is now `supports_fp8()`, so the suite runs on gfx950. The hardcoded attention dims were written as if one set served both GLM-5.2 and DeepSeek-V3.2, which it did not; they are now taken from each model's published config and parametrized over both. ## Test Plan Server: VLLM_ROCM_USE_AITER=1 \\ HF_HUB_OFFLINE=1 SAFETENSORS_FAST_GPU=1 \\ vllm serve /path-to-your/GLM-5.2-MXFP4 \\ --model-class-overrides '{\"GlmMoeDsaForCausalLM\": \"vllm.models.deepseek_v32:DeepseekV32ForCausalLM\"}' \\ --tensor-parallel-size 8 \\ --kv-cache-dtype auto \\ --max-model-len 65536 \\ --no-enable-prefix-caching \\ --trust-remote-code \\ --port 8000 Variants: --kv-cache-dtype fp8 for the fp8 KV row; --block-size {16,64} for the block-size matrix. Drop --model-class-overrides to run the legacy deepseek_v2 path for comparison. Accuracy: lm_eval --model local-completions \\ --model_args \"model=/models/GLM-5.2-MXFP4,base_url=http://localhost:8000/v1/completions,num_concurrent=32,tokenized_requests=False,max_retries=3,timeout=3600\" \\ --tasks gsm8k --num_fewshot 5 --gen_kwargs max_gen_toks=2048 --limit 200 Throughput: vllm bench serve --backend vllm --model /shareddata/models/GLM-5.2-MXFP4 \\ --dataset-name random --random-input-len 4096 --random-output-len 1024 \\ --num-prompts 200 --max-concurrency 32 --ignore-eos --port 8000 \\ --save-result --result-filename glm52_v32amd_bench.json Kernel unit tests: pytest tests/kernels/test_fused_deepseek_v32_norm_rope.py -v ## Test Result **gsm8k** — GLM-5.2 200 samples, 5-shot, `exact_match`, `deepseek_v32` AMD path | KV cache | Before (flexible / strict) | After (flexible / strict) | |-------------|----------------------------|------------------------------| | auto (bf16) | 0.0000 / 0.0000 | **0.9400 / 0.9400** ± 0.0168 | | fp8 | 0.0000 / 0.0000 \\* | **0.9350 / 0.9350** ± 0.0175 | \\* Not separately measured. fp8 KV goes through the same absorbed-weight bmm where the bug was, so it fails identically; only the bf16 run was evaluated pre-fix.", "url": "https://github.com/vllm-project/vllm/pull/51915", "createdAt": "2026-08-12T02:59:15Z", "updatedAt": "2026-08-13T17:59:08Z", "timestamp": "2026-08-13T17:59:08Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "bug", "rocm", "needs-rebase", "deepseek" ], "author": "jhu960213", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51924", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Refine FlashInfer one-sided All2All integration", "text": "## Purpose Refine the FlashInfer NVLink one-sided All2All integration for DeepSeek Blockwise FP8 MoE and sequence parallelism. - Describe one-sided activation payloads explicitly in bytes for BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8. - Dispatch E4M3 activations with FP32 1x128 scales and feed the received layout directly to the FlashInfer TRT-LLM DeepSeekFp8/BlockMajorK kernel. - Validate activation and scale shapes before converting scales from `[M, K/128]` to the TRT-LLM `[K/128, M]` layout. - Enable MoE sequence parallelism for `flashinfer_nvlink_one_sided`, avoiding routing and compute on TP-replicated post-attention tokens. - Document and exercise DeepSeek Blockwise FP8 support in the existing MoE and MNNVL test matrices. No kernel-warmup changes are included because current upstream `main` already provides generalized distributed FlashInfer autotuning and persistent-cache synchronization. ### Duplicate-work check No issue number was supplied, so an issue-specific lookup was not applicable. I checked open PRs with: ```bash gh pr list --repo vllm-project/vllm --state open --search 'FlashInfer one-sided All2All' gh pr list --repo vllm-project/vllm --state open --search 'DeepSeek Blockwise FP8 FlashInfer' gh pr list --repo vllm-project/vllm --state open --search 'sequence parallel FlashInfer one sided' ``` The related open PRs are distinct: - #47733 handles per-tensor FP8 scaling, invalid expert IDs, and communicator cleanup; it does not add E4M3 activation dispatch with FP32 1x128 scales, TRT-LLM BlockMajorK integration, or sequence parallelism. - #42034 only changes the one-sided padding sentinel for local expert maps. - #42133 handles delayed MXFP8 scale swizzling for FlashInfer CUTLASS. ## Test Plan - Run all pre-commit hooks on the changed files. - Compile every changed Python file. - Check the patch for whitespace errors. - Validate DeepSeek Blockwise FP8 payload sizing and TRT-LLM scale layout with focused assertions. - Run a reproducible model evaluation on a public checkpoint. ## Test Result Passed all configured pre-commit hooks on the changed files, including ruff check/format, typos, markdownlint, mypy, SPDX, forbidden-import, configuration, and API checks: ```bash .venv/bin/pre-commit run --files \\ docs/design/moe_kernel_features.md \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py ``` Passed: ```bash .venv/bin/python -m py_compile \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py git diff --check origin/main...HEAD ``` A focused Python assertion script covering BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8 dispatch byte layouts plus the TRT-LLM `[M, K/128]` to `[K/128, M]` scale conversion passed. The repository-wide collection command below could not run in this CPU development environment because the installed `xgrammar` lacks `normalize_tool_choice` and the CUDA flash-attention extension is unavailable: ```bash .venv/bin/python -m pytest -q --collect-only \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py ``` ### Model evaluation Full 1,319-example GSM8K on public `Qwen/Qwen3.5-35B-A3B-FP8`, using 8 B300 GPUs with TP2 x DP4 / EP8. The runtime used this PR plus the Qwen3Next SP correctness fix from #50685; that fix is not part of this PR. Common settings: `flashinfer_trtllm`, FlashInfer autotune enabled, `max_num_batched_tokens=4096`, `max_num_seqs=64`, 5-shot greedy decoding, `max_gen_toks=16384`, and 256 concurrent requests. Only the All2All backend changed. The task used `\"\\nQuestion:\"` instead of stock lm-eval's broad `\"Question:\"` stop to avoid terminating inside model reasoning. | All-to-all backend | Flexible exact match | Strict exact match | |---|---:|---:| | `allgather_reducescatter` | 95.75% (1263/1319) | 92.49% (1220/1319) | | `flashinfer_nvlink_one_sided` | 95.00% (1253/1319) | 90.98% (1200/1319) | Both runs completed without retries or engine errors. The flexible-score gap is 10 samples and is not statistically significant (`p=0.143`). Per-sample review and a repeated AG/RS run found trajectory and formatting variation, but no systematic one-sided correctness failure. ## AI assistance AI assistance was used for code exploration, cherry-pick conflict resolution, compatibility review, validation, and PR drafting. The human submitter must review every changed line, understand the change end-to-end, and be able to defend it before merging. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] The current test results and model-evaluation status. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/51924", "createdAt": "2026-08-12T04:41:12Z", "updatedAt": "2026-08-13T14:37:16Z", "timestamp": "2026-08-13T14:37:16Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "documentation", "ready", "nvidia", "verified" ], "author": "bobboli", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51927", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Refactor] Use common sp utils for Qwen3.5 MoE", "text": "## Summary This PR enables the common sequence-parallel path for Qwen3.5 MoE, keeps sequence-sharded tensors across attention/MoE boundaries, and restores full-token outputs only where required. It also preserves the Qwen3.5 MTP path by normalizing the local shard before gathering the final draft-model output. ## End-to-end flow ```text Qwen3_5ForConditionalGeneration.forward -> Qwen3_5Model.forward -> Qwen3NextDecoderLayer.forward -> Qwen3NextAttention.forward -> sp_shard / sp_reduce_scatter -> Qwen3NextSparseMoeBlock.forward -> sp_all_gather (only when the MoE input must be restored) -> _restore_sequence_parallel_outputs -> one packed sp_all_gather for final and auxiliary states MTP: Qwen3_5MTP.forward -> decoder layer -> RMSNorm on the local sequence shard -> sp_all_gather -> trim sequence padding ``` ## Results Model: `Qwen/Qwen3.5-122B-A10B-FP8`, TP=2, DP=2, EP enabled. ### Non-MTP: main vs. this PR The throughput numbers below use the steady-state runs (main's final run and the mean of this PR's final three runs). | Branch | Output throughput | Total token throughput | Mean TTFT | Mean TPOT | GSM8K 5-shot flexible / strict | |---|---:|---:|---:|---:|---:| | main | 1,853.15 tok/s | 20,384.67 tok/s | 1,013.50 ms | 13.68 ms | Not separately rerun | | this PR | 1,843.41 tok/s | 20,277.53 tok/s | 1,117.85 ms | 13.55 ms | 86.66% / 85.14% | Output throughput differs by **-0.53%**, while mean TPOT improves by **0.94%**. ```bash vllm bench serve \\ --model Qwen/Qwen3.5-122B-A10B-FP8 \\ --host localhost --port 8000 \\ --dataset-name random \\ --random-input-len 10240 \\ --random-output-len 1024 \\ --num-prompts 100 \\ --max-concurrency 32 \\ --temperature 0 --ignore-eos \\ --num-warmups 4 lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-122B-A10B-FP8,base_url=http://127.0.0.1:8000/v1/completions,tokenizer=Qwen/Qwen3.5-122B-A10B-FP8,num_concurrent=32,max_retries=3,tokenized_requests=False,timeout=600 \\ --tasks gsm8k --num_fewshot 5 --batch_size 1 \\ --gen_kwargs temperature=0,max_gen_toks=256 ``` ### MTP: this PR MTP was tested with two speculative tokens. A representative shape-stress run used 127 input tokens, 64 output tokens, 32 prompts, and concurrency 32. | Branch | Output throughput | Mean TTFT | Mean TPOT | MTP acceptance | GSM8K 5-shot flexible / strict | |---|---:|---:|---:|---:|---:| | this PR + MTP | 1,421.63 tok/s | 500.94 ms | 9.20 ms | 78.66% | 86.05% / 84.38% | All 32 requests completed successfully. Additional input lengths `1`, `3`, and `17` were also tested to cover small and non-divisible sequence shapes. ```bash vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\ --tensor-parallel-size 2 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --max-model-len 32768 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2,\"max_model_len\":32768}' vllm bench serve \\ --model Qwen/Qwen3.5-122B-A10B-FP8 \\ --host localhost --port 8000 \\ --dataset-name random \\ --random-input-len 127 \\ --random-output-len 64 \\ --num-prompts 32 \\ --max-concurrency 32 \\ --temperature 0 --ignore-eos lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-122B-A10B-FP8,base_url=http://127.0.0.1:8000/v1/completions,tokenizer=Qwen/Qwen3.5-122B-A10B-FP8,num_concurrent=32,max_retries=3,tokenized_requests=False,timeout=600 \\ --tasks gsm8k --num_fewshot 5 --batch_size 1 \\ --gen_kwargs temperature=0,max_gen_toks=256 ``` non-MTP: (main) ``` local-completions ({'model': 'Qwen/Qwen3.5-122B-A10B-FP8', 'base_url': 'http://127.0.0.1:8000/v1/completions', 'tokenizer': 'Qwen/Qwen3.5-122B-A10B-FP8', 'num_concurrent': 32, 'max_retries': 3, 'tokenized_requests': False, 'timeout': 600}), gen_kwargs: ({'temperature': 0, 'max_gen_toks': 256}), limit: None, num_fewshot: 5, batch_size: 1 |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8650|± |0.0094| | | |strict-match | 5|exact_match|↑ |0.8423|± |0.0100| ``` (This PR) ```text |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8666|± |0.0094| | | |strict-match | 5|exact_match|↑ |0.8514|± |0.0098| ``` MTP: ```text |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8605|± |0.0095| | | |strict-match | 5|exact_match|↑ |0.8438|± |0.0100| ```", "url": "https://github.com/vllm-project/vllm/pull/51927", "createdAt": "2026-08-12T05:20:36Z", "updatedAt": "2026-08-13T05:45:27Z", "timestamp": "2026-08-13T05:45:27Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen" ], "author": "gcanlin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51931", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Use VLLMValidationError in pooling input validation", "text": "## Purpose Part of #48227. Like #51753, this is an independent file-level Step 5 migration. Migrate four caller-caused validation errors in `vllm/entrypoints/pooling/base/io_processor.py` from raw `ValueError` to `VLLMValidationError`: - conflicting offline pooling tasks - untrusted request-level chat templates - mismatched prompt and pooling parameter counts - mismatched prompt and LoRA request counts This preserves the existing validation behavior and error messages. The `parameter` and `value` fields remain unset because the shared helpers serve both online and offline entrypoints whose public argument names differ. The invalid request type, unsupported internal render parameters, and priority-sequence mismatch remain raw `ValueError`s because they are internal guards or are not reachable through current public pooling paths. The existing `LLM.encode()` integration assertion is updated to expect the new semantic exception type. Online serving continues to return HTTP 400, while offline callers now receive `VLLMValidationError` instead of `ValueError`. ## Duplicate-work check I searched open PRs for #48227, `PoolingIOProcessor`, `VLLMValidationError`, and the affected validation messages. I found no open PR migrating these errors. PR #41890 touches the same file for an unrelated import refactor and does not overlap semantically. AI assistance disclosure: Codex assisted with implementation, test generation, and review. I reviewed and understand all changed lines and validated the behavior locally. ## Test Plan ```bash .venv/bin/python -m pytest tests/entrypoints/pooling/test_io_processor.py tests/entrypoints/pooling/scoring/test_utils.py -q .venv/bin/python -m pytest tests/entrypoints/pooling/basic/test_encode.py --collect-only -q env PRE_COMMIT_HOME=/private/tmp/vllm-pre-commit-cache .venv/bin/pre-commit run --from-ref origin/main --to-ref HEAD ``` ## Test Result ```text 8 passed in 0.72s 2 integration tests collected successfully All applicable pre-commit hooks passed. ``` The model-backed `tests/entrypoints/pooling/basic/test_encode.py` suite was not executed locally because it starts a vLLM model. Its updated assertion is covered by CI. Model evaluation is not applicable because this change does not affect inference output.", "url": "https://github.com/vllm-project/vllm/pull/51931", "createdAt": "2026-08-12T06:11:43Z", "updatedAt": "2026-08-13T06:13:25Z", "timestamp": "2026-08-13T06:13:25Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "frontend", "ready", "verified" ], "author": "frank-suwen", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Move context_lens compute into prefill path for Minimax", "text": "## Purpose Move `compute_num_computed_tokens()` from the top of the method into the `if num_prefills > 0` branch because it's only used in prefill. Since context_lens tensor is not used in decode path, this change avoids the tensor computed and discarded in decode path. Similar as #51913 ## Profiling Main: \"Screenshot PR: \"Screenshot Main: `compute_num_computed_tokens` called in MiniMaxM3IndexerTritonMetadataBuilder.build() in decode path, takes extra 74µs. PR: `compute_num_computed_tokens` not called. ## Benchmark ``` vllm serve MiniMaxAI/MiniMax-M3 \\ --tensor-parallel-size 8 \\ --max-num-seqs 16 \\ --block-size 128 \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model MiniMaxAI/MiniMax-M3 \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 119.30 Total input tokens: 15421 Total generated tokens: 18000 Request throughput (req/s): 0.50 Output token throughput (tok/s): 150.89 Peak output token throughput (tok/s): 158.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 280.15 ---------------Time to First Token---------------- Mean TTFT (ms): 85.10 Median TTFT (ms): 114.17 P99 TTFT (ms): 133.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.36 Median TPOT (ms): 6.36 P99 TPOT (ms): 6.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 6.36 Median ITL (ms): 6.36 P99 ITL (ms): 6.72 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 300.81 Total input tokens: 216233 Total generated tokens: 288000 Request throughput (req/s): 3.19 Output token throughput (tok/s): 957.42 Peak output token throughput (tok/s): 1072.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 1676.25 ---------------Time to First Token---------------- Mean TTFT (ms): 303.93 Median TTFT (ms): 293.91 P99 TTFT (ms): 401.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.75 Median TPOT (ms): 15.72 P99 TPOT (ms): 16.54 ---------------Inter-token Latency---------------- Mean ITL (ms): 15.75 Median ITL (ms): 15.39 P99 ITL (ms): 16.67 ================================================== ``` PR: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 119.22 Total input tokens: 15421 Total generated tokens: 18000 Request throughput (req/s): 0.50 Output token throughput (tok/s): 150.98 Peak output token throughput (tok/s): 158.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 280.32 ---------------Time to First Token---------------- Mean TTFT (ms): 84.08 Median TTFT (ms): 113.42 P99 TTFT (ms): 130.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.36 Median TPOT (ms): 6.36 P99 TPOT (ms): 6.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 6.36 Median ITL (ms): 6.36 P99 ITL (ms): 6.74 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 296.84 Total input tokens: 216233 Total generated tokens: 288000 Request throughput (req/s): 3.23 Output token throughput (tok/s): 970.22 Peak output token throughput (tok/s): 1057.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 1698.67 ---------------Time to First Token---------------- Mean TTFT (ms): 247.74 Median TTFT (ms): 260.92 P99 TTFT (ms): 285.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.71 Median TPOT (ms): 15.72 P99 TPOT (ms): 16.26 ---------------Inter-token Latency---------------- Mean ITL (ms): 15.71 Median ITL (ms): 15.43 P99 ITL (ms): 16.74 ================================================== ``` ## Accuracy Testing ``` python3 -m lm_eval --model local-completions \\ --model_args model=MiniMaxAI/MiniMax-M3,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9121|± |0.0078| | | |strict-match | 5|exact_match|↑ |0.9113|± |0.0078| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9121|± |0.0078| | | |strict-match | 5|exact_match|↑ |0.9121|± |0.0078| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51932", "createdAt": "2026-08-12T06:33:53Z", "updatedAt": "2026-08-12T22:38:43Z", "timestamp": "2026-08-12T22:38:43Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "xyang16", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51937", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Parser] Migrate Hermes tool parser to the new streaming Parser Engine", "text": "# [Parser] Migrate Hermes tool parser to the streaming Parser Engine ## Summary Migrates the Hermes tool parser from the legacy regex/diff implementation (~275 lines) to the declarative streaming Parser Engine, following the pattern already used by Qwen3, MiniMax M2, Gemma4, Kimi K2 and Inkling. - New `vllm/parser/hermes.py`: `HermesParser(ParserEngine)` + a declarative `ParserEngineConfig`. - `vllm/tool_parsers/hermes_tool_parser.py` becomes a thin bridge. The registry entry and the `Hermes2ProToolParser` class name are unchanged, so `--tool-call-parser hermes` is unaffected. - `structural_tag_model = \"hermes\"` is preserved, keeping guided decoding (and the `supports_required_and_named=False` it implies) identical. This follows the suggestion on #49535 to migrate tool parsers to the Parser Engine rather than patch the legacy implementations further. ### Why Hermes first, and not `hunyuan_a13b` / `xlam` Those two parsers emit a single undelimited JSON array (`[{...},{...}]`), where the boundary between one tool call and the next is a bare comma at the array's top nesting level — there is no terminal token. The engine's state machine only transitions on explicit terminal tokens; brace/bracket depth tracking in the engine exists only to decide which characters are safe to stream, not to drive a transition. Migrating those two needs new engine-level capability, not just a per-model config, so it's a separate design question rather than something to fold into this PR. Hermes uses the same per-call delimiter shape as the already-migrated models (``…`` repeated per call), which is proven engine capability, so it's independently actionable now. ## Root cause / design notes **Why an `arg_converter` is mandatory.** Hermes's tool-call body is a `{\"name\":..., \"arguments\":{...}}` wrapper, so the whole wrapper lands in `TOOL_ARGS`. `ParserEngine._compute_arg_delta` returns the raw delta and never calls `_fix_arg_types` when `arg_converter is None`, while non-streaming extraction always coerces via `_build_extracted_result`. Without a converter, streaming and non-streaming silently disagree on argument types. **Why a verbatim scanner rather than re-serialising.** The engine diffs successive converter outputs and requires each to extend the previous one (`startswith`); a violation silently drops argument deltas. Re-serialising through `json.dumps` changes whitespace and closes unterminated structures differently across ticks, so the only prefix-stable output is a verbatim substring of the input. `_hermes_arg_converter` therefore carves the `arguments` value out with a string/escape-aware scanner, mirroring `_inkling_arg_converter`, which has the same wrapper shape. The scanner is also what makes an argument literally named `\"name\"` safe — the bug class that started this work (#49535). **Why `token_id_terminals` is empty.** `StreamingParserEngine._process_lex_tokens` enters a strict token-id mode once any real token IDs have been seen, and refuses to accept a text-matched terminal that is listed in `token_id_terminals` unless it was also pre-lexed from an actual matching token ID. Hermes delimiters reliably arrive as literal text (`ParserEngine.adjust_request` forces `skip_special_tokens=False`), so token-ID pre-lexing buys nothing and would starve text matching for tokenizers where the delimiter is not a single true token. Inkling opts out the same way. **Non-streaming strictness is preserved explicitly.** The legacy parser required the *whole* tool-call body to parse as valid JSON and reported no tool calls at all otherwise. The engine's generic EOS path is best-effort and would happily emit a tool call from a truncated wrapper, so `HermesParser.extract_tool_calls_from_content` keeps the single-shot `json.loads` check. Streaming keeps the engine's normal best-effort behavior. **Longcat.** `LongcatFlashToolParser` subclassed `Hermes2ProToolParser` and customised the delimiters by assigning `tool_call_start_token` / `tool_call_end_token` / `tool_call_regex`, none of which the engine consults. Left as-is, the migration would have silently broken it (13 of its 17 tests). It now has its own `HermesParser` subclass overriding the delimiters as class attributes, which is why `TOOL_CALL_START`/`TOOL_CALL_END` are class attributes on `HermesParser` at all. ## Test-suite change, and why `tests/parser/test_streaming.py` used Hermes as its reference **non-engine** tool parser (a section titled \"Engine-based reasoning + non-engine tool parser\", asserting `engine_based_streaming is False`). This migration invalidates that premise by design. Those tests now use `Granite4ToolParser`, which is still non-engine, uses the identical ``/`` delimiters so `MODEL_OUTPUT` is unchanged, and has `supports_required_and_named=True` natively — which also removes the need for the `autouse` fixture that was force-setting that flag on Hermes. `test_parse_delta_finished_appends_remaining_args` still uses Hermes explicitly, since its subject is that parser's unstreamed-args behavior. No test was weakened or deleted; only the reference model changed. ## Out of scope (pre-existing, not introduced here) While validating the above I found that `tool_choice=\"required\"` crashes for **any** engine-based parser whose `supports_required_and_named` is `True`: `StreamState.advance()` passes only the latest delta for engine-based parsers, but `extract_required_tool_call_streaming()` expects the full accumulated text, so the final chunk reaches `partial_json_parser` as a bare `]` and raises `IndexError: pop from empty list`. This reproduces on the already-merged `KimiK2ToolParser` with no Hermes involvement, so it is not caused by this PR. It is normally unreachable because every engine-based parser sets `supports_required_and_named=False` via `structural_tag_model` + `VLLM_ENFORCE_STRICT_TOOL_CALLING` (default on), which Hermes continues to do here. Happy to file it separately. ## Not a duplicate - `gh pr list --repo vllm-project/vllm --search \"hermes parser engine\" --state all` and `--search \"migrate parser engine in:title is:open\"` — no PR migrates Hermes. In-flight migrations cover other models (Kimi K3 #50229, llama3_json/llama4_json #51577, Granite #49648, Olmo3 #48160, poolside_v1 #47167). - `vllm/parser/hermes.py` does not exist on `main`; the registry still points at the legacy `hermes_tool_parser` / `Hermes2ProToolParser`. - Three open PRs touch `vllm/tool_parsers/hermes_tool_parser.py` (#48864, #45310, #48353). This PR replaces that file wholesale, so it will conflict with them textually. Flagging explicitly so maintainers can sequence them — #48353 in particular changes tool-call boundary semantics, and if it is preferred first I am happy to rebase on top of it. - `gh pr list --search \"longcat\"` — no open PR touches `vllm/tool_parsers/longcat_tool_parser.py` (#45773 is the Rust frontend). ## Test plan Environment note: run offline against a torch-stubbed environment, so the CUDA/engine-dependent top-level `tests/conftest.py` is skipped via `--confcutdir=tests/tool_parsers`. ``` pytest tests/tool_parsers/test_hermes_tool_parser.py \\ tests/tool_parsers/test_longcat_tool_parser.py \\ tests/parser/test_parse.py \\ tests/parser/test_streaming.py \\ tests/parser/test_include_reasoning.py \\ -q --confcutdir=tests/tool_parsers ``` Result: **105 passed, 1 skipped, 1 xfailed** (the skip and xfail are pre-existing Granite4-specific cases, unchanged by this PR). Per-suite, before vs after this change: | Suite | Before | After | | --- | --- | --- | | `tests/tool_parsers/test_hermes_tool_parser.py` | 30 passed, 1 skipped | 34 passed, 1 skipped | | `tests/tool_parsers/test_longcat_tool_parser.py` | 16 passed, 1 xfailed | 16 passed, 1 xfailed | | `tests/parser/test_parse.py`, `test_streaming.py`, `test_include_reasoning.py` | 55 passed | 55 passed | The 4 added Hermes tests are one new case (`test_hermes_streaming_tool_call_with_name_key_argument`) across the existing `stream_interval` parametrisation (2/3/5/8), covering an argument literally named `\"name\"` — the bug class from #49535 — at multiple chunk granularities. Lint: `pre-commit` hooks ran on commit — `ruff check`, `ruff format`, and `mypy` for Python 3.10 all passed. ### End-to-end, against a real server and model Run separately on a T4 GPU, since the suite above is offline-only: ``` pytest tests/entrypoints/tool_parsers/test_hermes_tool_parser.py -k llama -v -s ``` Result: **4 passed, 4 deselected** (the deselected cases are the `granite4` config). This boots a real `vllm serve` with `--tool-call-parser hermes` against `meta-llama/Llama-3.2-1B-Instruct` plus the `minpeter/LoRA-Llama-3.2-1B-tool-vllm-ci` adapter, and drives it through the `openai` client — covering non-streaming, streaming, and parallel tool calls. The streaming case confirms argument type coercion survives the migration end to end, reconstructing `{'inserted': True, 'product_id': 7355608}` — a real boolean and a real integer rather than strings, which is the divergence the `arg_converter` exists to prevent. Longcat has no e2e suite upstream, so its coverage here is the 17 unit tests only — worth noting since it was changed as a consequence of the Hermes migration rather than being its target.", "url": "https://github.com/vllm-project/vllm/pull/51937", "createdAt": "2026-08-12T07:27:24Z", "updatedAt": "2026-08-13T10:48:19Z", "timestamp": "2026-08-13T10:48:19Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "tool-calling" ], "author": "muhammadfawaz1", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51938", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Register sparse DeepSeek V3.2 MTP", "text": "## Purpose Register the sparse DeepSeek V3.2 MTP implementation and select it for `glm_moe_dsa` speculative decoding configs. GLM MoE DSA previously inherited the generic `DeepSeekMTPModel` override even though its draft model needs the sparse DeepSeek V3.2 implementation. This PR: - maps the config to `DeepseekV32MTPModel`; - registers that architecture with the model registry; and - preserves its `(logits_hidden, feedback_hidden)` return contract in the legacy proposer. This PR is independent of the quantization optimization PRs split from #51936. ## Duplicate-work check No issue number was provided. I searched open PRs for `DeepseekV32MTPModel registry`; the only matching PR was the superseded draft #51936 from which this focused change was extracted. ## Tests ```bash .venv/bin/python -m pytest \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/v1/spec_decode/test_llm_base_proposer_sampling.py -q # 12 passed pre-commit run --files \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/models/registry.py \\ tests/v1/spec_decode/test_llm_base_proposer_sampling.py \\ vllm/config/speculative.py \\ vllm/model_executor/models/registry.py \\ vllm/v1/spec_decode/llm_base_proposer.py # all applicable hooks passed ``` ## AI assistance disclosure This change was developed with OpenAI Codex assistance. This is a draft PR; the human submitter must review every changed line and confirm they understand and can defend the change before marking it ready.", "url": "https://github.com/vllm-project/vllm/pull/51938", "createdAt": "2026-08-12T07:31:57Z", "updatedAt": "2026-08-13T00:13:05Z", "timestamp": "2026-08-13T00:13:05Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "new-model", "speculative-decoding", "needs-rebase", "deepseek" ], "author": "WoosukKwon", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51949", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Enable LoRA support for tower and connector in Cosmos3-Edge", "text": "## Purpose An operator serving `nvidia/Cosmos3-Edge` can attach a LoRA adapter to the language model, but not to the vision **tower** or the **connector**, so visual feature extraction and its projection into the language model's space are frozen. Adapting the model to a new visual domain — a different imaging modality, an unusual camera, a specialised document type — currently means full fine-tuning. Today the engine refuses to start at all: `Cosmos3EdgeForConditionalGeneration does not support LoRA yet`. The machinery for this already exists; each model opts in by declaring `SupportsLoRA` and implementing two token-count helpers that convert an LLM sequence token count into the row counts entering the tower and the connector. Cosmos3-Edge implemented neither, although its `get_mm_mapping` and `packed_modules_mapping` were already correct. Getting those counts wrong fails **silently** rather than loudly, which is what shapes the verification below: the LoRA metadata buffer is not cleared between forward passes, so any row an undercount fails to cover reads a stale adapter index from the previous pass, and the operator receives plausible output produced by the wrong adapter. Nothing raises. For that reason the row counts here are **measured against real tensors**, not derived on paper. Part of #31479, claimed there on 2026-08-07. That issue tracks this feature across many models and stays open until they are all done, so this PR references it without a closing keyword — matching every merged sibling in the series. This PR: - Declares `SupportsLoRA` on the model class. - Implements `get_num_mm_encoder_tokens` and `get_num_mm_connector_tokens`. - Fills in the LoRA cell for this model in `docs/models/supported_models.md`. ## Technical Details The two helpers are **chained**: `get_num_mm_connector_tokens` receives the *encoder row* count returned by `get_num_mm_encoder_tokens`, not the LLM sequence token count. ``` LLM sequence tokens --x merge²--> encoder rows --// merge²--> connector rows ``` `merge` is `spatial_merge_size`. The connector body is the exact inverse of the encoder body, so the round trip returns the original count. **The merge factor is read from `self.visual.spatial_merge_size`**, which differs from Qwen2.5-VL and Qwen3-VL, which read `config.vision_config.spatial_merge_size`. The reason is specific to this checkpoint: `Cosmos3EdgeConfig.__init__` assigns ```python self.vision_config.spatial_merge_size = self.projector_config.spatial_merge_size ``` (`vllm/transformers_utils/configs/cosmos3_edge.py:126`). `Siglip2VisionConfig` has no such field, so the vision config's attribute is a synthesized alias; `projector_config` is where the checkpoint stores it and what the projector is built from (`cosmos3_edge.py:227`). `cosmos3_edge.py:597` already computes this same quantity the same way when splitting vision embeddings per item. Both helpers read the factor once and the connector body is the exact inverse of the encoder body, so the round trip cancels regardless of the factor's value. Integer division only, since the runtime mapping multiplies a list by the result. No divisibility assertion: `vllm/lora/model_manager.py:250` calls the encoder helper once at engine init with a whole-batch budget that carries no divisibility guarantee. A non-positive input returns zero — the connector floor-divides, so a negative count would otherwise yield a silently empty mapping rather than an error. ## Duplicate check Per `AGENTS.md`, re-run immediately before opening: ```bash gh issue view 31479 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"31479 in:body\" gh pr list --repo vllm-project/vllm --state all --search \"Cosmos3-Edge LoRA\" gh pr list --repo vllm-project/vllm --state open --search \"tower connector lora\" gh pr list --repo vllm-project/vllm --state open --search \"Cosmos3EdgeForConditionalGeneration\" ``` No open or merged PR adds tower/connector LoRA to Cosmos3-Edge; the model-name and class-name searches return nothing. The eleven open PRs in this family target other models. Three neighbours a reviewer may find, none of which this duplicates: - **#51221** — adds EVS video pruning to this same model. Different feature, overlapping files; a rebase may be needed depending on merge order. See *Known limitations*. - **#42662** — migrates the helper interface to `get_mm_lora_token_counts`. Its default implementation delegates to the two methods added here, so this model migrates unchanged. - **#47400** — fixes a *different* defect in the same mapping builder: cache-served items are counted although the encoder forward skips them. Independent of this change. ## Test Plan ```bash # CI-visible: mocked models, no weights, no GPU .venv/bin/python -m pytest tests/v1/worker/test_gpu_model_runner.py -k \"cosmos3_edge or lora\" # Processing regression. tests/models/registry.py marks this model # is_available_online=False, so the file is skipped unless given a local path: SNAPSHOT=$(.venv/bin/python -c \\ \"from huggingface_hub import snapshot_download; print(snapshot_download('nvidia/Cosmos3-Edge'))\") COSMOS3_EDGE_MODEL_PATH=\"$SNAPSHOT\" .venv/bin/python -m pytest \\ tests/models/multimodal/processing/test_cosmos3_edge.py .venv/bin/python -m pytest tests/models/test_registry.py pre-commit run --all-files ``` End-to-end runs: single RTX 3090 (24 GB), CUDA 13.0, at this PR's base commit. ## Test Result **Before / after.** Starting the engine with `--enable-lora --enable-tower-connector-lora`: ``` # before — engine refuses to start ValueError: Cosmos3EdgeForConditionalGeneration does not support LoRA yet. $ echo $? 1 # after — engine starts with the feature active WARNING [model_manager.py:242] LoRA for the tower and connector of multimodal models is experimental and may contain bugs. Please report any related issues on GitHub if you encounter them. $ echo $? 0 ``` **Helper output vs. real tensor rows**, with `--enable-tower-connector-lora`. Forward hooks on `visual.encoder.encoder.layers.0.self_attn.qkv_proj` (tower) and `visual.projector.linear_fc1` (connector): | input | LLM tokens | encoder rows observed | helper | connector rows observed | helper | |---|---|---|---|---|---| | image | 1107 | 4428 | 4428 | 1107 | 1107 | | video, 4 frames | 4180 | 16720 | 16720 | 4180 | 4180 | The same counts hold with an adapter attached — applying LoRA does not perturb them. Extended on CPU across a range of image sizes and video lengths: encoder rows, connector rows and the round trip are exact in every case.
Script that produces the table above ```python import os os.environ.setdefault(\"VLLM_ENABLE_V1_MULTIPROCESSING\", \"0\") from types import SimpleNamespace from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from vllm.assets.video import VideoAsset from vllm.model_executor.models.cosmos3_edge import ( Cosmos3EdgeForConditionalGeneration as C3E, ) from vllm.transformers_utils.config import get_config MODEL = \"nvidia/Cosmos3-Edge\" TOWER = \"visual.encoder.encoder.layers.0.self_attn.qkv_proj\" CONNECTOR = \"visual.projector.linear_fc1\" seen: dict[str, list[int]] = {\"tower\": [], \"connector\": []} def install_hooks(model): def record(key): def hook(_module, args): seen[key].append(args[0].shape[-2]) return hook for name, module in model.named_modules(): if name == TOWER: module.register_forward_pre_hook(record(\"tower\")) elif name == CONNECTOR: module.register_forward_pre_hook(record(\"connector\")) llm = LLM( model=MODEL, max_model_len=16384, limit_mm_per_prompt={\"image\": 1, \"video\": 1}, gpu_memory_utilization=0.85, enforce_eager=True, allowed_local_media_path=\"/\", mm_processor_cache_gb=0, ) llm.apply_model(install_hooks) config = get_config(MODEL, trust_remote_code=False) tokenizer = llm.get_tokenizer() start = tokenizer.decode([config.vision_start_token_id]) end = tokenizer.decode([config.vision_end_token_id]) image_pad = tokenizer.decode([config.image_token_id]) params = SamplingParams(temperature=0.0, max_tokens=1) def check(label, output, token_id): llm_tokens = output[0].prompt_token_ids.count(token_id) encoder_rows, connector_rows = seen[\"tower\"][-1], seen[\"connector\"][-1] stub = SimpleNamespace( visual=SimpleNamespace( spatial_merge_size=config.projector_config.spatial_merge_size ) ) predicted_encoder = C3E.get_num_mm_encoder_tokens(stub, llm_tokens) predicted_connector = C3E.get_num_mm_connector_tokens(stub, predicted_encoder) print( f\"{label:<8} llm_tokens={llm_tokens} \" f\"encoder observed={encoder_rows} helper={predicted_encoder} \" f\"{'OK' if encoder_rows == predicted_encoder else 'MISMATCH'} \" f\"connector observed={connector_rows} helper={predicted_connector} \" f\"{'OK' if connector_rows == predicted_connector else 'MISMATCH'}\" ) check( \"image\", llm.generate( { \"prompt\": f\"<|im_start|>user\\n{start}{image_pad}{end}Describe.\" f\"<|im_end|>\\n<|im_start|>assistant\\n\", \"multi_modal_data\": {\"image\": ImageAsset(\"stop_sign\").pil_image}, }, params, ), config.image_token_id, ) video = VideoAsset(name=\"baby_reading\", num_frames=4) check( \"video\", llm.chat( [ { \"role\": \"user\", \"content\": [ { \"type\": \"video_url\", \"video_url\": {\"url\": f\"file://{video.video_path}\"}, }, {\"type\": \"text\", \"text\": \"Describe.\"}, ], } ], params, ), config.video_token_id, ) ``` The video prompt goes through the chat template. A hand-built video placeholder omits the per-frame timestamp tokens, which silently misplaces the embeddings and would invalidate the comparison.
**The adapter is applied, not silently skipped.** Serving with `--enable-tower-connector-lora` and a rank-8 adapter targeting one tower layer and the connector: the engine emits the tower/connector-LoRA experimental warning at startup, the Punica shrink and expand kernels are invoked during the adapter request, and the output tensors of the wrapped modules differ from the no-adapter run for images and for video alike. This shows the adapter path is *reached*; it is not a model-quality measurement. **Unit and regression tests** — all pass: ``` tests/v1/worker/test_gpu_model_runner.py -k \"cosmos3_edge or lora\" tests/models/multimodal/processing/test_cosmos3_edge.py tests/models/test_registry.py pre-commit run --all-files ``` ## Known limitations - **Video pruning (EVS) is not addressed.** This model does not implement `SupportsMultiModalPruning` at HEAD, so the interaction is unreachable here. On models that do support pruning, `PlaceholderRange.get_num_embeds()` returns a post-prune count while the tower has already processed the unpruned set, so these helpers undercount by roughly `1/(1-q)`, bounded by the frame count. That follows from the shared call site at `vllm/v1/worker/gpu/mm/lora.py:45` rather than from any one model, and already applies to `Qwen2_5_VLForConditionalGeneration`, `Qwen3VLForConditionalGeneration` and `Qwen3VLMoeForConditionalGeneration`. Images are unaffected. This PR neither introduces nor widens it. - **`visual.encoder.embeddings.patch_embedding` is a plain `nn.Linear`**, so LoRA skips it. It lives in `lfm2_siglip2.py`, shared with two other models. Pre-existing, out of scope, unchanged here. ## Acceptance criteria - [x] New tests added that exercise the changed code path - [x] Existing test suite passes locally (model-runner, multi-modal processing, model registry) - [x] Follows the project style guide — `pre-commit run --all-files` clean, including mypy - [x] No breaking changes: the change is additive, and no existing behaviour is modified - [x] Documentation updated (`docs/models/supported_models.md`) - [x] Duplicate-work checks re-run immediately before opening, per `AGENTS.md` - [x] AI assistance disclosed ## AI assistance AI assistance was used. I reviewed every changed line, and the results above are from real runs on the stated hardware.
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51949", "createdAt": "2026-08-12T08:01:44Z", "updatedAt": "2026-08-12T19:40:41Z", "timestamp": "2026-08-12T19:40:41Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "charitarthchugh", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51952", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "NIXL: Use int32 array for indices to avoid intermediate conversion", "text": "## Purpose As documented NIXL accepts KV indices as `int32` array, while vLLM creates and passes `int64` array. This results in extra allocation and conversion on NIXL side, which is redundant. Removing this conversion + adding zero-copy on NIXL side improves performance https://github.com/ai-dynamo/nixl/pull/1718 ## Test Result \"pareto_compare_nixl_zci_vs_concurrency_100538\"", "url": "https://github.com/vllm-project/vllm/pull/51952", "createdAt": "2026-08-12T08:15:50Z", "updatedAt": "2026-08-13T13:58:20Z", "timestamp": "2026-08-13T13:58:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "kv-connector" ], "author": "iyastreb", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51955", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Split Quantization job into three directory-based steps", "text": "## Purpose Part of the effort to get every CI job under ~30 minutes. The `Quantization` step is one of the 10 longest jobs in main CI: 57–67 min of test execution (timeout 75) in recent nightly full runs ([83094](https://buildkite.com/vllm/ci/builds/83094), [83068](https://buildkite.com/vllm/ci/builds/83068)). This splits `tests/quantization/` into subdirectories so each CI step runs a **whole directory** — no per-file lists in YAML, so a new test file is automatically covered by whichever step owns its directory (matching the existing convention of `tests/models/language/{generation,pooling}` and `tests/kernels/{attention,moe,quantization}`): | step | directory | measured test time (slow / fast night) | |---|---|---| | Quantization (Core) | `core/` + package root sweep | 22.3 / 17.8 min | | Quantization (Compressed Tensors) | `compressed_tensors/` | 21.6 / 17.9 min | | Quantization (Checkpoint Formats) | `checkpoint_formats/` | 23.5 / 21.1 min | | Quantized MoE Test (B200) (existing) | `blackwell/` | unchanged | Times come from per-test timestamps extracted from the Buildkite logs of the two nightly builds (529 tests; per-file sums reconcile with pytest's reported totals to within 0.1 min). Each new step lands at ~28–30 min wall including image pull, with `timeout_in_minutes: 45`. Design notes: - The Core step runs `pytest quantization/ --ignore` the sibling directories, so any file at the package root (including future ones) is always collected by exactly one step. Platform-gated files (`test_cpu_w8a8.py`, `test_cpu_wna16.py`, `test_gfx950_moe.py`) stay at the root — they self-skip off-platform and are referenced by exact path from the CPU/AMD pipelines. - `test_blackwell_moe.py` moves into `blackwell/`, so the old `--ignore quantization/test_blackwell_moe.py` disappears from the NVIDIA and AMD commands; the B200 step now runs the directory. - The torchao/conch installs are kept on all three steps to avoid any kernel-selection behavior change; narrowing them per-step can be a follow-up. - Moved files' relative imports were converted to absolute `tests.*` imports; all exact-path references were updated (AMD `test-amd.yaml`, Intel `intel_jobs/*`, CPU `hardware_tests/cpu.yaml` + `run-cpu-test-arm.sh`, `test_areas/kernels.yaml` DeepGEMM step, `docs/features/quantization/modelopt.md`). ## Not a duplicate Checked `gh pr list` for open PRs touching the Quantization CI job or `tests/quantization` structure (searches: \"quantization test split\", \"test_areas quantization\", CI breakdown) — no open PR addresses this. ## Test plan Config + file-move change; no runtime code touched, so no model evaluation is applicable. The test *set* is a pure re-partition: same files, same `-k 'not test_compressed_tensors_w4a8_fp8'` deselect (now scoped to the Compressed Tensors step, where that test lives), same blackwell exclusion semantics. Ran locally: - `pre-commit` on the commit — all hooks passed (ruff check/format, mypy 3.10, typos, shellcheck, SPDX headers) - YAML parse of all six touched pipeline files - `python -m py_compile` on all moved test files - Repo-wide sweep for stale references to every moved filename — none remain The three new steps in this PR's own CI run are the end-to-end validation that collection and imports resolve; reviewers can compare collected-test counts against the previous single job (485 passed / 46 skipped / 1 deselected in the nightlies). ## AI assistance This change was developed with AI assistance (Claude). Timing analysis was derived from nightly Buildkite logs; every changed line has been reviewed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51955", "timestamp": "2026-08-12T16:47:09Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "documentation", "ready", "ci/build", "cpu", "nvidia", "quantization" ], "author": "khluu", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:51959", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] DeepGEMM pin has no SM120 kernels: family-12 Blackwell cannot run hyperconnections", "text": "## The problem DeepGEMM dispatches the TF32 hyperconnection pre-norm GEMM on the device arch major, in `csrc/apis/hyperconnection.hpp`. At the revision this repository pins, `vllm-project/DeepGEMM` `e21c821`, that dispatch handles `arch_major` 9 and 10 only: ```cpp const auto arch_major = device_runtime->get_arch_major(); if (arch_major == 9) { sm90_tf32_hc_prenorm_gemm(...); } else if (arch_major == 10) { sm100_tf32_hc_prenorm_gemm(...); } else { DG_HOST_UNREACHABLE(\"Unsupported architecture\"); } ``` That revision ships no `sm120_*` implementation at all. `csrc/jit_kernels/impls/` contains `sm90_*` and `sm100_*` only. `tools/install_deepgemm.sh` currently comments that the pin targets nv-dev \"due to sm120 support\", but the pinned tree does not carry it. So every SM120/SM121 device aborts as soon as a model exercises hyperconnections. DeepSeek-V4 Flash is one: it is configured with `hc_mult=4`, so the path is hit on the first forward pass. ## The change Points both pins at `deepseek-ai/DeepGEMM` `a6b593d`, which carries `sm120_bf16_gemm`, `sm120_bmk_bnk_mn`, `sm120_fp8_fp4_gemm_1d1d` and `sm120_tf32_hc_prenorm_gemm`, and whose dispatch has the `arch_major == 12` branch. `cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh` both document that they must stay in sync, so both are updated. To be explicit: only the cmake path is exercised by my build; the `install_deepgemm.sh` edit is the same two values kept in sync, but I have not run that script. ## Testing With this change, DeepSeek-V4-Flash-0731 builds and serves on 2× NVIDIA GB10 (SM121, aarch64, CUDA 13.2, torch 2.13.0+cu132) with `--tensor-parallel-size 2` across two nodes. That is the configuration currently running here, and the fetched tree does contain the four `sm120_*` kernels. The failure itself does not really need a reproduction, since it is visible in the pinned tree: `csrc/apis/hyperconnection.hpp` at `e21c821` has no `arch_major == 12` branch, and `csrc/jit_kernels/impls/sm120_tf32_hc_prenorm_gemm.hpp` does not exist at that revision. For the record, we did hit `DG_HOST_UNREACHABLE(\"Unsupported architecture\")` on the old pin before switching, but I no longer have that log and am not claiming a fresh reproduction. Note it is a runtime abort rather than a build error: DeepGEMM JIT-compiles its kernels, so the build succeeds and the process dies on the first forward pass reaching the dispatch. Recipe, scripts and raw measurements from that setup: https://github.com/Mirrdhyn/dsv4-flash-dgx-spark ## Caveat, and a question for maintainers This is the minimal change that demonstrably fixes SM12x, but it may not be the change you want. The two hyperconnection kernels shared by both trees are byte-identical (`sm90_tf32_hc_prenorm_gemm.hpp` `c17d1b5`, `sm100_tf32_hc_prenorm_gemm.hpp` `0071e2c`), so the hyperconnection path itself is unaffected on SM90 and SM100. Other kernels have diverged, however: `sm90_bf16_gemm.hpp` differs between the two revisions, which suggests the fork carries vLLM-specific work that repointing would drop. I have no SM90 or SM100 hardware and have not tested those architectures. If the fork does carry changes worth keeping, the better fix is to port the `sm120_*` kernels into `vllm-project/DeepGEMM`, or to rebase the fork onto a newer upstream, and leave the pin where it is. I am happy to redo the patch that way. Either path is fine by me; the point of this PR is that SM12x is currently broken and the fix is small.", "url": "https://github.com/vllm-project/vllm/pull/51959", "createdAt": "2026-08-12T08:40:00Z", "updatedAt": "2026-08-13T14:47:11Z", "timestamp": "2026-08-13T14:47:11Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "needs-rebase", "ci/build" ], "author": "Mirrdhyn", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51967", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][DSV4] Optimize global top-k index kernel with compile-time constants", "text": "## Purpose Optimize global top-k index kernel with compile-time constants ## Test Plan ``` vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --tensor-parallel-size 8 --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --no-enable-prefix-caching --max-num-batched-tokens 16384 ``` ``` vllm bench serve --backend vllm --base-url http://localhost:8000 --model deepseek-ai/DeepSeek-V4-Flash-0731 --dataset-name random --random-input-len 1024 --random-output-len 64 --num-prompts 128 --num-warmups 8 --request-rate inf --ignore-eos --temperature 0 --seed 701 --save-result ``` ## Test Result ## Serving Benchmark Results | Version | Mean Output Throughput | Mean TPOT | Completed Requests | Relative Change | | --- | ---: | ---: | ---: | ---: | | main | 634.97 tokens/s | 103.33 ms | 128/128 | Baseline | | this pr | 638.12 tokens/s | 102.32 ms | 128/128 | Throughput **+0.50%**, TPOT **-0.98%** | ## Kernel Microbenchmark ```python import torch import triton from vllm.models.deepseek_v4.common.ops.cache_utils import ( _compute_global_topk_indices_and_lens_kernel, ) num_tokens = 512 topk = 512 block_size = 256 triton_block_size = 1024 global_topk_indices = torch.empty( (num_tokens, topk), device=\"cuda\", dtype=torch.int32 ) global_topk_lens = torch.empty(num_tokens, device=\"cuda\", dtype=torch.int32) topk_indices = torch.randint( 0, 8192, (num_tokens, topk), device=\"cuda\", dtype=torch.int32 ) block_table = torch.arange( 8192, device=\"cuda\", dtype=torch.int32 ).reshape(num_tokens, -1) def run(): _compute_global_topk_indices_and_lens_kernel[(num_tokens,)]( global_topk_indices, global_topk_indices.stride(0), global_topk_lens, topk_indices, topk_indices.stride(0), topk, block_table, block_table.stride(0), block_size, TRITON_BLOCK_SIZE=triton_block_size, ) for _ in range(20): run() torch.cuda.synchronize() latency_us = triton.testing.do_bench(run, warmup=100, rep=500) print(f\"Latency: {latency_us:.3f} us\") ``` | Kernel Version | Latency | Relative Change | | --- | ---: | ---: | | Original implementation | ~7.50 us | Baseline | | Compile-time-constant optimization | ~6.37 us | **15.1% faster** | ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51967", "createdAt": "2026-08-12T09:06:50Z", "updatedAt": "2026-08-13T02:10:36Z", "timestamp": "2026-08-13T02:10:36Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [], "author": "chaunceyjiang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51968", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][Tests] Make tests device-agnostic", "text": "## Purpose Two kernel tests hardcoded a CUDA device, so they gave no coverage on XPU even though the kernels they exercise are available there. This makes both resolve the device via `current_platform` instead. **`test_fused_recurrent_packed_decode.py`** — gated on `torch.cuda.is_available()`, so it skipped on every non-CUDA accelerator. Both `fused_recurrent_gated_delta_rule_packed_decode` and its `fused_recurrent_gated_delta_rule` reference are pure Triton with no CUDA-only intrinsics, so the gate was stricter than the code requires. **`test_merge_attn_states.py`** — hardcoded `device=\"cuda\"` in eleven places and gated `test_merge_attn_states` on `current_platform.is_cuda()`. `test_merge_attn_states_both_empty` had no guard at all, so on XPU it did not skip but failed outright with `AssertionError: Torch not compiled with CUDA enabled`. vllm-project/vllm-xpu-kernels#467 (\"align merge_attn_states kernel interface\") registers `merge_attn_states` for `torch::kXPU` under the same `_C` schema and argument order that `vllm._custom_ops.merge_attn_states` already calls, and `vllm/platforms/xpu.py` imports `vllm_xpu_kernels._C` at module load. The op therefore resolves on XPU with no test-side dispatch, so the only platform-specific thing left in that test was the device string. No separate code path, no per-platform reference and no FP8 skip are needed: the XPU kernel supports fp32/fp16/bf16 inputs and the FP8-output path (static per-tensor `output_scale`), matching the CUDA coverage. No production code changed, and CUDA behaviour is unchanged. ## Test Plan ``` pytest tests/kernels/test_fused_recurrent_packed_decode.py -v pytest tests/kernels/attention/test_merge_attn_states.py -v ``` ## Test Result Intel Arc Pro B70 (torch 2.13.0+xpu, Triton 3.7.2): | Test | Before | After | | --- | --- | --- | | `test_fused_recurrent_packed_decode` | 6 skipped | **6 passed** | | `test_merge_attn_states` | 6 failed, 2592 skipped | **2598 passed** | The 6 failures before are `test_merge_attn_states_both_empty`, which had no skip guard and hit `Torch not compiled with CUDA enabled`. No regression on NVIDIA L40S: `test_merge_attn_states` is 2598 passed both before and after; `test_fused_recurrent_packed_decode` was also checked on L40S. The `use_fp8=True` x `head_size=613` subset (216 cases) passes on XPU, confirming both the FP8 output path and the `head_size % pack_size` constraint.", "url": "https://github.com/vllm-project/vllm/pull/51968", "createdAt": "2026-08-12T09:09:05Z", "updatedAt": "2026-08-13T14:29:43Z", "timestamp": "2026-08-13T14:29:43Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "intel-gpu" ], "author": "pmanczak", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51969", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Enforce server-side num_frames ceiling in VideoMediaIO merge", "text": "## Summary - **Fixes the incomplete CVE-2026-34755 remediation** reported in [GHSA-vxqj-p4gw-9h4c](https://github.com/vllm-project/vllm/security/advisories/GHSA-vxqj-p4gw-9h4c): request-level `media_io_kwargs.video.num_frames=-1` (or an arbitrarily large positive value) could override the engine frame-count ceiling, restoring the unbounded `video/jpeg` decode path that PR #38636 was meant to close. - **Adds server-enforced clamping** in `VideoMediaIO.merge_kwargs`: when `runtime_kwargs` are present, the merged `num_frames` is clamped to the engine ceiling (positive engine value → hard max; engine `-1` → operator unlimited; absent → constructor default of 32). Requests may lower but never raise the ceiling. - **Covers the fps-only override path** where the existing fps/num_frames mutual-exclusion logic could pop the engine ceiling away, leaving the effective value unguarded. ## Test plan - [x] `TestMergeKwargsNumFramesCeiling`: 10 merge-policy unit tests covering clamp of `-1`, oversized, lower-allowed, engine unlimited, absent/empty defaults, fps-only wipe, and no-runtime passthrough. - [x] `test_merge_then_load_base64_enforces_ceiling`: integration test through `merge_media_io_kwargs` → `VideoMediaIO.load_base64(\"video/jpeg\", ...)` confirming frame count is bounded. - [x] All 46 tests in `tests/multimodal/media/test_video.py` pass (including pre-existing GPU-backend and decode tests). - [x] All pre-commit hooks pass (ruff, mypy, typos, etc.). ```bash .venv/bin/python -m pytest tests/multimodal/media/test_video.py -v # 46 passed in 6.87s ``` Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/51969", "createdAt": "2026-08-12T09:12:17Z", "updatedAt": "2026-08-13T06:03:58Z", "timestamp": "2026-08-13T06:03:58Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "multi-modality" ], "author": "jperezdealgaba", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51974", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "feat: Add support for profile_prefix payload in HTTP /start_profile endpoint", "text": "## Proposal: Upstream API Support for Dynamic Profiling Options Allow users to pass dynamic settings to the profiler over the API (e.g., toggling native options or `experimental_config` flags) without requiring engine server restarts. This will help leverage the various options supported by native [`torch.profiler.profile`](https://docs.pytorch.org/docs/2.13/profiler.html#api-reference) ### Current Gap Currently, there is a discrepancy in profiling capabilities between online and offline serving: * **❌ Online Serving (`api_server.py`):** The `/start_profile` endpoint currently does not let pass any dynamic profiling options to a live production server. * **✅ Offline Serving (`vllm.LLM`):** Developers can programmatically pass a string argument locally via `llm.start_profile(profile_prefix=\"...\")`. Downstream hardware-specific profiling backends can rely on this to absorb configuration requirements on-the-fly. ### Purpose & Implementation To safely enable dynamic profiling arguments upstream, we chose the most lightweight architectural path possible instead of inventing a new configuration schema. **This PR:** 1. **Patches the REST router (`api_router.py`):** Optionally extracts a lightweight JSON body mapping `{\"profile_prefix\": \"...\"}` or a query parameter string. 2. **Updates the Abstract Protocol (`EngineClient`):** Safely permits the protocol signature to seamlessly accept `async def start_profile(self, profile_prefix: str | None = None)`. *Why this approach?* The internal vLLM v1 architecture (including `AsyncLLM.start_profile`) **already natively expects and supports the `profile_prefix` parameter**. We simply bridge the HTTP layer gap. ## Test Plan 1. Start an API server: ```bash vllm serve facebook/opt-125m ``` 2. Trigger the profiler over HTTP, tunnelling some options inside the JSON body: ```bash curl -X POST http://localhost:8000/start_profile \\ -H \"Content-Type: application/json\" \\ -d '{\"profile_prefix\": \"host_tracer_level:3;e2e_enable_fw_throttle:true\"}' # example ``` 3. Issue standard generation requests to capture metrics. 4. Stop the profiler: ```bash curl -X POST http://localhost:8000/stop_profile ``` ## Test Result & Proof of Concept We captured TPU traces and uploaded and viewed them on XProf and got the results as expected from the passed options in the `/start_profile` API. ### Example trace on XProf \"image\" ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
Signed-off-by: Rushabh Lalwani ", "url": "https://github.com/vllm-project/vllm/pull/51974", "createdAt": "2026-08-12T09:35:06Z", "updatedAt": "2026-08-13T09:20:33Z", "timestamp": "2026-08-13T09:20:33Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend" ], "author": "rushabh-46", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51980", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][MoE] Update AITER MXFP4 W4A16 tests to the renamed expert_mask", "text": "## Purpose Fixes two errors in `tests/kernels/moe/test_rocm_aiter_moe.py` on gfx950: ``` TypeError: rocm_aiter_fused_experts() got an unexpected keyword argument 'expert_map' ``` #49758 renamed that keyword from `expert_map` to `expert_mask` and updated the production callers (`AiterExperts.apply`, quark), but not these two test call sites. This renames them; both pass `None`, so nothing else changes. The other `expert_mask=` uses in the same file call `torch.ops.vllm.rocm_aiter_fused_moe`, which has always used that name and is unaffected. Both tests are behind `on_gfx950()`, so the CI run on #49758 skipped them. ## Test Plan MI355 (gfx950), ROCm: ``` pytest -v -s tests/kernels/moe/test_rocm_aiter_moe.py ``` ## Test Result Before: `test_aiter_fused_moe_mi350_mxfp4_w4a16_accuracy` and `test_aiter_fused_moe_mi350_mxfp4_w4a16_determinism` both error out with the `TypeError` above. After: both pass; the whole file is **31 passed**. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51980", "createdAt": "2026-08-12T11:08:29Z", "updatedAt": "2026-08-12T17:45:46Z", "timestamp": "2026-08-12T17:45:46Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "rocm", "ready" ], "author": "stefankoncarevic", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51982", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add generic CI trace collectors", "text": "## What changed - export CMake/source → target → GNU build-ID artifact → mangled CUDA-kernel provenance from the CUDA image build - capture `ninja -t deps` and emit repo-local header → target edges - scan final nvcc object files and emit exact compiling translation-unit → mangled-kernel edges; ambiguous/generated/external objects are counted and left unmapped - persist immutable `build-graph.jsonl` and `kernel-map.jsonl` in the test image, then publish those bytes once from `image-build` with repository SHA, exact image digest, byte counts, and SHA-256 hashes - make runtime trace jobs publish only Python/GPU evidence plus hashes that reference the image-build provenance - wrap unchanged production pytest command lists with Python node/outcome and per-test execution collectors - capture fork-tree CUDA/NVTX/PyTorch evidence under pinned Nsight Systems and conservatively attribute launches as exact test, serial temporal test, or job union - add opt-in exact-node deep mode with ordered Python call/return events, CUDA launch callchains, active PyTorch/NVTX scopes, kernel identities, and native provenance - install the ordered recorder from the auto-loaded pytest trace plugin so ordinary in-process kernel tests and spawn-per-test engine tests both emit test-scoped call paths - export Nsight JSON payloads and recover launch-time Python stacks with `--cudabacktrace=kernel:0`, removing the prior 80 µs CUDA-API threshold - fail closed on missing correlated launches, callchains, Python launch stacks, test attribution, conflicting ABI aliases, or malformed frame counts ## Why The selector needs evidence from the exact production commands, image, and hardware without duplicating test definitions. Static build ownership now belongs to the immutable image build, while runtime jobs publish only facts they observe. Ambiguous runtime/native evidence remains explicit rather than being guessed. An open-PR search for CI trace collectors/test selection found no competing implementation beyond this draft. Companion ci-infra draft: https://github.com/vllm-project/ci-infra/pull/469 ## Validation - `45 passed` across `tests/tools/ci_test_selection` - all applicable repository pre-commit hooks passed for the 11 changed files - synthetic Nsight fixtures cover exact/temporal/job-union attribution, process-scoped correlation, ABI-alias deduplication, launch-time Python stacks, malformed-stack rejection, and exact object → translation-unit provenance - an independent clean replay from Buildkite #83631's raw SQLite trace produced byte-identical deep GPU, native-provenance, and summary outputs to the in-job parser - manifest tests verify exact static bytes, image digest, repository SHA, and runtime reference hashes - no model-output evaluation is applicable ## Live evidence [Buildkite #83555](https://buildkite.com/vllm/ci/builds/83555) is the accepted FlashMLA/H100 pilot: 157 passed/2 skipped, all 600,146 kernel launches exact-test attributed, 12,337 structural native edges, 9,460 artifact-to-kernel edges, and an 8/8 GNU build-ID join. [Buildkite #83563](https://buildkite.com/vllm/ci/builds/83563) is the accepted combined genericity pilot: Engine Python-only capture passed 142 nodes with zero GPU claims; Responses GPU capture passed 298 nodes with 7,198,722/7,198,723 temporal-test-attributed launches and zero job-union fallback. The integrity-clean combined graph contains 13,417 nodes/39,951 edges. [Buildkite #83631](https://buildkite.com/vllm/ci/builds/83631) is the accepted exact-node deep trace on an NVIDIA L4 at this PR's exact head `06b7bad16543ba2334cbe8cb7f26673fcb0e1b87` and ci-infra `0977a235b25fb033f8affb8a8048d4a3fad78df3`. All 3 jobs passed. The trace attributed 10,367/10,367 launches to `tests/v1/engine/test_engine_core_client.py::test_engine_core_client[False]`, recovered 9,975 exact launch-time Python stacks (96.22%, 358,491 frames), and joined all 4,333 vLLM-artifact-matched launches to exact object-derived translation units. The remaining 6,034 external/JIT launches and 392 launches without a sampleable Python context remain explicit rather than inferred. All 20 artifacts independently matched the manifest (449,377,710 bytes total), and a clean parser replay reproduced the three derived outputs byte-for-byte. The exact pytest wall time was 168s versus 1,638s in diagnostic #83614 (9.75× faster). Three short, manually auditable exact-node traces passed at this PR's current exact head `a0bd151aa56811d3897456025ae76373df16cc9a`, ci-infra `e404b7e040504f8973670a91821207dce94d96eb`, and one shared immutable image digest: - [#83643 SiluAndMul](https://buildkite.com/vllm/ci/builds/83643): 464 exact-test call/return events, 634 executed repository lines across 28 files, and 50/50 launches with exact Python stacks; all 3 vLLM-built launches joined exact translation units. - [#83646 RMSNorm](https://buildkite.com/vllm/ci/builds/83646): 500 exact-test call/return events, 670 lines across 30 files, and 98/98 launches with exact Python stacks; all 3 vLLM-built launches joined exact translation units. - [#83648 FlashMLA](https://buildkite.com/vllm/ci/builds/83648): 1,526 exact-test call/return events, 101 lines across 6 files, 4,197/4,197 exact-test launches, and 3,590 exact Python stacks (85.5%, 125,491 frames); all 605 vLLM-built launches joined exact translation units. Across these three builds, all 57 manifest files (125,298,874 bytes) independently hash-verified, all SQLite integrity checks passed, and clean parser replay reproduced every deep GPU/native/summary output byte-for-byte. The remaining 3,734 external/JIT launches are explicit and unmapped. ## Contribution notes AI assistance was used in implementation and validation. The human submitter must review every changed line and understand the design before moving this PR out of draft. This PR must remain draft. It does not merge, skip jobs, or enable production enforcement.", "url": "https://github.com/vllm-project/vllm/pull/51982", "createdAt": "2026-08-12T11:18:34Z", "updatedAt": "2026-08-13T02:25:01Z", "timestamp": "2026-08-13T02:25:01Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "ci/build" ], "author": "khluu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51987", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Attention] Add FlashInfer XQA decode support on SM12x\" (#49718)", "text": "Reverts #49718 ([Attention] Add FlashInfer XQA decode support on SM12x). ## Why `GPQA Eval (GPT-OSS) (DGX Spark)` regressed in nightly [build 83511](https://buildkite.com/vllm/ci/builds/83511) (commit `02ac178`) and is still red on the later build 83539. - Baseline nightly 83443 (same agent `spark-0df9-1`): **passed**, score 0.5619, 9m27s, zero harmony parse errors. - Build 83511: `FlashInfer resolved query dtypes: prefill=torch.bfloat16, decode=torch.bfloat16, decode_backend=xqa, kv_cache_dtype=torch.bfloat16, arch=sm121` — the dedicated XQA decode path this PR enables for `is_device_capability_family(120)`. That log line does not exist at baseline. - gpt-oss-20b decode output becomes gibberish: **1081** `openai_harmony.HarmonyError: unexpected tokens remaining in message header` (0 at baseline), and the eval exceeds its 1800s budget — `RuntimeError: Evaluation timed out`, 33m14s vs 9m27s. - The sibling GPQA jobs on 2xB200 and 2xH100 passed, so the blast radius is confined to SM12x, matching this PR's gating. gpt-oss uses attention sinks, which this PR routes through the new SM12x XQA path (`FlashInfer supports sinks on SM12x XQA and SM100 trtllm-gen`) — that is the most likely source of the corrupt decode output. Reverting restores the previous SM12x decode selection. A re-land should gate the dedicated XQA path (at minimum for sink-enabled models) until it is validated on GB10/sm121. - Failure count linked to this PR: 1 (`GPQA Eval (GPT-OSS) (DGX Spark)`) - Build: 83511 _Auto-generated by CI failure analyzer._", "url": "https://github.com/vllm-project/vllm/pull/51987", "createdAt": "2026-08-12T12:04:41Z", "updatedAt": "2026-08-13T15:44:47Z", "timestamp": "2026-08-13T15:44:47Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "nvidia" ], "author": "vllm-agent", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51989", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix Cosmos3-Edge processor after transformers 5.15 release", "text": "## Purpose This PR fixes Cosmos3-Edge processor which is broken when `transformers==5.15` is used, due to refactoring of underlying Qwen3-VL processor. With the fixes preprocessor will work correctly for both transformers==5.14 and 5.15. Also as model was released removed `is_available_online=False` from registry. ## Test Plan ``` python -m pytest tests/models/multimodal/processing/test_cosmos3_edge.py ``` ## Test Result ``` 3 passed, 14 warnings in 10.34s ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51989", "createdAt": "2026-08-12T12:28:23Z", "updatedAt": "2026-08-13T11:50:56Z", "timestamp": "2026-08-13T11:50:56Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "ready" ], "author": "bastefaniak", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:51994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fix DiffusionGemma silently freezing attention mask under CUDA graph replay", "text": "## Purpose While looking into whether FULL CUDA graphs are worth the added complexity for DiffusionGemma, I found that `DiffusionGemmaModelState._causal_buf` (the per-request flag that switches each request between causal/encoder and bidirectional/denoise attention) stops being updated once `CUDAGraphMode.FULL` graphs are captured. It's bool, but FlashAttentionMetadataBuilder.build() casts non-int32 causal tensors via `causal.to(torch.int32)`, allocating a new tensor every call. A FULL-mode graph binds to the address it saw at capture time, so once captured it keeps reading that orphaned tensor forever. Later updates to `_causal_buf` never reach it. Any request served through an already-captured FULL graph gets whatever causal/bidirectional pattern existed at capture (warmup) time for its slot, for the graph's whole lifetime, silently. This isn't opt-in: vLLM's default optimization level (O2) resolves cudagraph_mode to FULL_AND_PIECEWISE, whose decode_mode() is FULL, so `vllm serve google/diffusiongemma-26B-A4B-it` with no flags already runs every denoise/decode step under FULL cudagraphs. I couldn't pin observed output corruption on this specifically: an E2E FULL-vs-PIECEWISE comparison was inconclusive, since DiffusionGemma's denoising has its own run-to-run variance of similar magnitude. The capture/replay test below reproduces the mechanism directly instead. So overall, this is a silent bug that likely has some unpredictable impact on output quality. Fix: make `_causal_buf` int32 from construction and update it in place (`.copy_()` / direct assignment) instead of = from a bool source, so build() takes its dtype-matched fast path and never reallocates. Tagging @LucasWilkinson, who authored the original FULL-cudagraph path in #45163. AI-assisted, every line reviewed and tested by the submitter. ## Test Plan `pytest tests/models/test_diffusion_gemma_cudagraph.py -v` ## Test Result New regression test constructs a real DiffusionGemmaModelState and drives _causal_buf through an actual torch.cuda.CUDAGraph capture/replay cycle mirroring FlashAttention's cast. Fails on main: replay returns the stale `[0, 0, 0, 0]` instead of the updated `[1, 1, 1, 1]`. Passes with this patch. Run on H100 (SM90).", "url": "https://github.com/vllm-project/vllm/pull/51994", "createdAt": "2026-08-12T13:49:00Z", "updatedAt": "2026-08-12T18:40:26Z", "timestamp": "2026-08-12T18:40:26Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia" ], "author": "fjosw", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:51998", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "chore: Upstream Cohere parser fixes + tests", "text": "Adding some local fixes for Cohere parsers alongside corresponding tests", "url": "https://github.com/vllm-project/vllm/pull/51998", "createdAt": "2026-08-12T14:04:35Z", "updatedAt": "2026-08-13T12:05:37Z", "timestamp": "2026-08-13T12:05:37Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "ci/build", "tool-calling", "cpu", "nvidia" ], "author": "jasonozuzu-cohere", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52000", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[V1][CUDA graph] Dispatch uniform-decode batches to a padded FULL graph instead of falling to eager PIECEWISE", "text": "**TL;DR:** with speculative decode, some concurrencies silently fall back to running attention *eagerly* every step instead of from a captured CUDA graph, adding a fixed per-step latency bubble. This makes them dispatch to a captured graph instead. **What happens.** When `dispatch()` picks a CUDA graph for a batch, it looks up the smallest captured graph big enough to hold it. With speculative decode there are two kinds of graphs captured at *different* sizes: - **FULL decode** graphs (fast: whole decode step, incl. attention, is captured) - **PIECEWISE** graphs (attention runs eagerly, not captured) Their sizes interleave, so for some batch sizes the smallest graph that fits happens to be a PIECEWISE one — even though a slightly larger FULL decode graph exists and could serve the batch by padding it with a few dummy rows. PIECEWISE always matches, so the batch takes it and pays the eager-attention cost every decode step. **Example** (`decode_query_len = 3`, default capture ladder): | batch (conc) | tokens | smallest graph that fits | result | |---|---|---|---| | conc-4 | 12 | size-16 **PIECEWISE** | eager attention ❌ | | conc-4 (this PR) | 12 | size-18 **FULL decode** (pad 4→6 reqs) | captured ✅ | Symptom on a long-context agentic run: conc-4 / conc-12 show decode ITL p50 ~75 ms — *slower* than conc-24 doing 6× the work — with the GPU stuck at launch-bound power while reading \"100% busy\". **Fix.** FULL decode graphs are ignored by non-decode batches anyway (they only match uniform decode), so we simply offer them — smallest-first — before the PIECEWISE fallback for every batch size. `dispatch()` then picks the smallest FULL decode graph that fits (padding a few requests) and only falls back to PIECEWISE when none does. - No change to `dispatch()` or `_is_compatible()`. - **Zero extra captured graphs** — just reorders the existing candidate lists. - Non-decode and non-spec configs are unaffected. **One assumption:** the attention backend must tolerate padding a uniform-decode batch with dummy rows (4 real → 6). The ROCm AITER MLA backend already does this (`_uniform_padded_mtp_qo_len` in `rocm_aiter_mla.py`). Other backends with a separate FULL decode routine should confirm their metadata builder handles the padded rows. ## Test Plan ## Test Plan - `python -m py_compile vllm/v1/worker/gpu/cudagraph_utils.py`. - Unit test: build a `CudaGraphManager` with a separate decode routine, `decode_query_len=3`, capture sizes `[8,16,24]`; assert `dispatch(num_reqs=4, num_tokens=12, uniform_token_count=3, num_active_loras=0)` returns a FULL decode descriptor (not PIECEWISE / NONE). - E2E: Kimi-K3 TP8 on MI355X, spec decode (`num_spec=2`), `FULL_AND_PIECEWISE`, long-context agentic sweep; compare conc-4 / conc-12 decode ITL before vs after. ## Test Result - `py_compile` — clean. - Kimi-K3 MI355X: conc-4 / conc-12 previously dropped to eager PIECEWISE (ITL p50 ~75 ms, GPU launch-bound ~370 W); with this change they dispatch to the size-18 / size-42 FULL decode graphs, removing the bubble. - On-box before/after ITL numbers to be pasted here; consider opening as **draft** until measured.", "url": "https://github.com/vllm-project/vllm/pull/52000", "createdAt": "2026-08-12T14:13:19Z", "updatedAt": "2026-08-12T23:18:39Z", "timestamp": "2026-08-12T23:18:39Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "nvidia", "mrv2", "verified" ], "author": "xiaohuguo2023", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52001", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kimi-K3] Use MTP draft model config", "text": "## Bug Kimi-K3 constructs its MTP model with the target model's `hf_text_config` instead of the speculative draft model's config. The target config can legitimately have `num_nextn_predict_layers = 0`, while the MTP draft config has the actual draft layers (5 in our setup). Kimi therefore initializes `self.num_mtp_layers` to zero. During the profile run, MTP step selection executes: ```python spec_step_idx % self.num_mtp_layers ``` and startup fails with: ```text ZeroDivisionError: integer modulo by zero ``` ## Fix Read `hf_text_config` from `speculative_config.draft_model_config` in the AMD and NVIDIA Kimi-K3 MTP constructors. This PR intentionally changes only: - `vllm/models/kimi_k3/amd/mtp.py` - `vllm/models/kimi_k3/nvidia/mtp.py` - one focused regression test It does not change URLs, test asset paths, the generic EAGLE loader, or local-argmax behavior. ## Reproduction and A/B test The regression test creates: - target config: `num_nextn_predict_layers = 0` - draft config: `num_nextn_predict_layers = 5` Before the fix: - Kimi selects the target config - `0 % 0` raises `ZeroDivisionError` After the fix: - Kimi selects the draft config - `0 % 5 == 0` Run: ```bash pytest -q tests/models/kimi_k3/test_mtp.py ``` Result: `2 passed` (AMD and NVIDIA). Also checked Ruff formatting/lint and `git diff --check`.", "url": "https://github.com/vllm-project/vllm/pull/52001", "createdAt": "2026-08-12T14:15:50Z", "updatedAt": "2026-08-13T05:02:40Z", "timestamp": "2026-08-13T05:02:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "rocm", "speculative-decoding", "ci/build", "multi-modality", "deepseek", "cpu", "kv-connector", "mrv2", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52003", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Mypy Fix] Mypy fix for \"vllm/model_executor/models/[cC][dD]\"", "text": "## Purpose Mypy fix for \"vllm/model_executor/models/[cC][dD]\" ## Test ```bash pre-commit run --hook-stage manual mypy-3.13 -a Run mypy for Python 3.13.................................................Passed ```", "url": "https://github.com/vllm-project/vllm/pull/52003", "createdAt": "2026-08-12T14:55:21Z", "updatedAt": "2026-08-13T03:33:57Z", "timestamp": "2026-08-13T03:33:57Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "speculative-decoding", "ready", "multi-modality", "deepseek" ], "author": "yewentao256", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52005", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix .../mrope.py::apply_interleaved_rope() when torch.compile is used in torch==2.13", "text": "## Purpose This PR fixes incorrect outputs of `torch.compile` `.../mrope.py::apply_interleaved_rope()` when it's used with `torch==2.13` (which newest vLLM uses), In `torch==2.11` it worked correctly. We fix it by computing the same output in a way that `torch.compile` doesn't break . This method does only indexing and assigning so compiling it should not introduce errors. Added test case, comparing eager to `torch.compile` without updated function test case returns. ``` E Mismatched elements: 98229 / 524288 (18.7%) E Greatest absolute difference: 6.25 at index (3749, 9) E Greatest relative difference: 32512.0 at index (1541, 5) ``` [Issue](https://github.com/pytorch/pytorch/issues/193183) in pytorch github ## Test Plan ``` python -m pytest tests/kernels/core/test_mrope.py::test_apply_interleaved_rope tests/kernels/core/test_mrope.py::test_apply_interleaved_rope_torch_compile ``` Additionally it can be tested by running request to `nvidia/Cosmos3-Edge` (need to apply fixes from [PR51989](https://github.com/vllm-project/vllm/pull/51989)), server started with `VLLM_DISABLE_COMPILE_CACHE=1 vllm serve nvidia/Cosmos3-Edge --reasoning-parser nemotron_v3`: ``` curl -sS http://localhost:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ --data-binary @- <<'JSON' | { \"model\": \"nvidia/Cosmos3-Edge\", \"messages\": [ { \"role\": \"user\", \"content\": [ { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" } }, { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg\" } }, { \"type\": \"text\", \"text\": \"Identify the images independently. In the final answer, write exactly two short lines. Start the first line with 'Image 1:' and the second line with 'Image 2:'. Do not return JSON.\" } ] } ], \"max_tokens\": 1024, \"temperature\": 0, \"stream\": false, \"chat_template_kwargs\": { \"enable_thinking\": true } } JSON jq -r '.choices[0].message.content' ``` With the fix it outputs: ``` Image 1: A bee on a pink flower in a garden setting. Image 2: A rabbit in a blue coat standing on a dirt path with flowers and a house in the background. ```` Without: ``` Image 1: A rabbit in a blue coat standing on a path with flowers. Image 2: A cartoon rabbit dressed as a boy on a dirt path with a house and hills. ``` ## Test Result ``` 2 passed, 14 warnings in 0.77s ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52005", "createdAt": "2026-08-12T15:03:07Z", "updatedAt": "2026-08-13T14:00:49Z", "timestamp": "2026-08-13T14:00:49Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "torch.compile", "verified" ], "author": "bastefaniak", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52007", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI Bug] Fix ci qwen3.5", "text": "## Purpose Forward fix for https://github.com/vllm-project/vllm/pull/51908 Fixes https://buildkite.com/vllm/ci/builds/83443#019ff660-8010-40e8-851d-bb6479f24c68 ## Test Covered in CI", "url": "https://github.com/vllm-project/vllm/pull/52007", "timestamp": "2026-08-12T17:22:12Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready", "qwen" ], "author": "yewentao256", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:52009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI Bug] Fix ci moe test", "text": "## Purpose Fixes https://buildkite.com/vllm/ci/builds/83443#019ff2a1-641e-4d2b-bca9-9eda8a060573 There are two errors here: 1. vLLM side, we name it triton test, but actually running the flashinfer path, this PR fixes the issue 2. the root cause of flashinfer is a bug upstream with TRT-LLM BF16 MoE, we may wait for their fix, not related to this PR ## Test Covered in CI", "url": "https://github.com/vllm-project/vllm/pull/52009", "createdAt": "2026-08-12T15:13:37Z", "updatedAt": "2026-08-12T21:28:54Z", "timestamp": "2026-08-12T21:28:54Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "yewentao256", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52010", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Stop running shared experts on the aux stream", "text": "## Purpose `LM Eval Qwen3-5 Models` on MI355 fails on `Qwen3.5-35B-A3B-DEP2`: ``` FAILED evals/gsm8k/test_gsm8k_correctness.py::test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] AssertionError: GSM8K metric too low: 0.0182 < 0.8400 - 0.0300 = 0.8100 ``` Bisect over the window in which the job turned red lands on 47ececb5, \"[Perf][ROCm] Dual-stream decode with hipgraphs\" (#48223). That commit widened the shared-expert multi-stream overlap from `current_platform.is_cuda()` to `is_cuda_alike()`, gated on `dp_size > 1`, so the shared experts run on the aux stream on ROCm for the first time. The output is corrupted there: the server comes up clean and then answers with unparsable runs of tokens. `dp_size > 1` is also why only this config broke — the other two Qwen3.5 models in the same job run TP2 and never take the path. **This is a workaround, not a fix.** It turns the overlap off on ROCm, restoring the behaviour every ROCm deployment had before #48223, and leaves CUDA untouched. The new path itself still needs to be repaired, and I am happy to help with that; what is known so far: - It needs concurrency. A single greedy request answers correctly. - It is not tied to capturing the whole decode step: `cudagraph_mode=PIECEWISE` still fails. - It cannot be observed from Python, since graph replay bypasses that path, and instrumentation serialises enough to hide it. ## Test Plan gfx950 (MI355X), 2 GPUs, the failing job as CI runs it, unmodified config and full question set: ``` pytest -s -v evals/gsm8k/test_gsm8k_correctness.py \\ --config-list-file=configs/models-qwen35-mi355.txt ``` Run on current main and again with this change. ## Test Result Before, on current main: `Qwen3.5-35B-A3B-DEP2` fails, accuracy 0.0000 with an invalid rate of 0.705. After, same config and the full 1319 questions: ``` GSM8K Results for Qwen/Qwen3.5-35B-A3B: Measured metric: 0.8613 Expected metric: 0.8400 Tolerance: 0.0300 Questions: 1319 Invalid rate: 0.000 ✅ GSM8K test passed for Qwen/Qwen3.5-35B-A3B ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52010", "timestamp": "2026-08-12T17:06:33Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "stefankoncarevic", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:52014", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KVConnector][MoRIIO] Fix MoRIIO port collisions for deployments using both DP and TP", "text": "## Purpose Fix MoRIIO port collisions for deployments using both DP and TP. Port offsets now use `dp_rank * tp_size + tp_rank`, where `tp_size` belongs to the listening peer. This also handles heterogeneous TP, asynchronous WRITE completion, and the unknown-TP sentinel correctly. ## Test Plan Run the focused MoRIIO unit tests ## Test Result Unit tests passed ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52014", "createdAt": "2026-08-12T15:51:54Z", "updatedAt": "2026-08-13T01:42:15Z", "timestamp": "2026-08-13T01:42:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "kv-connector" ], "author": "zzaebok", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52015", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][MRV2] Support pipeline parallelism for DiffusionGemma", "text": "## Purpose - Support pipeline parallelism for DiffusionGemma (follow-up to #45828) - Broadcast the sampler-owned per-step state to the other ranks; the initial canvas is seeded per request, so all ranks agree without a broadcast - Fix a PP deadlock in the need-sampled mask for models that roll back `num_computed_tokens` - Harden `PPHandler.broadcast` against in-place overwrite during the async send - Not a duplicate: no open PR or issue touches DiffusionGemma PP ## Test Plan - `pytest tests/v1/worker/test_diffusion_pp_state.py -v` - PP=2 vs PP=1 GSM8K A/B on Modal 2x L40S (FP8-dynamic 26B, canvas 256) - PP=1 regression: serve + KV cache size vs main ## Test Result - 2 passed, 0 failed | | PP=1 | PP=2 | |---|---|---| | GSM8K 50q (3 runs) | 88 / 94 / 96% | 90 / 96 / 94% | | Throughput (q/s) | 4.35 | 5.41 | - PP=1 unchanged: KV cache size identical to main, outputs coherent - State broadcast is ~22 MB/step at 8 requests, same order as inter-stage hidden traffic - Verified at PP=2; deeper pipelines untested --- AI assistance was used for this change (Claude); every line was reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/52015", "createdAt": "2026-08-12T15:59:25Z", "updatedAt": "2026-08-13T10:12:30Z", "timestamp": "2026-08-13T10:12:30Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "mrv2" ], "author": "guan404ming", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X dense linear backends", "text": " ## Purpose This PR integrates [B12X](https://github.com/local-inference-lab/b12x) dense linear kernels for NVIDIA SM120 and SM121 GPUs through the existing vLLM linear backend interfaces. B12X is an optional dependency installed with `vllm[b12x]` and pinned to `b12x==1.2.4`; it is a pure-Python CuTe DSL package and requires no additional vLLM build step. Supported linear paths are: - Per-tensor FP8. - 128x128 block-scaled FP8. - MXFP8. - NVFP4 and MXFP4. B12X participates in automatic kernel selection after established optimized backends and before emulation. Users can also request it with `--linear-backend b12x`; linear families that B12X does not implement, such as dense W4A16, retain normal backend selection so mixed-format models continue to work. Warmup is limited to layers that selected a B12X kernel. It covers every configured CUDA-graph capture size plus `max_num_batched_tokens`. B12X internally deduplicates shapes that select the same kernel policy, while vLLM deliberately avoids duplicating B12X policy-selection heuristics. The documentation covers installation, selection, supported formats, and fallback behavior. This is the linear and shared-integration component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 target FlashInfer-embedded B12X paths or narrower integrations and do not provide this standalone optional linear backend. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused dense-kernel and warmup coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/model_executor/kernels/test_b12x_mxfp4_linear.py \\ tests/model_executor/kernels/test_b12x_nvfp4_linear.py \\ tests/model_executor/kernels/test_b12x_mxfp8_linear.py \\ tests/model_executor/test_b12x_warmup.py \\ tests/kernels/quantization/test_block_fp8.py \\ -k b12x -q ``` Run pre-commit over every changed file: ```bash mapfile -t changed_files < <(git diff --name-only upstream/main...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check upstream/main...HEAD ``` Benchmark `Qwen/Qwen3.6-27B-FP8` on an RTX PRO 6000 Blackwell Max-Q GPU, holding the attention backend constant and comparing B12X with CUTLASS. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10 at the exact rebased PR head: ```text 52 passed, 473 deselected in 24.86s ``` All changed-file pre-commit hooks passed, including Ruff, formatting, mypy, and markdownlint; `git diff --check` also passed. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | Dense block FP8 | Qwen3.6-27B-FP8 | 1 | CUTLASS | 54.3975 | 56.7518 | +4.33% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52016", "createdAt": "2026-08-12T16:04:49Z", "updatedAt": "2026-08-13T16:57:44Z", "timestamp": "2026-08-13T16:57:44Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "ready", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52017", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X causal paged attention backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency and shared lazy-import integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) causal paged-attention backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing attention backend interface. It does not modify generic model-runner behavior or introduce a new attention abstraction. Supported paths include: - Causal paged MHA, MQA, and GQA with BF16 model dtype. - BF16 and FP8 E4M3 KV caches. - Prefill, decode, and mixed batches. - Speculative verification, sliding-window attention, and attention sinks. - CUDA graph capture and replay. MLA, non-causal attention, ALiBi, logits soft capping, and context parallelism are unsupported. The documentation covers explicit `--attention-backend B12X_ATTN` selection, supported cache formats and head sizes, and unsupported attention modes. This is the causal-attention component split from #51696, which it supersedes rather than duplicates. Related open work targets different backend boundaries and does not provide this standalone optional-package attention integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused backend and reference-correctness coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/v1/attention/test_b12x_attn.py \\ tests/v1/attention/test_attention_backends.py \\ -k b12x -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `Qwen/Qwen3-8B` on an RTX PRO 6000 Blackwell Max-Q GPU with BF16 model weights, TP1, an 8192-token input, and a 512-token output. Hold linear execution constant and compare B12X with FlashInfer using both BF16 and FP8 E4M3 KV caches. Use three warmup and ten measured requests per result. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 41 passed, 124 deselected in 38.59s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | BF16-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 88.5677 | 89.6471 | +1.22% | | FP8-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 91.0632 | 92.3668 | +1.43% | The attention measurements exclude prefill and use 5,110 post-first-token outputs across ten requests per result. All 40 measured requests completed successfully. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52017", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:57:19Z", "timestamp": "2026-08-13T16:57:19Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52018", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X FP4 MoE backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency, shared lazy imports, packed-storage reuse, and warmup integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) FP4 MoE backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing fused-MoE backend interfaces. It does not introduce a new MoE abstraction. Supported paths include: - Native NVFP4 and MXFP4 W4A4. - W4A16 and supported dynamic W4A8 activation modes. - MXFP4 defaults to MXFP8 activations where supported; NVFP4 retains its checkpoint activation format. - `VLLM_B12X_MOE_FP4_FORCE_A16=1` forces both FP4 formats onto the BF16 activation path. - Tensor parallelism, warmup, reload-stable packed storage, and CUDA graph replay. Expert parallelism, expert maps, EXL3, and NF3 are unsupported. B12X 1.2.4 owns and registers the custom operator used by the backend. Warmup supplies the complete set of serving sizes available to vLLM. B12X deduplicates sizes that resolve to identical internal policy choices; vLLM does not duplicate or guess those package-internal policies. The documentation covers explicit `--moe-backend b12x` selection, supported activation modes, the A16 override, and unsupported expert configurations. This is the MoE component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 do not provide this standalone optional-package MoE integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the B12X MoE backend-selection, correctness, warmup, storage-reuse, and CUDA graph coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/kernels/moe/test_b12x_moe.py -q .venv/bin/python -m pytest \\ tests/model_executor/test_b12x_warmup.py \\ tests/quantization/test_auto_round.py::test_inc_mxfp4_moe_method_registers_weights_and_builds_kernel \\ -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `nvidia/MiniMax-M2.7-NVFP4` on an RTX PRO 6000 Blackwell Max-Q GPU, comparing B12X with FlashInfer CUTLASS MoE at TP2. Use five warmup and ten measured requests. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 54 MoE backend tests passed in 23.63s 2 warmup and weight-lifecycle tests passed in 1.09s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | NVFP4 MoE | MiniMax-M2.7-NVFP4 | 2 | FlashInfer CUTLASS MoE | 132.1558 | 137.8276 | +4.29% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52018", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:57:21Z", "timestamp": "2026-08-13T16:57:21Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build", "quantization" ], "author": "lukealonso", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52019", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Skip stage2 for single-split Triton MLA decode", "text": "## Purpose When grouped Triton MLA uses one KV split, its second-stage reduction has one input and is mathematically an identity. This PR lets stage1 write the final output and LSE directly and skips the stage2 launch. The fast path is restricted to MLA shapes for which the normal performance heuristic selects one split. Multi-split attention, MHA/GQA, and long contexts where batch invariance alone forces one split keep the existing two-stage path. The existing scratch workspace remains reserved, so this reduces HBM traffic and one launch when eligible but does not reduce reserved memory capacity. The final implementation reuses pointer, stride, and mode slots already present in the grouped kernel. This keeps the fallback Triton launch parameter list identical to the pre-patch implementation. For empty/padded rows, the direct path writes the attention-combine neutral values `output=0` and `LSE=-inf`. The old identity stage happened to produce `NaN/-inf` for those rows; they are not consumed as ordinary outputs, and the new values are safe for DCP merging. ## Duplicate-work check I searched open PRs for Triton MLA, single split, direct output, and stage2 and found no implementation of this optimization. Related work such as #33529 (split-selection/query-load tuning) and #42885 (full CUDA graph capture) is complementary; neither writes single-split MLA results directly nor skips the identity stage2 reduction. ## Test plan ```bash .venv/bin/python -m pytest -q tests/kernels/attention/test_triton_decode_attention.py .venv/bin/pre-commit run --files \\ vllm/v1/attention/ops/triton_decode_attention.py \\ vllm/v1/attention/backends/mla/triton_mla.py \\ tests/kernels/attention/test_triton_decode_attention.py git diff --check ``` The tests cover BF16/FP16 output, BF16/FP8 KV cache, active-row bitwise equivalence with the two-stage oracle, empty rows, grouped/multi-split fallback, and CUDA graph replay with changed sequence lengths. The behavior test was also run before implementation and failed because stage2 was still called once. ## Cross-platform validation The RTX PRO 6000 and H20 runs are independent, peer validations using DeepSeek-V2-Lite-Chat and `TRITON_MLA`. They use the same 15-shape kernel set: 13 shapes for which the production heuristic selects one split (`max_seq_len < 1024` on both GPUs), plus forced-single-split `B32/S2048` and `B64/S4096` controls. The tested revisions must be kept explicit: the RTX PRO 6000 was rerun on the current final source, `fa0bb68`; the H20 run evaluated the initial direct-output revision, `39b3e04`. The H20 results therefore validate the direct fast path on Hopper and identify the initial fallback-launch overhead, but they do not claim an H20 rerun of the final slot-reuse implementation. | | NVIDIA RTX PRO 6000 Blackwell Server Edition | NVIDIA H20 | | --- | --- | --- | | Architecture | sm120, 188 SMs | sm90, 78 SMs | | Driver | 580.159.03 | 535.161.08 | | PyTorch / Triton / CUDA | 2.11.0+cu130 / 3.6.0 / 13.0 | 2.11.0+cu130 / 3.6.0 / 13.0 | | Tested direct revision | **`fa0bb68` (final)** | **`39b3e04` (initial)** | | Fast-path baseline | true pre-patch two-stage source (`6accb779`) | true pre-patch kernel file matching `324f452f`, ported to the installed vLLM tree | | Microbenchmark repetition | 3 fresh processes; 12 balanced `A B C C B A` blocks per process | 12 balanced `A B C C B A` blocks | ### Result definitions - **Fast-path speedup** = `true pre-patch two-stage time / direct time`; values above 1 are faster. - **Fast-path saving** = `true pre-patch two-stage time - direct time`; positive values are faster. - **Fallback host delta** = `candidate two-stage host-enqueue time - true pre-patch host-enqueue time`; positive values are overhead. - **Fixed-work throughput change** is derived from inverse latency; it is not an online-serving throughput measurement. - RTX PRO 6000 kernel aggregates are the median of the three per-process paired medians. H20 aggregates are from its balanced three-arm run. Absolute latency is not compared across GPUs. For the three-arm microbenchmarks, `A` is true pre-patch two-stage, `B` is the tested revision with direct output disabled, and `C` is the tested revision with direct output enabled. ### Correctness | Check | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | Target kernel tests | 121 passed, 16 warnings | 121 passed | | Common 15-shape output and LSE vs true pre-patch | bitwise equal in 45/45 process-shape cases for both final arms | bitwise equal in 15/15 shapes | | Natural greedy generation, 16 prompts | identical token arrays across all three arms; 512 token IDs total, 209 distinct IDs | identical token IDs and text; 471 token IDs total, 248 distinct IDs | All applicable pre-commit hooks passed on the final source, and `git diff --check` passed. ### Kernel performance The graph region includes the same output-zeroing work used by the production path. | 13 production-eligible shapes | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | Graph speedup | **1.021x-1.152x** | **1.020x-1.144x** | | Graph saving | **1.667-2.365 us** | **1.584-3.309 us** | | Eager speedup | **1.044x-1.651x** | **1.023x-1.436x** | | Eager saving | **2.064-14.459 us** | **2.467-10.855 us** | The two control shapes were benchmarked with one split even though the production heuristic selects multiple splits: | Forced-single-split control | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | `B32/S2048`, graph / eager | 1.010x (+1.719 us) / 1.023x (+4.162 us) | 1.005x (+1.336 us) / 1.007x (+1.841 us) | | `B64/S4096`, graph / eager | 0.998x (-0.693 us) / 1.002x (+0.711 us) | 1.002x (+1.163 us) / 1.005x (+2.550 us) | For the RTX PRO 6000 `B64/S4096` graph control, per-process savings ranged from -1.262 to +0.359 us, so the small median regression is not treated as a stable architecture-level effect. The control results are not uniformly beneficial, and the production guard remains conservative rather than forcing one split for long contexts. ### End-to-end decode Both workloads used DeepSeek-V2-Lite-Chat, `TRITON_MLA`, `FULL_DECODE_ONLY`, `max_model_len=512`, batch 32, 64 forced decode tokens, 3 warmups, and 12 timed samples per process. The RTX PRO 6000 used two fresh processes per arm in `A-B-C-C-B-A` order; its estimate is the median of the two process medians and the bracketed values are their range. The H20 used an independent two-stage/direct/two-stage run. | Metric | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | True pre-patch median estimate | 481.402 ms [479.585, 483.220] | 390.572 ms (midpoint of 389.971 and 391.172 ms) | | Final two-stage median estimate | 481.587 ms [480.752, 482.423] | not separately measured in E2E | | Direct median estimate | 476.560 ms [475.119, 478.001] | 385.849 ms | | Direct saving vs true pre-patch | **4.842 ms** | **4.723 ms** | | Latency change vs true pre-patch | **-1.006%** | **-1.209%** | | Fixed-work inverse-latency change | **+1.016%** | **+1.224%** | | Pre-patch process/arm drift | +0.758% | +0.308% | | Stage1 / stage2 launches | 1,701 / 1,701 pre-patch and final two-stage; 1,701 / 0 direct | 1,701 / 1,701 pre-patch; 1,701 / 0 direct | On the RTX PRO 6000, both mirrored halves independently favored direct output; direct versus the final-source two-stage arm was also **-1.044%**. Profiling measured target-kernel totals of 18.318 ms for true pre-patch, 18.406 ms for final two-stage, and 14.765 ms for direct output. The H20 direct samples were completely separated from its two baseline sample ranges. The forced-token arrays only establish identical fixed work because token ID 198 was the sole allowed token; natural-token correctness is reported separately above. Cross-GPU absolute latency differences are not used to infer the size or cause of the speedup. ### Fallback launch audit The initial revision added five direct-only runtime values to every stage1 launch. The H20 experiment found host-enqueue overhead on `write_direct=False`; the same issue was reproduced independently on the RTX PRO 6000. The final revision reuses existing grouped-kernel slots and restores the pre-patch runtime/constexpr signature. Values below are fallback host deltas as defined above, reported for `B32/S128`, `B32/S512`, and `B32/S2048` respectively: | Two-stage implementation | NVIDIA RTX PRO 6000 | NVIDIA H20 | | --- | ---: | ---: | | Initial `39b3e04` | **+2.681 / +2.726 / +2.821 us** | **+2.342 / +2.246 / +1.848 us** | | Final `fa0bb68` | **+0.084 / +0.208 / +0.187 us** | not rerun; no final-H20 claim | On the RTX PRO 6000, final-minus-pre-patch CUDA-event deltas for the GPU-bound S512/S2048 cases were +0.013/-0.010 us. On H20, the initial revision's two-stage graph-region deltas across the production shapes were -0.048 to +0.016 us. These measurements support a host argument-packing cause for the initial overhead, rather than a fallback-kernel GPU regression. On the PRO 6000, the final source reduces the central host deltas from about 2.7-2.8 us to at most 0.21 us. ## AI assistance OpenAI Codex assisted with code analysis, implementation, validation and benchmark scripting, and drafting this PR description. --- - [x] The purpose of the PR is described above. - [x] The test plan includes the commands and workloads used. - [x] Test, model-output, before/after performance, and fallback-regression results are included. - [x] Duplicate-work and AI-assistance disclosures are included. - [x] Documentation changes are not required because this does not change a user-facing API or supported-model surface.", "url": "https://github.com/vllm-project/vllm/pull/52019", "createdAt": "2026-08-12T16:31:12Z", "updatedAt": "2026-08-13T13:16:40Z", "timestamp": "2026-08-13T13:16:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "BabyDrangoner", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52020", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Structured Outputs] Reject empty JSON schemas", "text": "## Purpose Fix #52011 by rejecting an empty JSON Schema before structured-output backend selection. The empty schema `{}` is a valid universal JSON Schema, but it leaves number generation unconstrained. If decoding enters the number branch, it may continue producing digits until `max_tokens` and return an unusable response with `finish_reason=\"length\"`. This intentionally changes empty schemas from an unconstrained request to an HTTP 400 validation error. It covers dictionary input (`json={}`), serialized input (`json=\"{}\"`), and serialized input with surrounding whitespace. Callers should provide a concrete non-empty schema, or use `json_object=True` only when any JSON object matches the intended output. The Python and Rust frontends now apply the same check before backend compilation or model execution. The Rust request boundary covers raw `structured_outputs`, OpenAI `response_format`, and the gRPC structured-output path. Non-empty schemas continue to pass request validation, and a failed request does not prevent subsequent requests from completing. This does not duplicate #40099, #51450, #45346, or #47176: those changes respectively address general repetition detection, whitespace-only Rust frontend strings, `json=\"\"`/`json_object=False`, and other Rust validation parity gaps such as null schemas and mutually exclusive constraints. Open PR #51450 touches the same Rust protocol file, but its check only uses `value.trim().is_empty()`; it neither rejects an object `{}` nor parses serialized `\"{}\"`. A duplicate check on 2026-08-13 found no assignee, claim comment, linked pull request, or open pull request implementing empty-schema rejection. AI assistance from OpenAI Codex was used to investigate the issue, implement the validation, add tests and documentation, and prepare this description. I reviewed every changed line, validated the behavior end to end, and ran the tests listed below. ## Test Plan - `.venv/bin/python -m pytest tests/v1/structured_output/test_validation.py -v` - `.venv/bin/python -m pytest tests/v1/structured_output -m cpu_test -q` - `.venv/bin/python -m pytest tests/entrypoints/openai/chat_completion/test_chat_completion.py::test_empty_json_schema_returns_bad_request -v -s` - `cd rust && cargo fmt --all --check` - `cd rust && cargo test -p vllm-engine-core-client` - `cd rust && cargo test -p vllm-server` - `cd rust && cargo clippy -p vllm-engine-core-client -p vllm-server --tests -- -D warnings` - `git diff --name-only -z origin/main...HEAD | xargs -0 .venv/bin/pre-commit run --files` - Manual OpenAI API smoke tests against both the Python and Rust frontends with Qwen2.5-1.5B-Instruct, covering dictionary and serialized empty schemas followed by a normal request. - `git diff --check origin/main...HEAD` ## Test Result - On unmodified `main`, a direct validation probe accepted `{}`, `\"{}\"`, and a whitespace-padded `\"{}\"`. - Focused validation tests: `9 passed, 14 warnings in 3.91s`. - Structured-output CPU tests: `25 passed, 23 deselected, 14 warnings in 9.38s`. - OpenAI API regression test: passed. - Rust engine-core-client tests: `99 passed`; Rust server tests: `342 passed`; Clippy with warnings denied and `cargo fmt --check` both passed. - Manual OpenAI API smoke tests with Qwen2.5-1.5B-Instruct produced the same result on both frontends: dictionary `{}` returned HTTP 400, serialized `\"{}\"` returned HTTP 400, and the immediately following normal request returned HTTP 200. - All applicable pre-commit hooks passed, including Ruff, markdownlint, typos, mypy for Python 3.10, SPDX, forbidden-import checks, and configuration validation, including Rust formatting. - `git diff --check origin/main...HEAD` passed. Model evaluation is not applicable because the affected requests fail during validation before model execution. Successful request generation is not modified. User-facing behavior and the compatibility tradeoff are documented in `docs/features/structured_outputs.md`. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR and the issue it resolves are described above. - [x] Reproducible unit, integration, lint, and manual test steps are listed. - [x] Before-and-after validation and HTTP results are listed. - [x] The necessary structured-output documentation is updated.
", "url": "https://github.com/vllm-project/vllm/pull/52020", "timestamp": "2026-08-12T16:42:03Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "documentation", "structured-output", "rust" ], "author": "fyrsta7", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:52021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve Anthropic disable_parallel_tool_use", "text": "## Purpose Anthropic’s `disable_parallel_tool_use` was silently discarded, leaving the converted OpenAI request with `parallel_tool_calls=True`. This preserves the field and maps it to the existing inverse OpenAI setting. ## Reproducer ```python request = AnthropicMessagesRequest( model=\"test\", messages=[{\"role\": \"user\", \"content\": \"Use a tool\"}], max_tokens=8, tool_choice={\"type\": \"auto\", \"disable_parallel_tool_use\": True}, ) converted = AnthropicServingMessages._convert_anthropic_to_openai_request(request) print(request.tool_choice.model_dump(exclude_none=True)) print(converted.parallel_tool_calls) ``` **On Main** ```text {'type': 'auto'} True ``` **On this branch** ```text {'type': 'auto', 'disable_parallel_tool_use': True} False ``` ## Test Plan and Results ```bash .venv/bin/python -m pytest tests/entrypoints/anthropic/test_anthropic_messages_conversion.py -q ``` ```text # 51 passed ```", "url": "https://github.com/vllm-project/vllm/pull/52021", "createdAt": "2026-08-12T16:49:55Z", "updatedAt": "2026-08-13T13:28:20Z", "timestamp": "2026-08-13T13:28:20Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "bug", "frontend", "ready" ], "author": "taneem-ibrahim", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52022", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Fix store threshold admission counting", "text": "## Purpose Fix `CPUOffloadingManager.store_threshold` admission so it measures distinct requests that actually produce storable KV blocks. Previously, the reuse counter was updated by `lookup()`. Prefix lookup stops at the first miss, so later store candidates in the same request were not observed. With `store_threshold >= 2`, reusable blocks could repeatedly reach `prepare_store()` without ever becoming eligible for offload. This change moves observation to `prepare_store()`, after already-stored blocks are removed. It counts every materialized candidate at most once per request and independently per manager. Per-candidate request tokens are retained only until the threshold is reached, then released. Their total is bounded by the configured tracker size and threshold. The LRU tracker protects candidates already seen in the current request, allowing candidate batches larger than the tracker to make progress across requests. The default behavior is unchanged: thresholds below 2 still bypass admission filtering. ### Relationship to related work This is not a duplicate of an existing open PR: - #50087 refactors the scheduler/store-policy seam and explicitly leaves per-candidate admission to follow-up work. This PR fixes the semantics of the existing `CPUOffloadingManager.store_threshold` implementation. - #44791 filters lazy offload by prefix length in `SimpleCPUOffloadConnector`. It uses a different connector and signal. - #49413 concerns candidate ordering, manager APIs, and event provenance. This fix is limited to the existing single-tier CPU manager and does not change the scheduler or worker protocol. Duplicate checks were repeated immediately before submission: ```text gh issue view 49413 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open \\ --search \"49413 in:body\" gh pr list --repo vllm-project/vllm --state open \\ --search \"KV offload store threshold admission\" gh pr list --repo vllm-project/vllm --state open \\ --search \"store_threshold offload\" gh pr list --repo vllm-project/vllm --state open \\ --search \"distinct requests CPU offload\" ``` No open PR implementing this fix was found. ## Test Plan ```bash PYTHONUTF8=1 .venv/bin/python -m pytest \\ tests/v1/kv_offload/cpu/test_manager.py -v \\ --confcutdir=tests/v1/kv_offload/cpu pre-commit run ruff-check --files \\ vllm/v1/kv_offload/cpu/manager.py \\ vllm/v1/kv_offload/cpu/spec.py \\ tests/v1/kv_offload/cpu/test_manager.py pre-commit run ruff-format --files \\ vllm/v1/kv_offload/cpu/manager.py \\ vllm/v1/kv_offload/cpu/spec.py \\ tests/v1/kv_offload/cpu/test_manager.py pre-commit run markdownlint-cli2 --files \\ docs/features/kv_offloading_usage.md pre-commit run mypy-3.12 --hook-stage manual --files \\ vllm/v1/kv_offload/cpu/manager.py \\ vllm/v1/kv_offload/cpu/spec.py git diff --check ``` ## Test Result - CPU manager tests: **33 passed**. - Exact current-source files in an isolated container: **33 passed**. - Relevant static checks passed: `ruff-check`, `ruff-format`, `typos`, `markdownlint-cli2`, mypy 3.10/3.12, SPDX headers, lazy imports, filenames, forbidden imports, CUDA call validation, config validation, boolean context, suggestions, and `git diff --check`. - The all-hooks run could not install `actionlint` because the test environment could not reach `proxy.golang.org`. The Dockerfile graph hook requires `/bin/bash`, unavailable on the Windows test host. Neither hook applies to the four changed Python/Markdown files. Added regression coverage verifies: - repeated `prepare_store()` calls for one request count each key once; - only materialized store candidates are counted; - managers sharing a request context track candidates independently; - interleaved requests do not double-count the same request; - per-candidate request tokens are released at threshold and with tracker eviction; and - batches larger than `max_tracker_size` make progress instead of churning. ### Model evaluation An end-to-end A/B/A experiment used Qwen3.5-9B BF16 on one NVIDIA L40S with a 16,384-token repeated prefix, a 21,000-token unique polluter after each hot request, 24 GiB of CPU KV offload capacity, and `store_threshold=2`. Across steady-state rounds 3-5: - baseline median TTFT: 1.9259 seconds; - candidate median TTFT reduction: **87.15%**; - case-bootstrap 95% confidence interval: **[86.82%, 88.98%]**; - 60 steady-state paired samples; and - 100/100 generated outputs matched exactly. On a no-reuse workload of 12 unique 16,384-token prompts, the candidate median TTFT was 0.10% slower, with no CPU KV store/load I/O and 12/12 exact output matches. A final three-round smoke used an adapter whose admission-state and admission method ASTs were identical to this current-source patch. It observed the expected sequence: | Round | TTFT | CPU KV activity | |---:|---:|---| | 1 | 3.797 s | observe; no store | | 2 | 1.723 s | store 1,102,970,880 bytes | | 3 | 0.154 s | load 587,857,920 bytes | All three outputs matched. The GPU experiments used the official vLLM v0.26.0 cu129 image because the current source commit's published native wheel requires CUDA 13, while the test host's driver supports the CUDA 12.x line. The full A/B/A run used the same threshold-2 admission semantics on the older interface; the final smoke used the exact final admission core. Current-source unit and static checks ran separately as reported above. ## AI assistance disclosure AI assistance was used for code exploration, implementation, tests, benchmarking, and drafting this description. The human submitter reviewed every changed line, confirmed they understand the change end-to-end, and verified the tests and evaluation results reported above. ---
Essential Elements of an Effective PR Description Checklist - [x] Purpose and relationship to related work are documented. - [x] Test commands are provided. - [x] Unit, static, and model-evaluation results are provided. - [x] User-facing configuration documentation is updated. - [x] AI assistance is disclosed.
", "url": "https://github.com/vllm-project/vllm/pull/52022", "timestamp": "2026-08-12T16:51:50Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "positive666", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:52024", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Perf][ROCm] Dual-stream decode with hipgraphs\"", "text": "Reverts vllm-project/vllm#48223", "url": "https://github.com/vllm-project/vllm/pull/52024", "createdAt": "2026-08-12T17:05:00Z", "updatedAt": "2026-08-13T08:01:42Z", "timestamp": "2026-08-13T08:01:42Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "rocm", "ready" ], "author": "simondanielsson", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52026", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Frontend] Import parser plugins under the vllm.* namespace so their logs survive", "text": "Fixes #52027 ## Purpose `ToolParserManager.import_tool_parser` and `ReasoningParserManager.import_reasoning_parser` import a plugin under a module name taken from its file name, so a plugin doing what every in-tree parser does - `logger = init_logger(__name__)` - ends up with a logger named after its file. Only the `vllm` logger is configured in `DEFAULT_LOGGING_CONFIG`, so that logger has no handler anywhere up its chain: * `logger.info` and `logger.debug` are dropped entirely - Python's `logging.lastResort` fallback is WARNING-level; * `logger.warning` and above leave *outside* vLLM's configuration: unformatted, and unaffected by `VLLM_LOGGING_LEVEL` or `VLLM_LOGGING_CONFIG_PATH`. Importing the plugin as `vllm.tool_parsers.plugins.` (and `vllm.reasoning.plugins.`) puts `__name__` inside the configured namespace, so plugin authors need to do nothing. Two notes on safety: * `import_from_path` uses `spec_from_file_location` and assigns `sys.modules` directly, so the dotted name needs no parent package. * Relative imports inside a plugin were already impossible under a bare module name, so nothing regresses there. ## Test Plan `tests/tool_use/test_plugin_logging.py`, parametrized over both managers: it writes a plugin that records `__name__`, imports it through the manager, and walks the logger chain from that name. ## Test Result The test asserts that a record from the plugin's own logger reaches a handler, and that the handler belongs to the `vllm` logger. Before this change the walk ends at the root with no handler; after it, it stops at `vllm`. Verified in isolation (stdlib only, vLLM's logging config shape): dummy_parser_plugin -> handler: None vllm.tool_parsers.plugins.dummy_parser_plugin -> handler: vllm Found while running a tool parser as a plugin: three parser bugs stayed invisible for a day because every warning the parser emitted was dropped.", "url": "https://github.com/vllm-project/vllm/pull/52026", "createdAt": "2026-08-12T17:19:19Z", "updatedAt": "2026-08-13T11:47:40Z", "timestamp": "2026-08-13T11:47:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "tool-calling" ], "author": "honziik", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52028", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Pin DeepEP by its full commit hash", "text": "## Purpose `tools/ep_kernels/install_python_libraries.sh` pins DeepEP by a 10-character abbreviation: ```bash DEEPEP_COMMIT_HASH=${DEEPEP_COMMIT_HASH:-\"d4f41e4e93\"} ``` An abbreviated object ID is not a ref, so it cannot be fetched directly. GitHub serves any *complete* commit via `allowAnySHA1InWant`, but an abbreviation is never a valid want: ```console $ git fetch --depth 1 origin d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb * branch d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb -> FETCH_HEAD $ git fetch --depth 1 origin d4f41e4e93 fatal: couldn't find remote ref d4f41e4e93 ``` Cloning the whole repository and checking out afterwards resolves the abbreviation locally, which is why this goes unnoticed in the common path. It breaks any consumer that fetches only the pinned commit — a reasonable thing to do for a large dependency — and the resulting error names a network-shaped problem rather than the pin. This is the same hash, written in full. `--deepep-ref` and the `DEEPEP_COMMIT_HASH` environment override are unaffected. ## Test Plan `bash -n tools/ep_kernels/install_python_libraries.sh`, plus the fetch comparison above run against `github.com/deepseek-ai/DeepEP`. ## Test Result The full hash fetches successfully; the abbreviation fails. The commit resolved is unchanged: `d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb` (\"Add fence.proxy.async.shared::cta between mbarrier wait and TMA load. (#642)\").", "url": "https://github.com/vllm-project/vllm/pull/52028", "createdAt": "2026-08-12T17:31:10Z", "updatedAt": "2026-08-12T20:01:33Z", "timestamp": "2026-08-12T20:01:33Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready" ], "author": "tlrmchlsmth", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52030", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix packed GDN decode launch for large batch-head grids", "text": "## Purpose Avoid a CUDA launch failure in packed GDN decode when `batch_size * num_value_heads` exceeds the maximum CUDA grid Y/Z dimension of 65,535. The existing launch is preserved for normal sizes. Only overflowing cases use a split `(value_tiles, value_heads, batch)` grid. ## Test Plan ## Test Result - Verified the failing Qwen shape (`B=1024`, `HV=64`, `K=V=128`) launches successfully. - Running `vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned94 -tp=2` doesn't crash anymore ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52030", "createdAt": "2026-08-12T17:44:32Z", "updatedAt": "2026-08-13T14:19:09Z", "timestamp": "2026-08-13T14:19:09Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "mgoin", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52031", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend][gRPC] Advertise LoRA capabilities", "text": "## Purpose Advertise the Rust frontend's LoRA capability and capacity through the existing engine ready handshake and gRPC discovery APIs. - Report whether LoRA is enabled and the configured maximum number of active adapters from each Python `EngineCoreProc`. - Decode the capability in the Rust engine-core client and reject internally inconsistent or mixed-capability engine ranks during startup. - Expose `max_loras` through `GetServerInfo` and `supports_lora` through `GetModelInfo`. - Preserve the existing inference and LoRA lifecycle behavior; this PR only reports and validates capability metadata. No overlapping open PR or issue was found after searching for Rust frontend LoRA capability reporting and the `supports_lora` and `max_loras` fields. ## Test Plan - `UV_PYTHON=/usr/bin/python3.12 cargo test --manifest-path rust/Cargo.toml -p vllm-engine-core-client` - `cargo test --manifest-path rust/Cargo.toml -p vllm-server --lib 'grpc::tests::'` - `cargo check --manifest-path rust/Cargo.toml -p vllm-server` - `cargo fmt --manifest-path rust/Cargo.toml --all -- --check` - `ruff check vllm/v1/engine/core.py vllm/v1/engine/__init__.py tests/v1/engine/test_engine_core_client.py rust/src/engine-core-client/src/tests/python_compat.py` - `ruff format --check vllm/v1/engine/core.py vllm/v1/engine/__init__.py tests/v1/engine/test_engine_core_client.py rust/src/engine-core-client/src/tests/python_compat.py` - `git diff --check upstream/main...HEAD` ## Test Result - Engine-core client: 97 passed. - Focused gRPC server suite: 31 passed. - Server compilation, Rust formatting, Python lint/format, and whitespace checks passed. - Model evaluation is not applicable because this change does not affect inference behavior, model output, or accuracy. **AI assistance disclosure:** This PR was authored with AI assistance and reviewed by the submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described. - [x] The test plan is included. - [x] The test results are included. - [x] No documentation update is required for this focused capability-reporting change.
", "url": "https://github.com/vllm-project/vllm/pull/52031", "createdAt": "2026-08-12T18:06:10Z", "updatedAt": "2026-08-12T18:10:50Z", "timestamp": "2026-08-12T18:10:50Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rust" ], "author": "connorcarpenter15", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52032", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add opt-in dynamic NVFP4 MoE GEMM2 quantization", "text": "### Summary This adds a default-off runtime option for ordinary per-routed-token NVFP4 quantization of the post-activation GEMM2 input in the FlashInfer TRTLLM MoE backend: ```bash VLLM_FLASHINFER_MOE_NVFP4_DYNAMIC_GEMM2=1 \\ vllm serve MODEL --moe-backend flashinfer_trtllm ... ``` The checkpoint's GEMM1 input remains quantized with its existing static outer scale and group-16 E4M3 block scales to allow working together with EP / EPLB. GEMM1 emits BF16 post-activation rows, FlashInfer computes one live FP32 outer scale per routed row, requantizes those rows to group-16 NVFP4, and runs GEMM2 with the existing W4A4 cubin. When the option is enabled, the checkpoint GEMM2 activation scale is ignored. The default path is unchanged. No FlashInfer or checkpoint change is required. ### Parallelism and state handling - Monolithic and modular TRTLLM expert wrappers share the same scale handling. - TP replicates the GEMM1 outer scale and constructs a rank-local row vector; no scale slicing or collective is needed. - Modular EP keeps `expects_unquantized_inputs=False`, so the established static NVFP4 GEMM1 input and block scales are dispatched normally. The row vector is created from the post-dispatch token count on each rank. - EPLB-visible values remain expert-major registered parameters. The shared GEMM1 outer decode scalar is a non-persistent buffer, so EPLB neither treats it as an expert tensor nor tries to permute it. - Existing online `nvfp4_per_token` remains supported; it already supplies live GEMM1 row scales and automatically uses dynamic GEMM2. However, it does not support EP or EPLB. ### Performance and quality evidence The exact production implementation was exercised on four B200 GPUs using the released Mistral Large 3 NVFP4, TP4, FlashInfer TRTLLM MoE, and concurrency one on an internal long context dataset. The results are as follows: | Checkpoint | Backend | Accuracy | | ------------- | ------------- | ------------- | | [Mistral Large 3 FP8](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512/commit/383ffea2c7d60dfd44ca960e8e691709d4fdb9cd) | Flashinfer TRTLLM | 88% | | [Mistral Large 3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4/commit/c8fe8a02a4b2fb6af0cdeb4ee71fcf147742e59c) | Flashinfer TRTLLM | 66% | | [Mistral Large 3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4/commit/c8fe8a02a4b2fb6af0cdeb4ee71fcf147742e59c) | Flashinfer TRTLLM (DYNAMIC_GEMM2=1) | 87% | ### Tests Successfully ran the following tests: ``` tests/quantization/test_trtllm_nvfp4_dynamic_gemm2.py tests/quantization/test_trtllm_nvfp4_hidden_dim_padding.py tests/test_envs.py tests/distributed/test_eplb_quant_scale_consistency.py tests/kernels/moe/test_trtllm_nvfp4_moe.py tests/quantization/test_trtllm_nvfp4_dynamic_gemm2.py ``` ### Duplicate check Merged PR #48538 is adjacent but different: it adds online weight quantization plus dynamic GEMM1/GEMM2 and restricts that mode to the monolithic path. This change targets already-quantized W4A4 checkpoints, keeps GEMM1 checkpoint-static, and supports the modular EP path. ### AI disclosure AI assistance was used to investigate, implement, test, and draft this change. The human submitter must review every changed line and be prepared to explain and maintain it. [Edited for conciseness] ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52032", "createdAt": "2026-08-12T18:29:36Z", "updatedAt": "2026-08-13T11:49:53Z", "timestamp": "2026-08-13T11:49:53Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia", "quantization" ], "author": "dbari", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52033", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][ROCm] Dual-stream decode with hipgraphs", "text": "## Changes since reverted in https://github.com/vllm-project/vllm/pull/52024 Re-opened https://github.com/vllm-project/vllm/pull/48223 with fix for Qwen3.5-style models (CI failed test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] after merging this PR previously). **Changes made since revert:** https://github.com/vllm-project/vllm/pull/52033/changes/50e80bb721d43c0d032d0a71d44546e3cdedcc6c. Issue stemmed from fact that Qwen3.5 uses the same tensor for hidden state and shared expert inputs, and later on mutates one of them in the routed experts. That causes a race condition during multi-stream. **Validation:** Re-ru nthe test with this config: https://github.com/vllm-project/vllm/blob/fe889ac9255491e82e58012d7787e920c6c7fbc2/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml#L6-L9 ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --max-model-len 4096 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --trust-remote-code \\ --disable-uvicorn-access-log ``` ``` docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c \"pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-35B-A3B,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k\" ``` **Results:** Running GSM8k on Qwen3.5 on MI350 with DP2EP gives (test pass requires acc >= 0.84 - 0.03 = 0.81) |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8522|± |0.0098| | | |strict-match | 5|exact_match|↑ |0.8362|± |0.0102| _From original PR:_ ## Purpose Fixes #48111. Enables (1) dual-stream decode for CUDA-like platforms with proper overlap (2) make them hip/cudagraph compatible. Only enabled on ROCm when using DP, as we observed [performance regression under TP](https://github.com/vllm-project/vllm/pull/48223#issuecomment-4993855510). Mutually exclusive with `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS`. Disable with `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` as usual. **Gain:** About -3-4% TPOT on 1k/1k and 8k/1k when using DPA vs default. **Note:** When Fused Shared experts (FSE) can be used, it provides even better TPOT than dual-stream. However, FSE cannot be used everywhere (like with MoRI a2a kernels). ### Technical details - Simply allowing `MULTI_STREAM_OVERLAPPED` for CUDA-like platforms is not sufficient for performant dual-stream decode (on ROCm, at least), because the shared expert was launched on the aux stream only _after_ the routed experts had completed. That leads to a sequential launch with no overlap: \"image\" - Instead, we need to launch the shared expert in the aux stream _before_ the dispatch. #### After this PR we get the following traces - Pure decodes: properly overlapped \"image\" - Mixed batches: no aux stream \"image\" - Pure prefills: no aux stream \"image\" ## Test Plan Tested on 8xMI300. 1. Run DSv3 with DPA+TP: ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` 2. Bench serve sweep, 1k/1k and 8k/1k ISL/OSL - 10x as many prompts as the concurrency level; 2x number of warmups. - Comparing to nightly `vllm/vllm-openai-rocm:nightly-a16dbd5b8572d4128be9f10b9dcff4999b594b25` with fused shared experts ```bash vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 \\ --num-warmups 256 ``` 4. GSM8k accuracy non-regression ```bash lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result - Couple % improved TPOT ### 1k/1k (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 232.57 | 23.67 | **+4.24%** | | 1 | nightly | 0.04 | 273.16 | 24.72 | - | | 2 | this branch | 0.08 | 258.38 | 24.16 | **+4.58%** | | 2 | nightly | 0.08 | 298.14 | 25.32 | - | | 4 | this branch | 0.15 | 397.90 | 25.52 | **+7.03%** | | 4 | nightly | 0.14 | 366.23 | 27.45 | - | | 8 | this branch | 0.25 | 602.93 | 30.84 | **+3.41%** | | 8 | nightly | 0.24 | 664.92 | 31.93 | - | | 16 | this branch | 0.44 | 1268.65 | 35.02 | -6.77% | | 16 | nightly | 0.46 | 851.06 | 32.80 | - | | 32 | this branch | 0.71 | 1830.04 | 42.03 | **+2.75%** | | 32 | nightly | 0.70 | 1698.74 | 43.22 | - | | 64 | this branch | 1.34 | 3228 | 43.51 | **+4.71%** | | 64 | nightly | 1.29 | 2528 | 45.66 | - | | 128 | this branch | 2.15 | 4487.07 | 53.85 | **+1.86%** | | 128 | nightly | 2.10 | 4543.42 | 54.87 | - | | 256 | this branch | 3.46 | 6554.86 | 65.74 | **+3.12%** | | 256 | nightly | 3.37 | 6386.54 | 67.86 | - | Note: TTFT numbers seems highty variable. Ran 64 concurrency yet another time, and then I got 1902 ms TTFT @ 1.37 QPS, substantially better than the first time. This PR should not affect prefills anyhow so not focusing too much on TTFT here anyways. ### 8k/1k | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 1868.16 | 25.22 | **+3.59%** | | 1 | nightly | 0.04 | 1860 | 26.16 | - | | 2 | this branch | 0.07 | 2034.06 | 25.42 | **+5.47%** | | 2 | nightly | 0.07 | 2014.35 | 26.89 | - | | 4 | this branch | 0.11 | 3800.62 | 31.35 | **+2.82%** | | 4 | nightly | 0.11 | 3687 | 32.26 | - | | 8 | this branch | 0.19 | 4105.90 | 37.01 | **+5.54%** | | 8 | nightly | 0.18 | 4122.95 | 39.18 | - | | 16 | this branch | 0.31 | 6355 | 45.37 | -3.07% | | 16 | nightly | 0.31 | 6321 | 44.02 | - | | 32 | this branch | 0.45 | 9616.31 | 60.48 | **+11.44%** | | 32 | nightly | 0.41 | 9173.59 | 68.29 | - | | 128 | this branch | 0.85 | 15579.93 | 131.24 | -3.71% | | 128 | nightly | 0.89 | 13966.30 | 126.55 | - | ### GSM8k This branch: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9416|± |0.0065| | | |strict-match | 5|exact_match|↑ |0.9409|± |0.006| Nightly: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52033", "createdAt": "2026-08-12T18:30:25Z", "updatedAt": "2026-08-13T15:19:49Z", "timestamp": "2026-08-13T15:19:49Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm" ], "author": "simondanielsson", "state": "open", "assignees": [ "shen-shanshan" ] }, { "id": "github:vllm-project/vllm:pull_request:52035", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Update DeepGEMM pin to deepseek-ai nv_dev tip", "text": "## Purpose Update both DeepGEMM pins (`cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh`, which are documented to stay in sync) from the `vllm-project/DeepGEMM` fork at `e21c821f` to upstream `deepseek-ai/DeepGEMM` at `8b1392b978f5a03c828dd1711090d7fb50958b8a`, the current tip of the `nv_dev` branch. The fork pin was kept because the plain `nv_dev` branch previously lacked SiTU support (see the removed TODO comment in the cmake file). The `nv_dev` tip now carries SiTU for FP8xFP4 MegaMoE (`situ_beta` / `situ_linear_beta` in `csrc/apis/mega.hpp`, `deep_gemm/mega/__init__.py`, `csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp`, `deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh`) alongside the SM120/SM121 kernels, so the fork indirection is no longer needed. ### Duplicate-work check Required searches were run (`gh pr list --search deepgemm`, `--search \"DeepGEMM pin\"`). Related open PRs, and why this is not a duplicate: - #51959 pins `deepseek-ai/DeepGEMM` at `a6b593d`, an older `nv_dev` commit. Verified that `a6b593d` has **no** SiTU support (no `situ` in `deep_gemm/mega/__init__.py` at that revision), so it would regress SiTU-dependent models. This PR pins the `nv_dev` tip which includes SiTU. - #51382 and #50796 keep the `vllm-project` fork and pin handcrafted `nv_dev+situ` merge/cherry-pick revisions. This PR achieves the same SM120 + SiTU combination using upstream's own `nv_dev` branch, dropping the fork dependency entirely. ## Test plan and results ```bash bash -n tools/install_deepgemm.sh # passed pre-commit run --files cmake/external_projects/deepgemm.cmake tools/install_deepgemm.sh # all applicable hooks passed; shellcheck hook # could not run (shellcheck not installed locally) git diff --check # clean ``` Additionally verified against the pinned commit that the vendoring layout consumed by `cmake/external_projects/deepgemm.cmake` is unchanged: submodules `third-party/cutlass` + `third-party/fmt`, `csrc/python_api.cpp`, and `deep_gemm/{__init__.py,utils/,testing/,legacy/,mega/,include/}` all present at `8b1392b`. A full CUDA build of the new pin and model evals were **not** run locally; requesting CI coverage. ## Disclosure This change was prepared with AI assistance (Kimi Code); the diff was reviewed line-by-line by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/52035", "createdAt": "2026-08-12T18:47:27Z", "updatedAt": "2026-08-12T20:49:04Z", "timestamp": "2026-08-12T20:49:04Z", "metrics": { "reactions": 4, "comments": 2 }, "labels": [ "ready", "ci/build", "deepseek" ], "author": "zyongye", "state": "closed", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52036", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails", "text": "## Purpose Fixes #51842. Serving `openbmb/MiniCPM-V-4_6` fails during weight loading, before the server starts: ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters ... are: {'qkv_proj.weight', 'qkv_proj.bias', 'out_proj.weight', 'out_proj.bias'} **Root cause.** The MiniCPM-V-4.6 ViT attention checkpoint stores separate `q_proj`/`k_proj`/`v_proj`, while vLLM fuses them into a single `QKVParallelLinear` (`qkv_proj`). `MiniCPMV4_6ViTWindowAttentionSelfAttn` tried to fuse them with a class-level `WeightsMapper(orig_to_new_stacked=...)` applied inside the submodule's own `load_weights`. Two problems make this a no-op: 1. `AutoWeightsLoader` strips the module prefix before dispatching to a submodule's `load_weights`, so the keys arriving here are bare (`k_proj.weight`), but the mapper substrings carry a leading dot (`.k_proj`) and never match. The unmapped `k_proj` then hits the \"no parameter named 'k_proj'\" error. 2. The leading dots cannot simply be dropped: `orig_to_new_stacked` applies every matching rule without breaking, and `qkv_proj` contains the substring `v_proj`, so a de-dotted mapper would re-map `k_proj → qkv_proj → qkqkv_proj` with the wrong shard id. ## Test Plan - Repro on 2× T4 (Turing, so `--dtype float16`), `tensor_parallel_size=2`, which triggers the failing weight-load path before and after the fix: ```python from vllm import LLM, SamplingParams llm = LLM(model=\"openbmb/MiniCPM-V-4_6\", trust_remote_code=True, dtype=\"float16\", tensor_parallel_size=2, max_model_len=4096, enforce_eager=True) print(llm.generate([\"Describe yourself in one sentence.\"], SamplingParams(max_tokens=32))[0].outputs[0].text) - Lint/compile on the change: ruff check and python -m py_compile. ## Test Result - Before: startup aborts with ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn . - After: model loads successfully ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52036", "createdAt": "2026-08-12T18:50:54Z", "updatedAt": "2026-08-12T18:56:09Z", "timestamp": "2026-08-12T18:56:09Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "songzhouran", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52037", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Skip unused Jina V5 output layers", "text": "## Purpose Jina Embeddings V5 models are pooling-only, but their vLLM wrappers inherit causal-LM classes. Because these wrappers already declare themselves as pooling models, they bypass the generic pooling adapter that replaces generation-only output layers. The encoder/nano variant therefore retained an unused `ParallelLMHead` with shape `[128256, 768]`. This change applies the existing pooling-model `no_init_weights` pattern to both Jina V5 wrappers, replacing `ParallelLMHead` and `LogitsProcessor` with a `StageMissingLayer`. The decoder/small checkpoint ties its output head to the input embedding (`VocabParallelEmbedding`), so that shared embedding remains intact. On an NVIDIA RTX PRO 6000 Blackwell (TP=1), the nano checkpoint's loaded model state changed as follows: | Metric | Before | After | |---|---:|---:| | Registered parameter bytes | 620,533,248 | 423,532,032 | | Unused `lm_head` bytes | 197,001,216 | 0 | | CUDA memory delta after model load | 1,283,457,024 | 1,086,324,736 | This removes 187.875 MiB of resident parameters (31.75%) and approximately 188 MiB of post-load CUDA memory. This is a resident-memory result; it is not a claim that the transient module-construction peak is eliminated. I searched open vLLM PRs and issues for Jina V5/Embeddings V5 combined with `lm_head`, output-layer, and pooling terms and found no direct or semantic duplicate. #32757 is the already-merged generic pooling mechanism used as the implementation precedent, not another Jina-specific fix. AI assistance (OpenAI Codex) was used to help investigate, implement, and test this change. I reviewed the complete diff and the validation results. ## Test Plan - Load and embed two natural prompts with both the base and patched `jinaai/jina-embeddings-v5-text-nano` checkpoints; compare parameter and post-load CUDA memory plus float32 embedding bytes. - Repeat the embedding comparison with `jinaai/jina-embeddings-v5-text-small` to cover its tied embedding. - Run the existing Jina pooling-model, adapter, and registry tests, then all pre-commit hooks applicable to the changed production file. Commands: ```bash .venv/bin/python -m pytest -q \\ tests/models/language/pooling/test_jina_embeddings_v5.py \\ tests/models/test_adapters.py \\ 'tests/models/test_registry.py::test_registry_imports[JinaEmbeddingsV5Model]' uvx pre-commit run --files \\ vllm/model_executor/models/jina.py git diff --check ``` ## Test Result - Pytest: **12 passed**. - Pre-commit: all applicable hooks passed, including Ruff, formatting, mypy, SPDX, forbidden-import, and configuration checks. - Nano real-model A/B: - Both runs produced embedding SHA-256 `f355f21e03004acfc86af056b3610282290563ab563dd5f9d04e5768bca06688`. - Both output vectors had shape `[768]` and L2 norm `1.0`. - Resident parameter and CUDA memory reductions are reported above. - Small real-model A/B: - Both runs retained 1,192,099,840 parameter bytes and no independent `lm_head`, confirming the tied input embedding was not removed. - Both produced embedding SHA-256 `c740490981d58e61cee407638b98c0ad724674e11341bc94086a7fa19a675942`. - Both output vectors had shape `[1024]` and L2 norm `1.0`.", "url": "https://github.com/vllm-project/vllm/pull/52037", "createdAt": "2026-08-12T19:00:55Z", "updatedAt": "2026-08-13T06:14:38Z", "timestamp": "2026-08-13T06:14:38Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "verified" ], "author": "BabyDrangoner", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52039", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix collect_env.py crash when pip list command fails", "text": "## Purpose Fixes the `collect_env.py` crash noted in #52027: in a uv venv with `uv` not on `PATH`, the pip-list command fails, `run_and_read_all` returns `None`, and `run_with_pip()` calls `None.splitlines()` → `AttributeError`. Separate from #52026 (the logging fix for the same issue), which doesn't touch this file. ## Changes - Return `None` when the list command fails, instead of calling `.splitlines()` on it (matches `get_conda_packages`; callers already handle `None`). - Remove a leftover `print(\"uv is set\")` debug line. ## Test `tests/test_collect_env.py`: calls `get_pip_packages` with a failing command and checks it returns `None` instead of crashing. ```bash .venv/bin/python -m pytest tests/test_collect_env.py -v ``` Passing; ruff clean. ## Related - Refs [#52027](https://github.com/vllm-project/vllm/issues/52027) (fixes the collect_env.py part) - Separate from #52026 (the logging fix for the same issue); doesn't touch this file Notes AI assistance was used; I reviewed every line and ran the test. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52039", "createdAt": "2026-08-12T19:14:16Z", "updatedAt": "2026-08-12T19:17:15Z", "timestamp": "2026-08-12T19:17:15Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "jahnavi-yelamanchi", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52041", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Skip broadcasting mm tensor data to workers for prefix-cache-covered items", "text": "## Purpose Fixes #52040. ## Background: why this gap exists and who it affects The EngineCore->TP-worker broadcast was designed when request inputs were token IDs (kilobytes), so shipping them unconditionally was both correct and free. Multimodal grafted large tensors onto the same path, but for the classic workload (a fresh image per chat request) the workers genuinely need every tensor, so unconditional shipping remained necessary. What changed recently is multimodal **prefix caching**: there now exists a state -- \"this item's tokens are fully KV-covered\" -- in which the shipped tensors are provably never consumed, and the broadcast becomes pure per-request CPU overhead. The workload that exposes this is the emerging **streaming / agentic multimodal** pattern: an assistant that re-sends a rolling window of frames every turn (a live camera or video feed with a position-stable layout, so prefix caching covers all history images). For that pattern the cost is linear in window size on every request -- ~19 ms x 25 images = ~0.5 s of EngineCore CPU per turn in our deployment -- while the tensors themselves (fixed-size fp32 pixel patches, ~15.5 MB per image at our processor settings) are re-serialized into the ring for workers that will never touch them. As this workload class grows (streaming VLM assistants, video agents, multi-turn visual memory), the benefit of skipping compounds: it turns fully-cached multimodal TTFT from linear-in-images into flat. When a multimodal item's placeholder span lies entirely within the prefix-cache-covered region, no encoder run can ever be scheduled for it — yet its processed tensors are still serialized into the EngineCore→TP-worker broadcast, per unique item, on every request. For workloads that re-send a rolling window of images (fully prefix-cached), this makes TTFT grow linearly with attached image count: we measured ~19 ms per unique image at TP=4 with 15.5 MB `pixel_values` per image (evidence, flamegraphs, and a duplicate-image control experiment in #52040). This PR extends the existing `MultiModalFeatureSpec.data = None` convention (already used to skip API-server→EngineCore IPC on processor-cache hits) one hop further, to the worker broadcast: - `NewRequestData.from_request` strips `data` from items whose span is fully inside `num_computed_tokens` (new helper `strip_covered_mm_data`, same file). - The scheduler-side `Request` keeps the full features, so nothing is lost engine-side. - **Preemption safety**: requests resumed from preemption re-ship their features via a new `CachedRequestData.resumed_mm_features` field, re-stripped against the post-eviction computed-token count — so items uncovered by eviction regain their data before their encoder runs are scheduled. The worker applies these in its resumed-request branch. - The worker's encoder-input batching already tolerates `data is None` (existing `continue` in `_batch_mm_inputs_from_scheduler`), and the v2-runner path folds resumed requests through `from_request`, so it re-evaluates the strip condition naturally. For the rolling-window workload this removes ~25 tensor broadcasts per request; the fully-cached TTFT floor becomes flat in image count (~4 ms/image of parse+hash remains, which is intrinsic to content addressing). ## Test Plan - Added unit tests for `strip_covered_mm_data` in `tests/v1/core/test_output.py`: fully-covered and boundary items stripped, uncovered kept, already-None preserved, input list not mutated, zero-computed no-op. - Logic additionally verified standalone; I was not able to run the full test suite locally (no GPU dev box) — relying on CI here, and happy to iterate. - Live-workload validation available on our deployment (TP=4, Gemma-4-31B vision, 26-image rolling window): expected effect is the fully-cached floor dropping from ~630 ms to near the 2-image floor. Can report numbers once a build with this change is deployable. ## Test Result Unit tests pass locally at the logic level; awaiting CI for the full suite. ## Notes for reviewers - The strip is deliberately conservative: it only removes data for items that *cannot* have encoder runs scheduled given current `num_computed_tokens` at ship time; every path that changes that quantity downward (preemption/resume) re-ships. - An alternative, larger design would move mm tensor transport to `scheduled_encoder_inputs` time entirely (ship exactly when an encoder run is scheduled). Happy to pursue that instead if preferred — this PR takes the minimal-diff route.", "url": "https://github.com/vllm-project/vllm/pull/52041", "createdAt": "2026-08-12T19:53:25Z", "updatedAt": "2026-08-13T00:55:11Z", "timestamp": "2026-08-13T00:55:11Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "verified" ], "author": "sseanliu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52043", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Force source builds for hybrid dependencies", "text": "Hybrid language-model CI began failing before pytest while `mamba-ssm` and `causal-conv1d` probed guessed GitHub release-wheel URLs. The requested CUDA/Torch and ROCm/Torch wheels do not exist, so the normal path is a caught HTTP 404 followed by a source build. During today's intermittent GitHub connectivity problems, some requests instead ended with `Remote end closed connection without response`, which escaped the installers' narrow exception handling and aborted the jobs. This change bypasses the fragile release probe and directly performs the same source builds used by successful runs. - Set `MAMBA_FORCE_BUILD=TRUE` and `CAUSAL_CONV1D_FORCE_BUILD=TRUE` for all affected language-model dependency installs. - Cover the active Hybrid, Granite L4, Extended Generation, and AMD mirror jobs, plus the analogous MI300 and MI355 jobs in `.buildkite/test-amd.yaml`. - Motivated by the [AMD Hybrid failure](https://buildkite.com/vllm/ci/builds/83591/canvas?jid=019ff759-6a91-445b-b5e2-55e2a28240a3&tab=output) and [NVIDIA Hybrid failure](https://buildkite.com/vllm/ci/builds/83591/canvas?jid=019ff75c-a52f-450b-a9d7-28c4a6e02bc2&tab=output), both of which stopped during dependency setup before pytest.", "url": "https://github.com/vllm-project/vllm/pull/52043", "createdAt": "2026-08-12T20:29:46Z", "updatedAt": "2026-08-13T01:33:30Z", "timestamp": "2026-08-13T01:33:30Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52044", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle DeepseekV4ForCausalLM in benchmark_moe get_model_params", "text": "## Summary Fixes [#52042](https://github.com/vllm-project/vllm/issues/52042): `benchmarks/kernels/benchmark_moe.py` crashes for `deepseek-ai/DeepSeek-V4-Flash-0731` with `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'`. While fixing it we audited every registered MoE architecture against `get_model_params()`'s dispatch table and found the same latent crash for ~28 more architectures (any MoE model whose config doesn't use the Mixtral `num_local_experts` convention). This PR addresses the whole class of bug, not just DeepSeek-V4. ## Root cause `get_model_params()` dispatches on the model architecture string. Any architecture not matched by an `elif` branch fell through to a silent \"default Mixtral\" branch that reads `config.num_local_experts`. For DeepSeek-V4 — and Gemma-4, EXAONE, OLMoE, Pangu, MiMo-V2, Step3.5, ERNIE-4.5-MoE, etc. — the config uses a different field convention, so it crashed with an opaque `AttributeError`. ## Changes 1. **DeepSeek-V4 fix**: `DeepseekV4ForCausalLM` added to the DeepSeek-V2/V3 branch (`n_routed_experts` / `num_experts_per_tok` / `moe_intermediate_size` — confirmed against the model's `config.json`). 2. **Register all remaining MoE architectures** from `registry.py` under their actual config conventions: - `n_routed_experts` family: `DeepseekForCausalLM`, `LongcatFlashForCausalLM`, `MiMoV2ForCausalLM`, `MiMoV2FlashForCausalLM`, `NemotronHPuzzleForCausalLM`, `PanguEmbeddedForCausalLM`, `PanguProMoEV2ForCausalLM`, `PanguUltraMoEForCausalLM` - `num_experts` / `moe_intermediate_size` family: `Qwen3_5MoeForCausalLM`, `AfmoeForCausalLM`, `BailingMoeForCausalLM`, `BailingMoeV2ForCausalLM`, `BailingMoeV2_5ForCausalLM`, `ExaoneMoeForCausalLM`, `LagunaForCausalLM`, `Lfm2MoeForCausalLM`, `Param2MoEForCausalLM`, `SarvamMLAForCausalLM`, `SarvamMoEForCausalLM` - `num_experts` / `intermediate_size` family: `Cohere2MoeForCausalLM`, `FlexOlmoForCausalLM`, `HYV3ForCausalLM`, `JambaForSequenceClassification`, `MiniCPMForCausalLM`, `OlmoeForCausalLM` - `num_local_experts` family (explicit branch, was the silent fallback): `MixtralForCausalLM`, `Llama4ForCausalLM`, `Llama4ForConditionalGeneration`, `ArcticForCausalLM`, `GptOssForCausalLM`, `GraniteMoeForCausalLM`, `GraniteMoeHybridForCausalLM`, `GraniteMoeSharedForCausalLM`, `MiniMaxM2ForCausalLM`, `PhiMoEForCausalLM` - `top_k_experts` family: `Gemma4ForCausalLM` - `moe_num_experts` family: `Ernie4_5_MoeForCausalLM`, `Step3TextForCausalLM`, `Step3p5ForCausalLM` 3. **Replace the silent Mixtral fallback with an explicit `ValueError`** for unsupported architectures, so future models fail with an actionable message (\"add it to get_model_params()\") instead of a misleading `AttributeError`. 4. **Testability**: `get_model_params` is extracted into a dependency-free `benchmarks/kernels/benchmark_moe_utils.py` (no torch/ray/triton imports), so it can be unit-tested without the benchmark's heavy deps. ## Testing - New unit test `tests/kernels/test_benchmark_moe_model_params.py`: all 59 registered MoE architectures resolve the correct `(E, topk, intermediate_size, hidden_size)` per their field convention, plus the `ValueError` for unsupported architectures. Runs with no GPU / no ray (`15 passed`). - Real configs via transformers: `deepseek-ai/DeepSeek-V3` → `(256, 8, 2048, 7168)`; `mistralai/Mixtral-8x7B-v0.1` → `(8, 2, 14336, 4096)`; DeepSeek-V4 config data → `(256, 6, 2048, 4096)`.", "url": "https://github.com/vllm-project/vllm/pull/52044", "createdAt": "2026-08-12T20:54:27Z", "updatedAt": "2026-08-13T08:58:00Z", "timestamp": "2026-08-13T08:58:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "performance", "deepseek" ], "author": "SayHelloToWorld", "state": "open", "assignees": [ "jeejeelee" ] }, { "id": "github:vllm-project/vllm:pull_request:52045", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Workaround contradictory mamba validation after CPU MLA disables prefix caching", "text": "## Purpose Partial workaround for #52008. **Not the full fix.** #52008 asks for one of: - **A.** Keep prefix caching and chunked prefill enabled on CPU when KimiLinear requires them - **B.** Fail fast with a clear “KimiLinearForCausalLM is not supported on CPU” - **C.** Honor `--hf-overrides '{\"use_mla\": false}'` before the CPU platform check **This PR does not implement A, B, or C.** What it does: after the existing CPU MLA path turns prefix caching and chunked prefill **off**, also reset auto-applied mamba `align` / `mamba_block_size` so `VllmConfig` does not raise: ``` ValueError: --mamba-block-size can only be set with --enable-prefix-caching ``` That is only the contradictory validation. It does not leave prefix caching or chunked prefill on. It does not treat `use_mla=false` as a real opt-out (`use_mla` is still derived from `kv_lora_rank` / `kimi_linear`). It does not make `inference-optimization/Kimi-K3-0.40B` a supported CPU serving path. If the user passed `--mamba-block-size` themselves, we raise a clear error instead of resetting it. That is still not option B for the default serve command. ## Out of scope - CPU MLA prefix caching / chunked prefill (option A) - A hard “architecture not supported on CPU” on default flags (option B) - Honoring `hf_overrides[\"use_mla\"]` (option C) - End-to-end generation for Kimi-K3-0.40B on CPU (CPU MLA decode is still limited, e.g. `head_dim=576`) ## Test Plan ```bash python -m pytest --noconftest -v tests/config/test_cpu_mla_hybrid_kimi.py ``` Local `config.json` fixture only. No weight download. No serve/generate. ## Test Result On macOS arm64 / Python 3.12: ``` 3 passed ``` `EngineArgs.create_engine_config()` no longer raises the `mamba-block-size` validation error. Resulting config still has prefix caching and chunked prefill **disabled**, `use_mla=True`, mamba mode `none`. This PR includes AI-assisted edits (Grok). I reproduced the validation error, limited the change to that conflict, and ran the tests above. - [x] Purpose (workaround only; not A/B/C) - [x] Test plan - [x] Test results (config construction only) - [ ] Docs (none; this is not a supported CPU Kimi-K3 path)", "url": "https://github.com/vllm-project/vllm/pull/52045", "createdAt": "2026-08-12T20:54:58Z", "updatedAt": "2026-08-13T09:41:32Z", "timestamp": "2026-08-13T09:41:32Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "bug", "cpu", "kimi" ], "author": "akesar01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52046", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][NVIDIA] Add PCP support to DeepSeek V3.2 attention", "text": "## Summary - add prefill context parallelism (PCP) to the existing DeepSeek V3.2 attention implementation - keep TP and PCP on the same fused attention path instead of maintaining a separate PCP forward - materialize fused MLA and indexer K outputs only when PCP needs to gather them before cache insertion - specialize optional materialization and cache writes directly from nullable Triton pointers, without separate compile-time mode flags - preserve dense-MHA dispatch for short prefill while using the sparse-MQA path for long prefill and decode - combine PCP attention outputs with the existing LSE-aware reduction and finalization helpers ## Scope and dependencies This PR contains only generic DeepSeek V3.2 PCP support. It does not change model routing, registries, speculative-decoding model selection, or GLM-specific GEMM behavior. GLM-5.2 end-to-end validation was performed with #49790 applied, because that PR routes GLM-5.2 and its MTP model to the DeepSeek V3.2 implementation. Once #49790 lands, GLM-5.2 can use the PCP support added here without any GLM-specific code in this PR. ## Non-duplication - #49790 supplies GLM-5.2-to-DeepSeek-V3.2 routing; this PR supplies PCP inside the DeepSeek V3.2 attention implementation. - #50005 fixes DCP correctness in the older NVIDIA fused override, including query gathering and output/LSE merging. This PR targets PCP in the current common DeepSeek V3.2 path and additionally handles PCP K materialization, cache gathering/insertion, and short-prefill dense-MHA dispatch. - #46178 addresses generic sparse-attention DCP plumbing rather than this DeepSeek V3.2 PCP path. ## Validation Current PCP-only head: `86793bce26` - touched-file pre-commit suite: passed, including Ruff, formatting, mypy, SPDX, forbidden-import, and accelerator-API checks - `pytest tests/models/deepseek_v32/test_pcp_dispatch.py -q`: 2 passed - B300 fused-kernel suite on the optional-output implementation: 57 passed - GB200 nullable-cache and nullable-`q_pe_out` validation: 29 fused norm/RoPE/cache cases and 24 Triton fused-Q cases passed - 8xB300 GLM-5.2 NVFP4 PCP8/EP8 smoke with #49790 routing: startup, warmup, and requests completed with zero failures - 8xB300 PCP8/EP8 prefill-only profile with #49790 routing, FP8 KV cache, 32K max batched tokens, and 16K long-prefill threshold: 32/32 requests completed, 1,399,021 input tokens in 9.48 seconds, 147.58K input tokens/s; full-32K forward p50 was 366.9 ms No model-quality evaluation has been run on this revision, so the PR remains draft. ## AI assistance and human review OpenAI Codex assisted with investigation, implementation, validation orchestration, and drafting this description. The human submitter must review every changed line and be able to explain and defend the PCP cache-gather and LSE-combine protocol before marking this ready for review.", "url": "https://github.com/vllm-project/vllm/pull/52046", "createdAt": "2026-08-12T21:00:58Z", "updatedAt": "2026-08-13T07:46:07Z", "timestamp": "2026-08-13T07:46:07Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "new-model", "speculative-decoding", "deepseek", "nvidia" ], "author": "GirasoleY", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52047", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][AMD] Annotate draft KV cache groups on the hybrid grouping path", "text": "## Purpose Serving **Kimi-K3** (`KimiLinearForCausalLM` — 93 layers: 24 full-attention MLA + 69 KDA linear-attention) with the **DSpark MTP draft** (`Inferact/Kimi-K3-DSpark`) and a CPU KV offload tier, the external prefix cache never serves anything: `vllm:external_prefix_cache_hits_total` stays pinned at exactly 0 for the life of the server while `vllm:kv_offload_store_bytes_total` grows without bound. The offload tier writes continuously and reads back nothing. The same latent defect affects any hybrid (attention + Mamba) model with an EAGLE-family drafter that reaches the general multi-group KV-cache grouping path. ## Root cause `is_eagle_group` is set in exactly one place in the tree: `_annotate_eagle_groups_deepseek_v4` (`vllm/v1/core/kv_cache_utils.py`). It is gated on `model_version == \"deepseek_v4\"` and is reachable only from the `group_and_unify_kv_cache_specs` branch of `get_kv_cache_groups`. Kimi-K3 is MLA + Mamba, so it lands on the general multi-group path and **no group is annotated at all**. Consumers respond to \"no group annotated\" by flagging *every* group as a draft group — both `HybridKVCacheCoordinator.find_longest_cache_hit` and the `OffloadingConnector` scheduler (`offloading/scheduler.py`, `eagle_groups = set(range(len(kv_cache_groups)))`). That fallback is documented as conservative, and it is — for full-attention groups. For a **Mamba** group it is not: 1. `get_sliding_window_size_in_chunks` returns `1` for a `MambaSpec` — \"Mamba depends on a single state\". 2. The eagle flag then applies `required_window += 1` → **2**. 3. `_sliding_window_lookup` reports a hit only on `consecutive_hits == sliding_window_size`, i.e. **two adjacent stored chunks**. 4. Under `mamba_cache_mode=\"align\"`, exactly one state checkpoint is materialized per scheduler step; the chunks in between are null blocks (`block_id == 0`) and are skipped on store. 5. Two adjacent mamba chunks are therefore **never both stored**, so the group can never report a hit. 6. `_lookup` takes the minimum across groups, so `num_hit_chunks == 0` zeroes the whole request. ## What this means concretely for Kimi-K3 + DSpark The Mamba page forces a **1536-token block**, and the draft's 5 MLA layers are the smallest spec bucket, so `group_size` collapses to 5 — giving **20 KV cache groups: 5 target-MLA, 14 Mamba, 1 draft**. (Consistent with the two page-padding warnings every DSpark run logs: `4.17% = 1/24` and `1.45% = 1/69`.) At `--max-num-batched-tokens 32768` a KDA checkpoint lands every **21 chunks = 32,256 tokens**. | | Groups flagged | Mamba group lookup | Result | |---|---|---|---| | **before** | 20 / 20 | needs 2 adjacent chunks at a 21-chunk stride | unsatisfiable → `get_num_new_matched_tokens` returns 0 for every prompt-shaped request | | **after** | 1 / 20 (the draft) | window stays 1, satisfied by a single checkpoint | lookup is no longer structurally zeroed; the 5 target-MLA groups also stop taking the eagle last-chunk drop | The failure is silent today. A vetoed lookup is not counted as a *query* either, so the hit **rate** stays empty rather than visibly dropping — there is no counter an operator can watch go wrong. Hence the second half of this patch. ## The fix Annotate **by spec rather than by model version**. `non_causal_multi_token_decode` is set exactly on drafter attention layers (`vllm/models/kimi_k3/nvidia/dspark_mla.py`), is declared on `MLAAttentionSpec` as \"Marks draft groups that flatten a non-causal query block into decode rows\", and is propagated through `MLAAttentionSpec.merge` via `any(...)`. So it identifies the draft group wherever grouping happens to land. Two functions, both called at the end of the general multi-group path: - `_annotate_eagle_groups_from_draft_spec` — flags groups whose merged spec carries the marker. - `_warn_if_unannotated_eagle_mamba` — warns when speculative decoding is on, no group could be identified, and Mamba groups are present. That combination still takes the flag-all path and should not fail silently. ## Why this is not a duplicate | PR | What it does | Why it does not cover this | |---|---|---| | **#47891** — Exclude DSpark draft KV-cache group from OffloadingConnector lookup poisoning | Adds `eagle_group_is_veto_exempt` so an *identified* draft group's miss does not veto the request | The flag is only ever set inside `_annotate_eagle_groups_deepseek_v4`. On an unannotated model the exempt set is empty and nothing changes. The PR says so explicitly: *\"Left empty in the 'flag all groups' fallback below … we conservatively keep the original behavior.\"* Kimi-K3 is precisely that case. | | **#48459** — core-tier half of #47891 | Same mechanism on `HybridKVCacheCoordinator` | Same single DSV4-gated setter; same gap. | | **#48375** — Honor `drop_eagle_block` in `MambaManager` | Fixes silent recurrent-state corruption on the GPU tier | Different bug. Does not touch group annotation or the offload lookup window. | | **#41640** — Add `is_eagle` in `AttentionSpec` | Generic annotation infrastructure that would attack the same root | Open and untouched since May 2026. If it lands, this patch should be rebased onto it. | No open PR mentions `required_window` or `_sliding_window_lookup` at all. **This patch is the missing precondition for #47891/#48459 rather than a competitor.** It stops the flag-all fallback from firing; they stop the identified draft group from vetoing. They compose, and on a hybrid model neither is sufficient alone. ## Test plan E2E KK3 test Four tests added to `tests/v1/core/test_kv_cache_utils.py`: - `test_draft_group_annotated_on_hybrid_general_path` — on an MLA + Mamba + draft-MLA fixture, exactly one group is flagged, it is the draft's, and no Mamba group is flagged. - `test_draft_group_not_annotated_without_spec_decode` — the marker alone flags nothing. - `test_unidentifiable_draft_with_mamba_warns` — the warning fires when no draft can be identified. - `test_no_warning_when_draft_group_is_identified` — and stays quiet when one can. ## Test result Unit tests: ``` tests/v1/core/test_kv_cache_utils.py::test_mla_draft_prefers_standard_layout_when_pages_can_be_unified PASSED [ 16%] tests/v1/core/test_kv_cache_utils.py::test_draft_group_annotated_on_hybrid_general_path PASSED [ 33%] tests/v1/core/test_kv_cache_utils.py::test_mamba_groups_never_flagged_even_when_draft_shares_a_group PASSED [ 50%] tests/v1/core/test_kv_cache_utils.py::test_draft_group_not_annotated_without_spec_decode PASSED [ 66%] tests/v1/core/test_kv_cache_utils.py::test_unidentifiable_draft_with_mamba_warns PASSED [ 83%] tests/v1/core/test_kv_cache_utils.py::test_no_warning_when_draft_group_is_identified PASSED [100%] ``` E2E Kimik3: on E2E I've measured the following improvements with the PR Config: ISL 8192 / OSL 1024 / conc 16, 24 sessions, TP8, DSpark, 64 GiB offload pool, GPU KV capped to 163,840 tokens so the 221 k working set evicts. \"image\" ## Accuracy / serving eval ``` ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52047", "createdAt": "2026-08-12T21:08:09Z", "updatedAt": "2026-08-12T23:52:12Z", "timestamp": "2026-08-12T23:52:12Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm" ], "author": "okorzh-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52048", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add DeepseekV4ForCausalLM to benchmark_moe.py model param dispatch", "text": "## Purpose Fixes #52042. `benchmark_moe.py --tune` crashes for DeepSeek-V4 models (e.g. `deepseek-ai/DeepSeek-V4-Flash-0731`): ```text AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` `get_model_params()` dispatches on the architecture string. The DeepSeek-family branch reads `n_routed_experts` / `num_experts_per_tok` / `moe_intermediate_size`, but `DeepseekV4ForCausalLM` is missing from its tuple, so V4 falls through to the Mixtral default, which reads `num_local_experts` — not present on DeepSeek configs. This adds `DeepseekV4ForCausalLM` (the registered arch name, see `vllm/model_executor/models/registry.py`) to that tuple. vLLM's own V4 model reads exactly those three fields when building the MoE gate and experts (`vllm/models/deepseek_v4/nvidia/model.py`), so the branch returns the correct shapes rather than plausible-but-wrong ones. `DeepSeekV4MTPModel` / `DSparkDraftModel` are intentionally left out: the dispatch carries no MTP/draft architecture for any family, and this benchmark tunes main-model MoE shapes. **Relation to existing PRs:** #49088 generalizes this dispatch to alias-based field probing and would make this change redundant, but it has been inactive since 2026-07-19; this is a minimal targeted unblock for the filed issue, and I am happy to close it if #49088 lands first. #40546 only improves the fallback error message. #50082 adds Kimi K3 via the same one-architecture pattern. ## Test Plan The change only affects config-field dispatch, so no GPU is required. Called `get_model_params()` with a `DeepseekV4Config` carrying V4-Flash MoE fields (`n_routed_experts=256`, `num_experts_per_tok=8`, `moe_intermediate_size=2048`, `hidden_size=7168`), plus a `DeepseekV3ForCausalLM` control to confirm the existing branch is unaffected. ## Test Result - Before: `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'` — the exact error from #52042 - After: returns `(256, 8, 2048, 7168)` - V3 control unchanged: `(160, 6, 1536, 5120)` - `ruff check` and `ruff format --check` clean on the file Model evaluation: not applicable — this is a benchmarking-script fix with no effect on model output, accuracy, or serving. --- This change was developed with AI assistance; I reviewed and tested every line.", "url": "https://github.com/vllm-project/vllm/pull/52048", "createdAt": "2026-08-12T21:09:52Z", "updatedAt": "2026-08-13T15:42:07Z", "timestamp": "2026-08-13T15:42:07Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "performance", "deepseek" ], "author": "rajathpi", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52050", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fall back to FA2 for Blackwell head-dim-256 paged attention", "text": "## Purpose I came across this issue while enabling ColPali for model runner v2 pooling migration. ColPali MRV2 exposed the issue because its decoder attention supplies `seqused_k,` while FA4’s specialized SM100 head-dim-256 kernel currently rejects `seqused_q/k`. This caused the inference to abort after selecting FA4. This PR consistently resolves FA2 when sequence-length tensors are required, rejects unsupported FP8, DiffKV, and FA4-only masking combinations during backend selection, and preserves FA4 for encoder and other callers that do not pass `seqused_q/k`. ## Reproducer ```bash CUDA_VISIBLE_DEVICES=0 \\ VLLM_USE_V2_MODEL_RUNNER=1 \\ VLLM_ATTENTION_BACKEND=FLASH_ATTN \\ .venv/bin/python -m pytest \\ tests/models/multimodal/pooling/test_colpali.py::test_colpali_multimodal_text_query_image_docs \\ -v ``` ## Output on main / on branch ```text main: AssertionError: SM100 forward with head_dim=256 does not support seqused_q/seqused_k branch: PASSED [100%] 1 passed, 15 warnings in 85.21s ``` ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/kernels/attention/test_attention_selector.py \\ -k sm100_hd256 -q ``` ```text 3 passed, 38 deselected, 20 warnings in 0.89s ``` ```bash CUDA_VISIBLE_DEVICES=0 \\ VLLM_ATTENTION_BACKEND=FLASH_ATTN \\ .venv/bin/python -m pytest \\ tests/models/multimodal/pooling/test_colpali.py::test_colpali_multimodal_text_query_image_docs \\ -v ``` ```text 1 passed, 15 warnings in 85.21s ``` ```bash .venv/bin/pre-commit run --files \\ $(git diff --name-only upstream/main...HEAD) ``` ```text All hooks passed ``` ## Model evaluation Using `vidore/colpali-v1.3-hf`: | Comparison | Result | |---|---| | V1 versus MRV2 query output | Exact match | | V1 versus MRV2 image cosine | `0.999953` red, `0.999977` blue | | V1 MaxSim | red `3.766706`, blue `3.047093` | | MRV2 MaxSim | red `3.766586`, blue `3.047139` | | HF MaxSim | red `3.743096`, blue `3.112920` | V1, MRV2, and the HF reference all produced the same document ranking. ## Performance impact On Blackwell, decoder attention with `head_size=256` and `seqused_q/k` now uses FA2. A direct FA4 performance comparison is unavailable because FA4 aborts for this input. Encoder and other paths that do not require these tensors continue using FA4. ## AI assistance disclosure OpenAI Codex (GPT-5) assisted with drafting the code. I reviewed every changed line and personally ran the tests and model evaluations listed above.", "url": "https://github.com/vllm-project/vllm/pull/52050", "createdAt": "2026-08-12T21:30:39Z", "updatedAt": "2026-08-13T13:29:20Z", "timestamp": "2026-08-13T13:29:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "multi-modality" ], "author": "taneem-ibrahim", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52052", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Use silu_and_mul_with_clamp's torch._C op", "text": "This is limited to alpha=1.0 and beta=0.0 as a safety consideration due to MiniMax previously avoiding this kernel in other cases. During DeepSeekV4, day 0 support, forward_cuda was disabled in favor of forward_native for silu_and_mul_with_clamp. This can be reverted for similar accuracy and around a 6% speedup in cases without speculative decode. Example Command: vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --attention_backend ROCM_AITER_UNIFIED_ATTN \\ --compilation_config '{\"mode\":3,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}' \\ --gpu_memory_utilization 0.9 \\ --host 0.0.0.0 \\ --max_model_len 4096 \\ --max_num_seqs 512 \\ --no_enable_prefix_caching \\ --port 39817 \\ --tensor_parallel_size 4 \\ --no-enable-log-requests \\ --dtype auto \\ --kv-cache-dtype fp8 \\ --max-num-batched-tokens 8192 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --reasoning-parser deepseek_v4 \\ --tool-call-parser deepseek_v4 \\ --enable-auto-tool-choice Accuracy for gsm8k with num_fewshot=8 remained in the 0.94 range. Performance (tok/s before->after) for ISL=512, OSL=1024, temperature=0: concurrency=16: 1762.42->1883.16 (6.85%) concurrency=128: 9152.36-> 9602.00 (5%)", "url": "https://github.com/vllm-project/vllm/pull/52052", "createdAt": "2026-08-12T21:34:09Z", "updatedAt": "2026-08-12T21:34:44Z", "timestamp": "2026-08-12T21:34:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "tpopp", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52054", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][V1] Chunk prompt-logprobs logits to bound the activation peak", "text": "## Purpose `_get_prompt_logprobs_dict` materializes the full `[num_scheduled_tokens, vocab_size]` logits tensor in one shot and then upcasts it to float32 via `log_softmax`. Both tensors are live simultaneously, so the transient peak is roughly: ``` num_scheduled_tokens * vocab_size * (2 bytes bf16 + 4 bytes fp32) ``` At 8192 scheduled tokens and a 152k vocabulary that is **~7.5 GiB**. That peak is never observed by memory profiling. `profile_run` calls `_dummy_sampler_run`, which constructs its `SamplingMetadata` with: https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu_model_runner.py#L6362 ```python max_num_logprobs=None, ``` so the logprobs path is not exercised and the float32 score tensor is never allocated during profiling. KV cache capacity is sized against a peak that excludes prompt logprobs, and the allocation has to come out of whatever is left at serving time. This is exactly the mechanism @robertgshaw2-redhat described in #5907: > When determining the KV cache size, we calculate peak memory running a long prefill *without logprobs*. If a prompt requests many logprobs, however, this is an additional source of memory usage which is not considered during warmup and can cause OOM because we have nothing in scheduler to prevent this. ## Approach Rather than restructure profiling, this bounds the allocation directly — the \"chunked logits processing\" approach proposed by @tjohnson31415 in #5907 and endorsed in that thread as \"the ideal solution\". `compute_logits` and the score computation now run over row chunks of `VLLM_PROMPT_LOGPROBS_CHUNK_SIZE` (default 1024). Each chunk's results are copied into the existing per-request CPU tensors before the next chunk is materialized, so the peak becomes independent of `max_num_batched_tokens`. Behaviour is unchanged: same logits, same scores, same destination rows, same `logprobs_mode` handling. Only the number of rows materialized at once differs. The default of 1024 leaves the common case (short prompts, small `max_num_batched_tokens`) at a single chunk, so there is no change in behaviour or performance for most deployments. ## Why this is not duplicating an existing PR Checks run: ``` gh issue view 5907 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"prompt_logprobs memory\" gh pr list --repo vllm-project/vllm --state open --search \"prompt logprobs profile_run\" gh pr list --repo vllm-project/vllm --state open --search \"prompt_logprobs OOM\" gh pr list --repo vllm-project/vllm --state merged --search \"5907 in:body\" ``` - **#5907** — the tracking issue for this exact defect. Closed as **stale**, never fixed. - **#5355** (\"[Bugfix] Take the VRAM usage of prompt_logprobs into account\") — the referenced fix attempt. **Closed, unmerged.** - No merged PR addresses it. Verified the behaviour is still present on `main` at `23f360edaa`. - The four open `prompt_logprobs` PRs are unrelated concerns: **#51846** (`prompt_logprobs=0` admission guards / chat echo), **#49622** (`kv_sharing_fast_prefill` incompatibility validation), **#42245** (non-determinism with prefix caching), **#51953** (DiffusionGemma tiled logits projection — a different model's sampler path). ## Test commands run and results New test file `tests/v1/worker/test_gpu_prompt_logprobs_chunking.py`, modelled on the existing `tests/v1/worker/test_gpu_rejection_sampler_chunking.py` convention: ``` python -m pytest tests/v1/worker/test_gpu_prompt_logprobs_chunking.py -v 9 passed ``` Coverage — chunked output must match a single-chunk reference exactly, across `chunk_size` ∈ {1, 3, 4, 1024} (degenerate, short-final-chunk, evenly-dividing, single-chunk) and `logprobs_mode` ∈ {`raw_logprobs`, `raw_logits`}, plus an assertion that no single `compute_logits` call exceeds the configured chunk size and that every row is covered exactly once. **Verified by negation:** injecting the classic chunking bug (destination slice ignoring the chunk offset, `slice(start_idx, start_idx + (end - pos))`) makes 6 of the 9 tests fail with a 100%-mismatched tensor. The 3 that still pass are the single-chunk cases, where the bug is unreachable. So the tests genuinely pin the row mapping rather than passing vacuously. The tests are CPU-runnable. They disable `PIN_MEMORY` via an autouse fixture because `async_tensor_h2d` pins the source buffer, which requires a CUDA allocator; the chunking logic under test is unaffected. ## AI assistance This change was developed with AI assistance. The problem analysis, the upstream-vs-fork verification, the patch and the tests were produced with an AI coding agent; the duplicate-work checks above were run as part of that process. A human submitter *has* reviewed every changed line. The chunking approach is a port of a fix that has been running in production in a downstream fork, where it resolved this OOM.", "url": "https://github.com/vllm-project/vllm/pull/52054", "createdAt": "2026-08-12T21:45:27Z", "updatedAt": "2026-08-12T22:59:02Z", "timestamp": "2026-08-12T22:59:02Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "mrv1-only" ], "author": "malaiwah", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52055", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle HarmonyError in process_chunk to fix gpt-oss streaming 500s", "text": "FIX #51977 ## Purpose gpt-oss requests fail with HTTP 500 (`openai_harmony.HarmonyError: unexpected tokens remaining in message header`) when the model samples a malformed Harmony message header. `HarmonyParser.flush()` already catches `HarmonyError` and recovers by returning the raw decoded text, but `process_chunk()` calls `self._harmony_parser.process(token_id)` with no handling, so a mid-stream parse failure escapes to the API server and kills the request. This gives `process_chunk()` the same recovery contract as `flush()`: on `HarmonyError`, decode the buffered message tokens plus the unconsumed tail of the chunk as raw final-channel text, emit it as a delta segment plus a completed message, reset parser state, and stop processing the chunk. The dual-segment shape matches what `flush()` already emits, so streaming, non-streaming `parse()`, and the Responses message loop all surface the text. `parse()` and `HarmonyContext.append_output` read only `completed_message`, so a delta-only segment would have turned the 500 into an empty 200 instead. Nothing the model generated is dropped, and the request completes. Note on the trigger: the issue points at a hyphen in the tool name, but openai_harmony 0.0.8 accepts hyphenated recipients. Reproducing directly against the library shows the real trigger is leftover tokens in the header. A duplicated `to=` clause reproduces the reporter's exact error string, which also explains why the failure was intermittent rather than reliable. Scope: vLLM cannot stop a model sampling a malformed header, so it has to tolerate `HarmonyError` from the library. `flush()` established that defensive contract already; this completes it for the mid-stream path. A leniency fix upstream in openai_harmony would be complementary, not a substitute. Not duplicating existing work: #51977 is unclaimed, and no open PR references it or touches `vllm/parser/harmony.py`. ## Test Plan New regression test `tests/parser/test_harmony.py::test_process_chunk_recovers_from_malformed_header` drives a duplicated `to=` header through `process_chunk`, asserts the generated text comes back as a final-channel delta segment plus a completed message instead of an exception, then asserts a normal message parses afterwards to prove the state reset. It fails without the source change. ```bash pytest tests/parser/test_harmony.py tests/reasoning/test_gptoss_reasoning_parser.py ``` ## Test Result ```text 66 passed ``` With `vllm/parser/harmony.py` reverted, the new test fails: ```text FAILED tests/parser/test_harmony.py::test_process_chunk_recovers_from_malformed_header ``` `ruff check` and `ruff format --check` clean on both files; mypy (CI configuration) reports no issues on either file. No model evaluation run: only the error-recovery path changes, and behaviour on well-formed output is untouched, covered by the 65 pre-existing tests in this file. --- This change was developed with AI assistance; I reviewed and tested every line.", "url": "https://github.com/vllm-project/vllm/pull/52055", "createdAt": "2026-08-12T21:45:49Z", "updatedAt": "2026-08-13T11:24:45Z", "timestamp": "2026-08-13T11:24:45Z", "metrics": { "reactions": 4, "comments": 4 }, "labels": [ "bug", "tool-calling", "gpt-oss" ], "author": "rajathpi", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52056", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build][AMD] Update triton_kernels pin for RDNA MXFP4 matmul_ogs config", "text": "Advances the ROCm triton_kernels pin to 74e4569a70, which backports upstream Triton PR #10810 (\"[AMD][kernels] Improve MXFP4 matmul_ogs config on RDNA\"). Avoids register spilling in the MXFP4 MoE GEMMs on RDNA. ## Purpose Advances the ROCm `triton_kernels` pin in `cmake/external_projects/triton_kernels.cmake` from `0f380657` to `74e4569a70`, picking up the RDNA MXFP4 `matmul_ogs` configuration change. - Upstream Triton PR: https://github.com/triton-lang/triton/pull/10810 - ROCm backport (the commit this pin now points at): https://github.com/ROCm/triton/pull/951 **Why it matters.** On RDNA the current pin selects `block_m = 128` tiles for the MXFP4 MoE GEMMs. The fp32 accumulator alone is `BLOCK_M × BLOCK_N` registers, which at that tile size consumes half of the 256-VGPR wave32 budget before any operand is loaded. Register demand exceeds the architectural cap, the compiler spills to scratch memory, and the spills land inside the K-loop where the cost is paid on every iteration. WMMA instruction counts are unchanged — the arithmetic is identical and the extra time is pure spill overhead. The backported configuration sets `block_m` explicitly for MXFP4 on RDNA, so the spilling tiles are never built. This is a pin bump only. The kernel selection logic is ROCm's backport of upstream work; no vLLM code is changed beyond the pinned SHA. ## Test Plan `gpt-oss-20b` (MXFP4 MoE) on RDNA, TP=1. ```bash vllm bench throughput \\ --model openai/gpt-oss-20b --max_model_len 4096 --trust-remote-code \\ --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \\ --num_prompts 1000 --tensor_parallel_size 1 --gpu-memory-utilization 0.8 ``` Register and spill counts read from `.vgpr_count` / `.vgpr_spill_count` in the compiled `.amdgcn` products, selecting the autotune-chosen variant by `num_warps`. ## Test Result Confirmed the performance gain on RDNA that the linked PRs describe. Throughput improved and no remaining kernel spills observed.", "url": "https://github.com/vllm-project/vllm/pull/52056", "createdAt": "2026-08-12T21:52:07Z", "updatedAt": "2026-08-12T22:33:06Z", "timestamp": "2026-08-12T22:33:06Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "umarkovi-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52058", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Bound KV block zeroing launch geometry", "text": "## Purpose Fix the `KVBlockZeroer` launch overflow reproduced on [`main` nightly #83443](https://buildkite.com/vllm/ci/builds/83443/canvas?jid=019ff2a1-641e-4916-9149-30a5074c8a9c&tab=output), at commit [`3e372c5ff2`](https://github.com/vllm-project/vllm/commit/3e372c5ff23438eeeafc86c7d8d51026f3dacb6a): ``` OverflowError: signed integer is greater than maximum ``` DeepSeek-V4 combines 181 KV segments with 9,344- and 292-element pages. The old zeroer selected the largest common power-of-two divisor, so the 292-element page forced every segment to use four-element chunks. Zeroing 6,870 blocks therefore flattened to: ``` 6,870 * 181 * 2,336 = 2,904,745,920 programs ``` That overflows the signed launch dimension passed by the NVIDIA wrapper. This change: - maps blocks, segments, and chunks directly onto a 3-D grid, keeping block IDs on the large x-axis; - uses up to 1,024 elements per program and masks each segment's tail, so small pages no longer shrink all chunks; - reduces the failing geometry to `(6,870, 181, 10)`, or 12,434,700 programs (233.6x fewer); - preserves one compiled kernel across different block counts. The H100/H200 `sm90_paged_mqa_logits_metadata` failures from the same CI build were separate. They are already fixed on current `main` by #52035; its [exact 4xH100 KV-offload job](https://buildkite.com/vllm/ci/builds/83596#019ff753-5357-42e3-a6ba-8d51b6b40b2f) passed both DeepSeek-V4 cases (`2 passed`). ## Duplicate-work check Required open PR/issue searches were run for `KVBlockZeroer overflow`, `KV block zeroing grid`, and `signed integer zeroing`. #50485 also proposes a 3-D grid, but this is materially different: - #50485 is currently conflicted and predates packed block strides; - it puts block IDs on the z-axis, which is limited to 65,535 entries on CUDA; - it retains narrow divisor-based chunks; - this change puts blocks on x and adds masked wide chunks, addressing both the observed NVIDIA overflow and the 233.6x excess work. ## Test plan and results ```bash CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m pytest \\ tests/v1/worker/test_kv_block_zeroer.py \\ tests/v1/worker/test_dsv4_packed_zeroer_geometry.py -q # 9 passed CUDA_VISIBLE_DEVICES=0 compute-sanitizer --tool memcheck --error-exitcode 99 \\ .venv/bin/python -m pytest tests/v1/worker/test_kv_block_zeroer.py -q # 8 passed; ERROR SUMMARY: 0 errors .venv/bin/pre-commit run --files \\ vllm/v1/worker/utils.py tests/v1/worker/test_kv_block_zeroer.py # passed ``` An exact-shape B200 validation launched grid `(6870, 181, 10)`, zeroed all storage, and completed in 0.334 seconds. No model eval was run because this only changes how already-selected KV-cache bytes are zeroed, not model outputs or scheduling semantics. ## Disclosure AI assistance (OpenAI Codex) was used to investigate the CI history, develop the change, and draft this description. reviewed by the submitter ## Breaking PR and overlap The failure became reachable after #51749 generalized worker-side KV zeroing to every allocating `AttentionSpec`. That change is needed to prevent stale FP8 sliding-window pages; it exposed a pre-existing launch-geometry scaling bug when DeepSeek-V4 contributes many heterogeneous segments. #52062 addresses the same observed overflow by reverting #51749. This PR is materially different: it preserves generalized zeroing and bounds the kernel launch using a 3-D grid plus masked wide chunks, fixing the overflow without restoring the stale-cache bug that #51749 corrected.", "url": "https://github.com/vllm-project/vllm/pull/52058", "createdAt": "2026-08-12T22:01:44Z", "updatedAt": "2026-08-13T06:42:52Z", "timestamp": "2026-08-13T06:42:52Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "ready" ], "author": "LucasWilkinson", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52059", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Split MiniMax-M3 prefill index-score K loop", "text": "> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (incorrect expert count caused silent accuracy loss) > > Base image: `vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106` ## Purpose MiniMax-M3 prefill computes index scores by scanning every visible 128-token index-K block for each query block. The current gfx942 kernel uses 64 query tokens per program and processes the full K range in one loop. At long context lengths, this repeatedly loads index-K data and leaves insufficient parallel work for the GPU. ## Suggested Fix For gfx942 prefill with at least 128 query tokens and an 8K sequence length: - Increase the query block size from 64 to 128. - Split the state-free K loop across multiple programs. - Select a power-of-two split count using the query workload, sequence length, and available compute units. - Cap the split count at 16. Each split writes a separate range of score columns, so no atomics or reduction pass are required. Other architectures and smaller shapes retain the existing 64-token, single-split configuration. ## Test Plan ```bash pytest -q tests/kernels/attention/test_minimax_m3.py \\ -k prefill_index_topk_correctness ``` The test covers the existing short-context path and the gfx942 long-context split path. It compares every score written by the kernel, including causal masking, and verifies the final top-k block sets.
Serve command ```bash export HIP_VISIBLE_DEVICES=0,1,2,3 export VLLM_ROCM_USE_AITER=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER_MOE=0 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 vllm serve EmbeddedLLM/MiniMax-M3-FP8-dynamic \\ --served-model-name minimax-m3 \\ --tensor-parallel-size 4 \\ --block-size 128 \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.92 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 32768 \\ --max-num-seqs 32 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --tool-call-parser minimax_m3 \\ --enable-auto-tool-choice \\ --reasoning-parser minimax_m3 \\ --kv-cache-dtype fp8 \\ --language-model-only \\ --port 8000 \\ --attention-backend ROCM_AITER_UNIFIED_ATTN ```
## Test Result Kernel accuracy: 2/2 passed (short-context fallback + long-context split-K). End-to-end serving on 4x MI325X (gfx942), TP4, `EmbeddedLLM/MiniMax-M3-FP8-dynamic`, FP8 KV cache: | ISL/OSL | Concurrency | Metric | Baseline | This PR | Change | |---------|-------------|--------|----------|---------|--------| | 8k/1k | 4 | Output tok/s | 309.18 | 310.03 | +0.3% | | 8k/1k | 8 | Output tok/s | 470.73 | 469.60 | -0.2% | | 128k/1k | 4 | Output tok/s | 93.48 | 95.24 | +1.9% | | 128k/1k | 4 | TTFT median (ms) | 19229 | 18766 | **-2.4%** | | 128k/1k | 8 | Output tok/s | 105.05 | 106.97 | +1.8% | | 128k/1k | 8 | TTFT median (ms) | 34297 | 33430 | **-2.5%** | | 256k/1k | 2 | Output tok/s | 44.19 | 45.69 | **+3.4%** | | 256k/1k | 2 | TTFT median (ms) | 25539 | 24367 | **-4.6%** | | 256k/1k | 4 | Output tok/s | 49.73 | 51.58 | **+3.7%** | | 256k/1k | 4 | TTFT median (ms) | 42422 | 40513 | **-4.5%** | Long-context prefill improvement scales with sequence length. Short-context workloads are unaffected (selector falls back to unsplit path). Long-context prefill shows a consistent TTFT improvement. | Task | Filter | n-shot | Baseline | This PR | |------|--------|--------|----------|---------| | GSM8K (full, 1319) | flexible-extract | 8 | 93.18% ± 0.69 | 92.87% ± 0.71 | | GSM8K (full, 1319) | strict-match | 8 | 93.18% ± 0.69 | 92.80% ± 0.71 |", "url": "https://github.com/vllm-project/vllm/pull/52059", "createdAt": "2026-08-12T22:04:47Z", "updatedAt": "2026-08-13T06:20:04Z", "timestamp": "2026-08-13T06:20:04Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "akii96", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52060", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fold scalar KV scales in MiniMax-M3 sparse attention", "text": "> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (incorrect expert count caused silent accuracy loss) > > Base image: `vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106` ## Purpose MiniMax-M3 Triton sparse attention currently applies scalar FP8 KV scales to every K and V element loaded inside the attention loop. These scales are loop invariant, so the repeated conversions and multiplications add unnecessary work. ## Suggested Fix For scalar scales (`KV_SCALE_MODE == 1`), fold the K scale into the attention score scale and apply the V scale once to the normalized FP32 accumulator. The change covers AMD prefill, common prefill, and common decode. No-scale and per-token/head scale modes remain unchanged through the existing compile-time dispatch. ## Test Plan ```bash pytest -q tests/kernels/test_minimax_m3_sparse_attn_fp8_scale.py ``` Compares scalar and per-token/head FP8 KV-cache outputs against a dequantized BF16 reference for both prefill and decode.
Serve command ```bash export HIP_VISIBLE_DEVICES=0,1,2,3 export VLLM_ROCM_USE_AITER=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER_MOE=0 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 vllm serve EmbeddedLLM/MiniMax-M3-FP8-dynamic \\ --served-model-name minimax-m3 \\ --tensor-parallel-size 4 \\ --block-size 128 \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.92 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 32768 \\ --max-num-seqs 32 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --tool-call-parser minimax_m3 \\ --enable-auto-tool-choice \\ --reasoning-parser minimax_m3 \\ --kv-cache-dtype fp8 \\ --language-model-only \\ --port 8000 \\ --attention-backend ROCM_AITER_UNIFIED_ATTN ```
## Test Result Kernel accuracy: 4/4 passed (scalar + per-token/head, prefill + decode). End-to-end serving on 4x MI325X (gfx942), TP4, `EmbeddedLLM/MiniMax-M3-FP8-dynamic`, FP8 KV cache: | ISL/OSL | Concurrency | Metric | Baseline | This PR | Change | |---------|-------------|--------|----------|---------|--------| | 8k/1k | 4 | Output tok/s | 309.18 | 315.52 | **+2.0%** | | 8k/1k | 4 | TTFT median (ms) | 1683 | 1541 | **-8.5%** | | 8k/1k | 8 | Output tok/s | 470.73 | 485.37 | **+3.1%** | | 8k/1k | 8 | TTFT median (ms) | 2123 | 2041 | **-3.9%** | | 128k/1k | 4 | Output tok/s | 93.48 | 99.85 | **+6.8%** | | 128k/1k | 4 | TTFT median (ms) | 19229 | 17552 | **-8.7%** | | 128k/1k | 8 | Output tok/s | 105.05 | 113.14 | **+7.7%** | | 128k/1k | 8 | TTFT median (ms) | 34297 | 31230 | **-8.9%** | | Task | Filter | n-shot | Baseline | This PR | |------|--------|--------|----------|---------| | GSM8K (full, 1319) | flexible-extract | 8 | 93.18% ± 0.69 | 92.95% ± 0.71 | | GSM8K (full, 1319) | strict-match | 8 | 93.18% ± 0.69 | 92.87% ± 0.71 |", "url": "https://github.com/vllm-project/vllm/pull/52060", "createdAt": "2026-08-12T22:12:24Z", "updatedAt": "2026-08-13T05:12:34Z", "timestamp": "2026-08-13T05:12:34Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "akii96", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52061", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add native forward-pass metrics emission", "text": "## Purpose Related to #38760. Add an opt-in native vLLM forward-pass metrics (FPM) pipeline for production per-iteration observability. This moves production FPM emission out of an external custom scheduler while keeping the disabled path lightweight. The change: - defines the FPM v1 wire contract and utilities together in `vllm/v1/metrics/forward_pass_metrics.py`; - captures scheduled and queued prefill/decode aggregates from an explicit scheduler interface; - measures `execute_model` directly with pooled CUDA events on the executor output rank; - drains completed events without synchronizing the inference hot path and only waits during the cold transition to idle/shutdown; - joins delayed CUDA timings to their originating scheduler snapshots by iteration ID; - follows vLLM's existing native [KV-event ZMQ emission pattern](https://github.com/vllm-project/vllm/blob/98f86b9c02329200a0390aecfe598e27928cbf40/vllm/distributed/kv_events.py#L290-L367): one publisher per DP rank, a background publishing thread, and `(topic, sequence, msgpack payload)` multipart frames ([framing reference](https://github.com/vllm-project/vllm/blob/98f86b9c02329200a0390aecfe598e27928cbf40/vllm/distributed/kv_events.py#L459-L466)); - keeps FPM delivery intentionally best-effort and non-blocking, with a bounded queue and no replay endpoint, so observability cannot backpressure inference; and - exposes opt-in CLI/config fields for the port, worker ID, and publisher queue bound. When disabled, no CUDA events, publisher, or pending snapshot state are allocated. This does not duplicate #48292. That PR records the wall-clock time spent in `future.result()` in iteration-detail logs. This PR implements a versioned streaming metrics contract, scheduler state aggregation, direct CUDA timing, and external ZMQ emission. It also does not move external self-benchmark state machines into vLLM. ## Difference from the current Dynamo implementation The references below are pinned to [`ai-dynamo/dynamo@427046c`](https://github.com/ai-dynamo/dynamo/tree/427046cf5c0d84c2d879d2c82b1e127665540961) so this comparison remains stable as Dynamo evolves. - **Ownership and activation:** Dynamo currently enables production FPM by replacing vLLM's scheduler with `InstrumentedScheduler`; an already configured custom scheduler prevents that injection ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/args.py#L314-L336)). This PR makes production FPM a native, opt-in vLLM engine facility and uses a narrow scheduler-state interface, so production FPM no longer requires a Dynamo scheduler subclass. - **Iteration timing:** Dynamo records host `time.monotonic()` timestamps around `schedule()` and model-output arrival, then estimates `wall_time` from either consecutive output updates or the schedule-to-update interval ([measurement description](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L52-L68), [implementation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1594-L1611), [calculation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1644-L1674)). This PR records pooled CUDA events immediately around `execute_model`, drains them asynchronously, and joins the result to the originating engine iteration. Thus the engine directly emits measured `execute_model` GPU timeline duration instead of a scheduler-derived approximation. - **Aggregation and emission:** Dynamo currently computes scheduled/queued aggregates in `InstrumentedScheduler` and passes each payload to its own background ZMQ publisher ([aggregation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1676-L1692), [publisher](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1347-L1439)); Dynamo's parent process then relays that raw stream to the event plane ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/main.py#L445-L488)). This PR moves the aggregation, iteration association, CUDA timing, and raw ZMQ emission into vLLM. The existing Dynamo relay remains the downstream integration point. - **Self-benchmarking:** Dynamo benchmark mode currently requires the same `InstrumentedScheduler` ([configuration](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/args.py#L338-L357)), and its benchmark state machine controls synthetic scheduler outputs and FPM collection inside that subclass ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1523-L1554)). This PR deliberately leaves self-benchmarking in Dynamo; Dynamo can retain its custom scheduler for benchmark mode while ordinary serving uses native vLLM FPM. - **Wire contract:** Dynamo's current payload is version 1 ([version](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/common/forward_pass_metrics.py#L45), [schema](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/common/forward_pass_metrics.py#L153-L189)). This PR keeps `version=1` and explicitly labels the changed `wall_time` semantics with `timing_scope=execute_model_cuda`; it does not bump the wire version. ## Test Plan - Run the focused FPM and argument parsing unit tests. - Run all staged pre-commit hooks. - Compare FPM disabled/enabled serving performance with an interleaved A/B campaign and a live subscriber. - Compare every non-timing FPM field against the existing Dynamo scheduler implementation under serial, batched, prefix-cache, chunked-prefill, queued-prefill, decode-heavy, and forced-preemption traffic. ## Test Result All staged pre-commit hooks passed: ```bash .venv/bin/pre-commit run ``` This included ruff check/format, mypy, SPDX, forbidden-import, CUDA-API, configuration-default, and repository-specific validation hooks. The focused local pytest command was attempted after installing the documented precompiled editable build: ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto .venv/bin/python -m pytest \\ tests/v1/metrics/test_forward_pass_metrics.py \\ tests/engine/test_arg_utils.py -q ``` Collection remains blocked on this host because its environment contains CPU Torch while the installed precompiled vLLM extension requires `libtorch_cuda.so`. No test assertion was reached; CI should run these tests in a consistent build environment. GPU validation used one H100 NVL with `Qwen/Qwen3-0.6B` and controlled images built from the same vLLM source: - 132/132 overhead benchmark samples completed successfully across serial, concurrent, decode-heavy, prefill-heavy, chunked-prefill, prefix-cache, and open-loop traffic. - The stable worst-case signal was about 4.4 microseconds per iteration (`+0.283%` TPOT and `-0.269%` output throughput) for concurrency-1 short serial traffic. Decode-heavy output throughput changed by `+0.036%`. Longer concurrent and prefix-cache reruns showed no reproducible throughput regression. - A live subscriber received 151,263 FPM messages with zero observed sequence gaps, queue-full/drop logs, publisher errors, or server failures. - Native FPM matched the existing scheduler implementation on 1,249/1,249 deterministic active iterations for every non-timing field, including scheduled/queued request counts, token and KV-token sums, and variances. Forced memory pressure observed queued prefill and preempted decode states on both implementations. - All serving requests succeeded and prompt/completion token totals matched; this change does not alter model output or accuracy. The payload continues to use wire `version=1`. Timing values intentionally differ from the existing scheduler approximation, so consumers must use the explicit `timing_scope=execute_model_cuda` field when interpreting `wall_time`. This implementation and validation were developed with assistance from OpenAI Codex. This draft PR is intentionally open for the submitter's line-by-line human review and must not be marked ready until that review is complete. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR and related issue are described. - [x] The test plan and commands are included. - [x] Test and performance results are included. - [x] No documentation update is required for this opt-in internal interface.
", "url": "https://github.com/vllm-project/vllm/pull/52061", "createdAt": "2026-08-12T22:38:02Z", "updatedAt": "2026-08-12T23:21:45Z", "timestamp": "2026-08-12T23:21:45Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "mrv2" ], "author": "tedzhouhk", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52062", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert KV block zeroing generalization due to ROCm launch overflow", "text": "## Summary - Revert #51749, which generalized KV block zeroing from `FullAttentionSpec` to every `AttentionSpec`. - On DeepSeek-V4-Pro with ROCm and 64 concurrent GSM8K requests, the broader zeroing path makes `_zero_kv_blocks_kernel` launch with a grid that exceeds the AMD Triton driver's signed integer limit. - The resulting `OverflowError: signed integer is greater than maximum` kills the engine and causes all in-flight `/v1/chat/completions` requests to return HTTP 500. ## Duplicate check I searched open PRs for `51749`, `KVBlockZeroer OverflowError ROCm`, and `Generalize KV block zeroing revert`; no existing PR addresses this failure. ## Test plan - [x] `.venv/bin/python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/worker/test_kv_block_zeroer.py -q` — 14 passed - [x] Before revert: `dsv4_rocm_bench/run_gsm8k.sh` at concurrency 64 reproduced the worker failure in `KVBlockZeroer.zero_block_ids`, followed by `EngineDeadError` and HTTP 500 responses. - [x] After revert: the full 1,319-sample GSM8K run completed at concurrency 64 in 192.2 seconds. - strict exact match: 0.96285 - flexible exact match: 0.96209 This PR intentionally makes no claim about output quality from raw `/v1/completions` requests; DeepSeek-V4-Pro expects chat-template formatting, and the post-revert GSM8K evaluation used `/v1/chat/completions`. AI assistance was used for log triage, identifying the failing code path, and drafting this PR. The human submitter reviewed the changes and ran the tests above. Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/52062", "createdAt": "2026-08-12T22:50:19Z", "updatedAt": "2026-08-13T13:45:31Z", "timestamp": "2026-08-13T13:45:31Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "needs-rebase" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52063", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Avoid AITER FP8 BMM for MLA K projection", "text": "## Purpose This changes the ROCm AITER FP8 MLA decode path so the K-side up-projection stays dense after the post-load transform, while the V-side up-projection continues to use the AITER FP8 BMM. The K-side MLA projection is a skinny batched matmul. Requantizing the transformed K weight and routing it through AITER FP8 BMM adds overhead for this shape and adds another quantization step. Keeping K dense reuses the existing `torch.bmm` path, keeps the K-side math tied directly to the dequantized model weight, and leaves the V-side AITER path unchanged. What changed: - remove the AITER FP8 runtime branch for the K-side MLA projection - keep `W_UK_T` dense in `process_weights_after_loading` for the AITER FP8 path - keep quantizing/precompiling `W_V` for the V-side AITER FP8 BMM - add a unit test that checks both the post-load storage contract and the K-side projection math This PR was developed with AI assistance. I reviewed the changed code paths and ran the checks below. ## Duplicate-work check No linked issue exists for this change. I checked for open duplicate work with: - `gh pr list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` - `gh pr list --repo vllm-project/vllm --state open --search \"rocm aiter mla fp8 bmm\"` - `gh pr list --repo vllm-project/vllm --state open --search \"triton_fp8_bmm mla warmup\"` - `gh issue list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` The closest related open PR is #52096, which is also mine. It is related but not a duplicate: #52096 changes startup precompile bounds for the AITER FP8 BMM path, while this PR changes the runtime K-side MLA projection path. If one lands first, the other may need a small rebase in the same `process_weights_after_loading` area. ## Test Plan Local focused correctness: ```bash PYTHONPATH=. uv run --no-sync --python 3.12 \\ --with-requirements requirements/common.txt \\ --with pytest --with torch --with tblib \\ pytest -q --confcutdir=tests/v1/attention \\ tests/v1/attention/test_mla_backends.py::test_mla_kv_cache_spec_uses_layer_cache_dtype \\ tests/v1/attention/test_mla_backends.py::test_mla_post_load_preserves_runtime_weight_addresses \\ tests/v1/attention/test_mla_backends.py::test_mla_aiter_fp8_post_load_keeps_k_side_dense ``` Local lint: ```bash pre-commit run --files \\ vllm/model_executor/layers/attention/mla_attention.py \\ tests/v1/attention/test_mla_backends.py ``` MI300X focused correctness and microbench remain from the earlier draft run. I have not rerun full serve/generation on this branch after rebasing to current `origin/main`. ## Test Result Local focused correctness: `4 passed, 14 warnings`. Local lint: `pre-commit run --files ...` passed. Earlier MI300X focused correctness: `4 passed, 14 warnings`. Earlier MI300X isolated K-side projection results: | tokens | old AITER FP8 K BMM median us | dense K `torch.bmm` median us | median speedup | |---:|---:|---:|---:| | 1 | 35.205 | 14.536 | +142.2% | | 2 | 34.709 | 14.630 | +137.2% | | 4 | 34.777 | 14.795 | +135.1% | | 8 | 34.813 | 14.665 | +137.4% | | 16 | 34.778 | 14.692 | +136.7% | | 32 | 34.852 | 14.800 | +135.5% | | 64 | 34.728 | 13.852 | +150.7% | | 128 | 34.272 | 13.934 | +146.0% | | 256 | 34.517 | 13.887 | +148.6% | Caveat: I could not complete a full editable vLLM ROCm build on the current RunPod image because it ships PyTorch `2.10.0+rocm7.1.1`, while current vLLM's ROCm `_C_stable_libtorch` extension expects a newer PyTorch stable C++ API. The build failed before this patch was involved, at `_C_stable_libtorch` with missing `torch::stable::Tensor::layout`. A full serve/generation check should be run in the official vLLM ROCm dev image or another environment with the current supported ROCm/PyTorch stack.", "url": "https://github.com/vllm-project/vllm/pull/52063", "createdAt": "2026-08-12T22:53:37Z", "updatedAt": "2026-08-13T04:39:55Z", "timestamp": "2026-08-13T04:39:55Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "rohash123", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52064", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Mirror external test assets in vLLM S3", "text": "## Summary - mirror externally hosted video, image, and GSM8K test assets into the public vLLM S3 bucket - update affected tests to use the shared `VLLM_S3_BUCKET_URL` constant ## Why [Buildkite build 83608](https://buildkite.com/vllm/ci/builds/83608) had multiple test failures caused by direct dependencies on third-party hosts, including `RemoteDisconnected` while fetching OpenCV and Bogotobogo video fixtures. The mirrored objects are publicly readable from `vllm-public-assets` in `us-west-2`, and each object was verified byte-for-byte against its source. ## Duplicate work I searched open PRs for the affected video fixtures and GSM8K/JinaVL S3 mirroring and found no overlapping change. #52043 handles separate missing hybrid-dependency wheel failures and is intentionally outside this PR. ## Validation - full pre-commit selection against `origin/main...HEAD` passed, including Ruff, formatting, mypy, and repository validation hooks - `git diff --check` - all 7 mirrored test assets fetched from public S3 and matched their source SHA256 checksums - OpenCV decoded all mirrored videos successfully: - `Megamind.avi`: 270 frames at 23.976 fps - `slow_traffic_small.mp4`: 914 frames at 29.970 fps - `vtest.avi`: 795 frames at 10 fps - GSM8K train and test mirrors passed line-by-line JSON parsing - targeted pytest collection was attempted, but the local test environment is missing `numpy` and fails while loading `tests/conftest.py` Model evaluations were not run because this changes CI asset locations only and does not affect model behavior. ## AI assistance AI assistance was used to investigate the CI failures and prepare this change. The human submitter is responsible for reviewing every changed line and validating the change before merge.", "url": "https://github.com/vllm-project/vllm/pull/52064", "createdAt": "2026-08-12T22:55:52Z", "updatedAt": "2026-08-13T02:08:59Z", "timestamp": "2026-08-13T02:08:59Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "ready", "ci/build", "multi-modality" ], "author": "khluu", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52066", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Fix ragged-N FP8 block-scale GEMM and sparse-MLA metadata sync", "text": "Two XPU-only fixes needed to serve sparse-MLA FP8 MoE models (e.g. GLM-5.2, DeepSeek DSA) on Intel GPUs. 1. Ragged-N FP8 block-scale GEMM (scaled_mm/xpu.py): oneDNN derives the weight block-group width as wei_group_n = N/ceil(N/128), so when the output dim N is not a multiple of 128 it is < 128 and oneDNN cannot create the matmul primitive (\"could not create a primitive descriptor\"). This hits the MLA fused projections whose N is ragged (fused_qkv_a_proj N=2624, kv_a_proj_with_mqa N=576). Pad the weight rows up to the 128 boundary with zeros (which contribute nothing), run the GEMM, and slice the padding off the output; no-op when N is already a multiple of 128. Verified numerically: padded ragged-N matches a block-dequant reference at the same ~0.004 rel-err floor as divisible-N. 2. Sparse-MLA metadata sync (xpu_mla_sparse.py): The shared MLA layer (mla_attention.py::forward_impl) unconditionally reads num_decodes/num_prefills/num_decode_tokens on every MLA metadata; the CUDA sparse backends carry them via SparseMLACommonMetadataBuilder, but the XPU sparse backend built its own metadata without them, so a sparse-MLA run on XPU crashed with \"'XPUMLASparseMetadata' object has no attribute 'num_decode_tokens'\". Add the fields and populate them so all tokens route through the top-k sparse MQA path (num_decode_tokens == num_actual_tokens, num_prefills == 0), which this backend already implements for both prefill and decode; this keeps num_mha_tokens at 0 and avoids the dense-MHA prefill branch (whose prefill-only fields this backend does not provide). ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52066", "createdAt": "2026-08-12T23:02:18Z", "updatedAt": "2026-08-13T05:55:37Z", "timestamp": "2026-08-13T05:55:37Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu" ], "author": "libinta", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52067", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Forward ownership in KV cache events", "text": "## Purpose Add optional ownership information to KV cache events so events emitted by a KV secondary tier can be distinguished from framework-owned events. No conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py", "url": "https://github.com/vllm-project/vllm/pull/52067", "createdAt": "2026-08-12T23:23:46Z", "updatedAt": "2026-08-13T01:11:25Z", "timestamp": "2026-08-13T01:11:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "mkhazraee", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52068", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Preserve KV event metadata until final residency removal", "text": "## Purpose Track offloaded-block residencies by medium and ownership so removing one residency does not discard metadata still needed by another. Depends on PR #52067, no conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py", "url": "https://github.com/vllm-project/vllm/pull/52068", "createdAt": "2026-08-12T23:26:46Z", "updatedAt": "2026-08-12T23:27:25Z", "timestamp": "2026-08-12T23:27:25Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kv-connector" ], "author": "mkhazraee", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52070", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][Spec Decode] Disable EAGLE cache-hit block drop for always-K=0 dynamic DSD", "text": "## Purpose Opt-in skip of EAGLE-style last-block drop on prefix-cache hits when dynamic speculative decoding is **always K=0**. Default is off, so stock cache-hit behavior and observed greedy tokens are unchanged. Always-K=0 DSD (`num_speculative_tokens_per_batch_size=[[1,16,0]]`) produces no draft tokens, but `use_eagle=True` still drops the last matched KV block on a prefix-cache hit. That inflates scheduled-token volume on a long-context warm workload (stock: 4 steps / ~24k tokens per burst vs nospec ~1 step). This PR keeps `use_eagle=True` and only disables that drop when the operator sets `disable_eagle_cache_drop_for_k0=True` **and** all of: prefix caching on, no KV connector, `use_eagle()` (EAGLE / EAGLE3 / MTP / DFlash / DSpark), dense lookup `max(lookup[1:])==0`. Related: #49548 (always-K=0 slice only; not a universal fix). This is a **performance tradeoff**, not a correctness fix. Greedy outputs can change when the flag is on (measured divergence at generated pos 22 multi-request / pos 67 single-request vs stock K=0). ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/v1/core/test_eagle_cache_drop_k0.py \\ tests/v1/core/test_mamba_align_chunk_split.py \\ tests/v1/core/test_prefix_caching.py::test_eagle_enabled_removes_last_block \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_partial_blocks \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_sliding_window \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_combinations_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_swa_alignment_caches_extra_block \\ tests/v1/spec_decode/test_dynamic_sd.py -q ``` Enable (example): ```python speculative_config={ \"method\": \"mtp\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [[1, 16, 0]], \"disable_eagle_cache_drop_for_k0\": True, } # plus enable_prefix_caching=True; no KV connector ``` ## Test Result CPU unit tests above: **64 passed** (2026-08-12, local). ThinkingCap-Qwen3.6-27B-FP8, dual RTX 5090 TP=2, always-K=0 `[[1,16,0]]`, ~49k prompt with prefix-cache hit, 8 sessions × 2 rounds × 128 tokens, inproc. GPU validation used commit `da159d447d` overlaid into the production venv. The only extra local change was an uncommitted scheduler JSONL helper for the burst census; it does not change scheduling decisions. **Primary comparison (this flag, one outing):** C0 **82.6 t/s** (n=1) vs B2 **136.7 t/s** (n=3, stdev 0.3), about **1.65×**. Mixed C3 `[[1,2,2],[3,16,0]]` does not enable the flag. **Burst census (same outing):** | Config | Warm steps/burst | Warm Σ tok/burst | |---|---:|---:| | Stock always-K=0 (C0) | 4 | 24,238 | | This change (B2, flag on) | 2 | 11,429 | | Coarse `use_eagle=False` (B1, earlier run) | 2 | 11,429 | B2 matches B1’s 4→2 / 24k→11k shape. `use_eagle` stays True. **Earlier local prototype (not this flag; do not mix with the numbers above):** same workload family, util 0.90, one-rep: C0 89.4 t/s, local B2 (`VLLM_SCHED_WARM_BATCH`) 147.0 t/s, B1 147.7 t/s. Nearby nospec ~178 t/s. Residual vs nospec is not attributed. Greedy identity vs stock K=0 with the flag on: **FAIL** at pos 22 (multi) / pos 67 (single), 1 rep each. **Small accuracy probe** (not GSM8K): 8 grade-school integer problems, greedy, ~2k-token shared stem cached first so the last-block drop actually runs. C0 **7/8**, B2 **8/8**. The seven unambiguous items matched. The eighth is an ambiguous wording trap (capacity 80 vs 80−15=65); C0 answered 65, B2 answered 80. Do **not** read 8/8 vs 7/8 as a quality win. Cliff check only: the flag did not break the clear math items. No named model eval (GSM8K / `tests/evals/`) was run. ## Why this is not a duplicate No open PR implements an always-K=0 opt-in skip of `drop_eagle_on_cache_hit`. **#50897** is the general correctness approach (successor-aware hashes so the last block can be kept without changing greedy tokens). It is **not** a substitute we can cite for this workload today: - 43 files; **merge-conflicting** as of 2026-08-11. Collaborator review asked to **split** it (too pervasive; still keeps the old drop path, so core complexity goes up). A rebase alone does not address that. No LGTM. - We have **not** measured #50897 on the always-K=0 ~49k warm-burst bench. Same speed as this flag is a hypothesis. This PR is a small, default-off option for that measured always-K=0 volume defect. If successor-aware hashing lands and removes the defect, **deprecate and remove this flag**. The two do not conflict today (off by default; no hash-format change). Not discussed on #50897. | Related | Difference | |---|---| | #50897 | Correctness-preserving hash redesign (see above). Not merge-ready as-is; unmeasured here. | | #51769 | Warns when EAGLE-style methods cost a large prefix-cache hit. Diagnostic only. | | #51295 | Hybrid attention miss/corruption from eagle drop. Different bug. | | #48375 / #45614 | Mamba eagle-drop plumbing / hit accuracy. Not DSD K=0 scheduling volume. | | #51466 / #51575 | Separate DSD work; #51466 was withdrawn after a negative policy experiment. Neither is bundled. | KV-connector / Mooncake Store eagle-drop paths are **out of scope** (gate requires `connector is None`). ## AI disclosure AI assistance was used to draft the implementation and tests. A human (Greg) is accountable for the PR and will review every changed line before merge. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52070", "createdAt": "2026-08-12T23:52:41Z", "updatedAt": "2026-08-13T17:35:53Z", "timestamp": "2026-08-13T17:35:53Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "Suppressor72", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52072", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Apply suppress_tokens on the Gemma 4 MTP sparse path", "text": "## Purpose When `use_ordered_embeddings` is set on a Gemma 4 MTP drafter, `Gemma4Proposer._greedy_sample` samples through `get_top_tokens` rather than `compute_logits`: ```python # vllm/v1/spec_decode/gemma4.py:113 return self.model.get_top_tokens(hidden_states) ``` That reaches `Gemma4MTPMaskedEmbedder.get_top_tokens`, which takes a sparse argmax directly over the centroid-selected candidates: ```python logits, indices = self._select_and_score(hidden_states, lm_head_weight) return indices.gather(-1, logits.argmax(-1, keepdim=True)).squeeze(-1) ``` `compute_logits` — the only place `suppress_tokens` is applied — is never called on this path, so **the drafter can propose tokens the generation config marks as suppressed**. The same applies to the CUDA-graph-captured variant, since `_setup_centroids_cuda_graphs` captures `masked_emb.get_top_tokens` directly. This is a correctness gap rather than a crash, so it is silent: the suppressed token is proposed by the drafter and then either accepted or rejected by the verifier depending on the target model's own distribution. ## Approach Give `Gemma4MTPMaskedEmbedder` a boolean mask over the vocabulary, populated from the draft model's generation config in `Gemma4MTP.__init__`, and apply it to the candidate logits before the argmax. The mask lives on the embedder rather than on `Gemma4MTP` because there are two call sites — `Gemma4MTP.get_top_tokens` and the proposer's direct `masked_emb.get_top_tokens` during graph capture — and only module-level state covers both. It is registered with `persistent=False`: it is derived from the generation config, not a checkpoint weight, and a persistent buffer would surface as an unexpected key during weight loading. The dense path is untouched — `forward` still returns unsuppressed full-vocab logits and `compute_logits` masks them, which the test asserts explicitly. ## Test plan Adds `tests/model_executor/test_gemma4_mtp_sparse_suppress.py` (CPU-only, `@pytest.mark.cpu_test`). `Gemma4MTPMaskedEmbedder` has no `VllmConfig` dependency, so these construct it for real with a deterministic centroid layout: an identity `token_ordering`, fixed centroid weights so the selected candidate set is always tokens 0–7, and an `lm_head_weight` giving an unambiguous ranking (token 5 > 2 > 6). - suppressing the best candidate returns the second-best, and suppressing both returns the third; - `[]` and `None` are no-ops; - `suppress_mask` stays out of `state_dict` while `token_ordering` remains in it; - suppression holds across a batched `hidden_states`; - `forward()` output is byte-identical with and without suppression configured. ## Known degenerate case, deliberately not matched If *every* centroid-selected candidate is suppressed, the sparse argmax returns an arbitrary suppressed candidate, whereas the dense path returns the lowest-id non-selected token — non-selected positions are filled with `finfo.min`, which compares above `-inf`. Reaching this requires `suppress_tokens` to cover every token in the selected clusters. I left the behavior rather than encoding a somewhat arbitrary dense-path artifact as a contract, but happy to match it if reviewers prefer. ## Relationship to existing work #48693 by @dumko2001 also touches sparse-path parity as part of a broader Gemma 4 refactor; it has been in draft with merge conflicts since July. None of its code is used here. This is deliberately scoped to the sparse suppression gap alone so it can be reviewed independently. Depends on nothing, but is adjacent to the separate CUDA-graph fix for `compute_logits` on the dense path (#48503). ## Validation caveat No GPU access here, so this has not been exercised on a real Gemma 4 MTP run, and the test file has not been executed against a real vLLM install. What was verified: the assertions above all pass against the actual patched `Gemma4MTPMaskedEmbedder` class source executed standalone against real torch, and `ruff check` / `ruff format` (pinned v0.14.0) are clean. Hardware confirmation would be welcome. ## AI assistance disclosure Parts of this change and its test were drafted with AI assistance (Claude). I have reviewed every changed line and validated the behavior described above.", "url": "https://github.com/vllm-project/vllm/pull/52072", "createdAt": "2026-08-12T23:54:08Z", "updatedAt": "2026-08-12T23:54:44Z", "timestamp": "2026-08-12T23:54:44Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "aminsamir45", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52073", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Test extract_hidden_states on NemotronH hybrid models", "text": "## Purpose `NemotronHForCausalLM` is the architecture vLLM's own contributor docs point to as the reference implementation for Mamba2 + attention hybrids, and it already implements the `EagleModelMixin` / `SupportsEagle3` hooks that `extract_hidden_states` consumes. It has no coverage in the `extract_hidden_states` integration tests. The only hybrid verifier currently covered there is Qwen3.5 (`test_extract_hidden_states_qwen35_hybrid_smoke`), whose stack is Mamba2 + full attention. NemotronH differs in two ways that both land on the arithmetic this feature is sensitive to: - It builds its layer stack from `hybrid_override_pattern`, interleaving **MLP and MoE** layers with Mamba2 and attention. A given aux layer id is therefore not necessarily an attention or Mamba layer, and the stack contains two block types the current tests never exercise. - `NemotronHForCausalLMConfig.verify_and_update_config` forces `mamba_ssm_cache_dtype` to `float32`, so its Mamba page size is materially larger than a model whose SSM state stays in the model dtype. Both feed the Mamba page size, and the hidden-state cache group's block size is derived from the common page it has to align to (`get_kv_cache_groups`, the `HiddenStateCacheSpec` branch). That is exactly the arithmetic that regressed silently for hybrid verifiers in #46301, where shapes stayed correct while every extracted value read back as zero — a failure mode that yielded a draft model with ~0% acceptance and \"converged\" cleanly in training before anyone noticed. **This is coverage for a supported-but-untested combination, not a bug fix.** Before writing the test I verified on an A100-80GB against `nvidia/Nemotron-H-8B-Base-8K` that extraction works today and is numerically correct, and that the resolved geometry genuinely differs from the covered case: | | attention block size | mamba page padding | |---|---|---| | Qwen3.5-0.8B (covered today) | 544 | 1.49% | | Nemotron-H-8B | **1056** | **1.15%** | With real weights and aux ids `[10, 26, 40]` over a 13-token prompt, the extracted states are distinct and deepen as expected — i.e. correct, not a repeated buffer: | pair | cosine | max abs diff | |---|---|---| | L10 vs L26 | 0.974 | 33.1 | | L10 vs L40 | 0.318 | 1866.2 | | L26 vs L40 | 0.334 | 1882.5 | `abs_mean` per layer: 0.084 → 0.377 → 2.268. Follows the precedent of #39949 (hybrid support) and #46301 (hybrid block-size fix). ## Test Plan Adds `test_extract_hidden_states_nemotron_h_hybrid_smoke`, built on a tiny synthetic `NemotronHConfig` rather than a real checkpoint, so it fits the 18 GB device the `Extract Hidden States Integration` CI job runs on. The pattern `\"M-M*M-ME*E-ME\"` covers all four layer types, and the aux layer ids `[4, 8, 12]` capture the outputs of an attention layer, an MoE layer and a Mamba2 layer respectively (aux id `k` is the residual stream after layer `k-1`). Assertions come from the existing `get_and_check_output` helper: shape, token-id alignment, and states-not-all-zero. ```bash export VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s v1/kv_connector/extract_hidden_states_integration/test_extraction.py \\ -k \"nemotron_h or qwen35\" ``` ## Test Result Both the new test and the existing Qwen3.5 hybrid test pass, on one A100-80GB, against this branch: ``` tests/.../test_extraction.py::test_extract_hidden_states_qwen35_hybrid_smoke PASSED tests/.../test_extraction.py::test_extract_hidden_states_nemotron_h_hybrid_smoke PASSED =========== 2 passed, 2 deselected, 18 warnings in 622.10s (0:10:22) =========== ``` The Qwen3.5 case is included as a no-regression control. Lint on the changed file: ``` ruff check -> All checks passed! ruff format -> 1 file already formatted ``` Worth noting that the tiny config lands in its own alignment regime rather than mirroring a real checkpoint's, which is useful coverage in itself: ``` Using auxiliary layers from speculative config: (4, 8, 12) Setting attention block size to 48 tokens to ensure that attention page size is >= mamba page size. Padding mamba page size by 29.73% to ensure that mamba page size and attention page size are exactly equal. ``` (against 1056 / 1.15% for the real Nemotron-H-8B, and 544 / 1.49% for Qwen3.5.) ## Note on what this test does and does not assert Worth stating explicitly, because it bounds what the test can catch. Under `load_format=\"dummy\"` the weights are drawn from `U(-1e-3, 1e-3)`, and NemotronH's MLP activation is relu², so each block contributes on the order of `1e-10` of the residual and all captured layers read back as a bfloat16 rounding of the embeddings. (Measured: `abs_mean` identical to the last digit across all three aux slots under dummy weights, matching a standalone simulation of the embedding magnitude to three significant figures; the real-weights numbers above are the contrast. Qwen3.5 does not show this because SwiGLU stays linear near zero.) So this test asserts the plumbing — shapes, token alignment, non-zero states through a Mamba2 + attention + MoE stack — and deliberately does not assert per-layer values, which `test_extract_hidden_states_with_predictable_dummy_model` already covers with a model that emits known per-layer outputs. The docstring says the same, so a passing run isn't later read as a stronger guarantee than it gives. Prompts are passed as explicit `TokensPrompt` ids rather than text: the tiny config declares `vocab_size=1000` while the borrowed TinyLlama tokenizer emits ids up to ~18.9k, which indexes past the embedding table and trips a device-side assert. 🤖 Generated with [Weave Router](https://router.workweave.ai)", "url": "https://github.com/vllm-project/vllm/pull/52073", "createdAt": "2026-08-13T00:18:21Z", "updatedAt": "2026-08-13T00:58:37Z", "timestamp": "2026-08-13T00:58:37Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "aminsamir45", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52074", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] extract_hidden_states supports chunked prefill", "text": "## Summary The `extract_hidden_states` docs state: > Chunked prefill is not compatible with this feature and must be disabled. That appears to be stale. The feature's own integration test exercises chunked prefill deliberately, in `tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py`: ``` 2. **Chunked prefill**: max_num_batched_tokens=128 with ~500-token prompts so each is split across multiple scheduler iterations — verifies hidden states are reassembled correctly. ``` It runs with `max_num_batched_tokens=128` against ~500-token prompts, so each prompt is split across scheduler iterations, and the test asserts the states are reassembled correctly. I found this while enabling the feature on a NemotronH hybrid model, where the engine also ran with chunked prefill enabled and returned correct per-layer hidden states. ## Test Plan Docs-only change; no code paths touched. The behavioural claim is backed by the existing test above. ## Test Result n/a — documentation only.", "url": "https://github.com/vllm-project/vllm/pull/52074", "createdAt": "2026-08-13T00:19:33Z", "updatedAt": "2026-08-13T00:58:27Z", "timestamp": "2026-08-13T00:58:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "aminsamir45", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52075", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Improve context-parallel backend error guidance", "text": "## Summary Improve context-parallel compatibility errors so they give users concrete recovery steps. - tell DCP users to select another backend with `--attention-backend`, or disable DCP with `--decode-context-parallel-size 1` - add equivalent actionable guidance to the PCP compatibility error - add regression tests for both messages ## Related issue Closes #28407 ## Testing - Python syntax checked for both changed files - Full pytest execution was not available in the local environment because PyTorch is not installed", "url": "https://github.com/vllm-project/vllm/pull/52075", "createdAt": "2026-08-13T00:33:24Z", "updatedAt": "2026-08-13T00:35:28Z", "timestamp": "2026-08-13T00:35:28Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "veerareddyvishal144", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52076", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Clearer comments in `BlockPool.free_blocks()`", "text": "The comments explaining eviction precedence in `BlockPool.free_blocks()` were ambiguous/confusing. Make them clearer / more explicit.", "url": "https://github.com/vllm-project/vllm/pull/52076", "createdAt": "2026-08-13T00:42:30Z", "updatedAt": "2026-08-13T03:46:23Z", "timestamp": "2026-08-13T03:46:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "njhill", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52077", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle empty YAML config in `--config` parsing", "text": "## Description Passing an empty or comments-only YAML file to `--config` caused `yaml.safe_load()` to return `None`, which then crashed with `AttributeError: 'NoneType' object has no attribute 'items'` in `FlexibleArgumentParser.load_config_file`. This PR: - Treats `None` (empty/comments-only YAML) as an empty configuration so the parser proceeds with defaults and any explicit CLI args. - Validates that the loaded YAML is a mapping at the top level. For lists, scalars, or other non-mapping types, raises a clear `ValueError` with an actionable message instead of an obscure internal `AttributeError`. ## Fix In `vllm/utils/argparse_utils.py` `load_config_file()`: ```python if config is None: config = {} if not isinstance(config, dict): raise ValueError( \"Config file must contain a YAML mapping at the top level. \" f\"Got {type(config).__name__} instead in {file_path}.\" ) ``` ## Tests Added 5 regression tests in `tests/utils_/test_argparse_utils.py`: - `test_empty_yaml_config` — empty YAML file treated as empty config - `test_comments_only_yaml_config` — comments-only YAML treated as empty config - `test_empty_yaml_config_with_cli_overrides` — empty YAML + CLI args keep CLI values - `test_yaml_top_level_list_rejected` — top-level YAML list raises `ValueError` - `test_yaml_top_level_scalar_rejected` — top-level YAML scalar raises `ValueError` All existing tests continue to pass. Closes #51618", "url": "https://github.com/vllm-project/vllm/pull/52077", "createdAt": "2026-08-13T00:58:25Z", "updatedAt": "2026-08-13T00:59:01Z", "timestamp": "2026-08-13T00:59:01Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "veerareddyvishal144", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52078", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Avoid redundant mask compute in GDN metadata build", "text": "## Purpose The spec decode detection in GDNAttentionMetadataBuilder.build() computed tensors multiple times: 1. computed the `num_decode_draft_tokens_cpu >= 0` mask twice in line 192 and 200. Each >= 0 creates a fresh boolean tensor. 2. computed `~spec_sequence_masks_cpu` 4 times This PR avoid redundant compute: 1. Compute `num_decode_draft_tokens_cpu` only once 2. Only compute `num_decode_draft_tokens_cpu[spec_sequence_masks_cpu].sum().item()` iff `num_spec_decodes == 0` is false 3. Compute `~spec_sequence_masks_cpu` once into `non_spec_sequence_masks_cpu` 4. Behavior is preserved exactly ## Profiling Main: \"Screenshot PR: \"Screenshot Main: `aten::ge` launched twice. PR: `aten::ge` launched once. ## Benchmark ``` vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"z-lab/Qwen3.6-35B-A3B-DFlash\",\"method\":\"dflash\",\"num_speculative_tokens\":8}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 84.82 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.71 Output token throughput (tok/s): 212.22 Peak output token throughput (tok/s): 62.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 405.69 ---------------Time to First Token---------------- Mean TTFT (ms): 113.86 Median TTFT (ms): 131.82 P99 TTFT (ms): 142.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.35 Median TPOT (ms): 4.45 P99 TPOT (ms): 5.99 ---------------Inter-token Latency---------------- Mean ITL (ms): 16.15 Median ITL (ms): 16.15 P99 ITL (ms): 16.64 ---------------Speculative Decoding--------------- Acceptance rate (%): 34.34 Acceptance length: 3.75 Drafts: 4827 Draft tokens: 38616 Accepted tokens: 13260 Per-position acceptance (%): Position 0: 75.08 Position 1: 55.13 Position 2: 41.06 Position 3: 31.32 Position 4: 24.72 Position 5: 19.23 Position 6: 15.66 Position 7: 12.51 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 206.67 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.65 Output token throughput (tok/s): 1393.52 Peak output token throughput (tok/s): 585.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2494.53 ---------------Time to First Token---------------- Mean TTFT (ms): 209.36 Median TTFT (ms): 193.29 P99 TTFT (ms): 414.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.76 Median TPOT (ms): 10.80 P99 TPOT (ms): 17.09 ---------------Inter-token Latency---------------- Mean ITL (ms): 38.40 Median ITL (ms): 22.08 P99 ITL (ms): 156.83 ---------------Speculative Decoding--------------- Acceptance rate (%): 32.45 Acceptance length: 3.60 Drafts: 80447 Draft tokens: 643576 Accepted tokens: 208834 Per-position acceptance (%): Position 0: 74.47 Position 1: 52.20 Position 2: 37.66 Position 3: 28.59 Position 4: 22.55 Position 5: 17.95 Position 6: 14.43 Position 7: 11.74 ================================================== ``` PR: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 84.28 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.71 Output token throughput (tok/s): 213.58 Peak output token throughput (tok/s): 63.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 408.29 ---------------Time to First Token---------------- Mean TTFT (ms): 112.45 Median TTFT (ms): 130.62 P99 TTFT (ms): 138.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.32 Median TPOT (ms): 4.42 P99 TPOT (ms): 5.97 ---------------Inter-token Latency---------------- Mean ITL (ms): 16.06 Median ITL (ms): 16.05 P99 ITL (ms): 16.51 ---------------Speculative Decoding--------------- Acceptance rate (%): 34.34 Acceptance length: 3.75 Drafts: 4827 Draft tokens: 38616 Accepted tokens: 13260 Per-position acceptance (%): Position 0: 75.08 Position 1: 55.13 Position 2: 41.06 Position 3: 31.32 Position 4: 24.72 Position 5: 19.23 Position 6: 15.66 Position 7: 12.51 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 204.94 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.68 Output token throughput (tok/s): 1405.30 Peak output token throughput (tok/s): 619.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2515.61 ---------------Time to First Token---------------- Mean TTFT (ms): 206.68 Median TTFT (ms): 190.26 P99 TTFT (ms): 410.25 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.67 Median TPOT (ms): 10.75 P99 TPOT (ms): 16.93 ---------------Inter-token Latency---------------- Mean ITL (ms): 38.09 Median ITL (ms): 22.18 P99 ITL (ms): 153.25 ---------------Speculative Decoding--------------- Acceptance rate (%): 32.45 Acceptance length: 3.60 Drafts: 80447 Draft tokens: 643576 Accepted tokens: 208834 Per-position acceptance (%): Position 0: 74.47 Position 1: 52.20 Position 2: 37.66 Position 3: 28.59 Position 4: 22.55 Position 5: 17.95 Position 6: 14.43 Position 7: 11.74 ================================================== ``` ## Accuracy Testing ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.6-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3533|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3290|± |0.0129| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3563|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3321|± |0.0130| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52078", "createdAt": "2026-08-13T01:02:20Z", "updatedAt": "2026-08-13T04:57:50Z", "timestamp": "2026-08-13T04:57:50Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "xyang16", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52079", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3] Add GEMM-RS for sequence parallelism", "text": "## Purpose Add GEMM-RS kernel for Blackwell, based on https://github.com/NVIDIA/cutlass/blob/dcf215a/examples/python/CuTeDSL/cute/blackwell/kernel/distributed/distributed_gemm_reduce_scatter_blackwell.py (`multimem.ld_reduce`) - Supports any value of M e.g. M=1023. However, only uses GEMM-RS when M>=128 since the kernel was not optimized for small/medium M. Only supports TP<=16, and requires all rank on the same NVLink domain. - The sharding behavior follows existing RS logic i.e. eac rank holds `ceil(M / world_size)`, with the exception of the last rank - Requires an opt-in flag `VLLM_KIMI_K3_GEMM_RS`, which enables GEMM-RS for O-proj and shared experts+dense MLP down-proj - Symmetric memory workspace (per-GPU): `max_num_batched_tokens x 7168 x 2 bytes` = 448 MiB for MNBT=32k. Confirmed in vLLM logs KV memory 39.99 GiB (before) -> 39.78 GiB (after) -> not much Initialization and runtime logic - Whether to initialize GEMM-RS: done in `maybe_init_gemm_rs()`, which also logs the reason if it fails. When `VLLM_KIMI_K3_GEMM_RS=0`, it doesn't do anything - At each layer's `__init__()`, we call `self.run_gemm_rs = get_gemm_rs().can_run(self.down_proj.weight)`. This is to further validate supported weight shapes and dtype - When the checks fail, we fallback to standard behavior. - In `forward()`, we check again with `should_run()`, which is the heuristics M>=128. The kernel supports any values of M, but right now the baseline is better for M<128 Though technically this can work with any SP in general, this PR only enables GEMM-RS for Kimi-K3. A future extension is to make this into GEMM-AR by adding `multimem.st` (all-gather) after `multimem.ld_reduce` (reduce-scatter). ## Microbenchmark `benchmarks/kernels/benchmark_kimi_k3_gemm_rs.py` in this PR. CUDA graph with rotating buffers. All benchmarks were done with GB300. ### TP4 Note: K=1536 is shared expert down-proj, K=3072 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 1536 | 48.13 | 46.27 | 38.74 | 1.242 | 1.195 | | 512 | 7168 | 1536 | 84.64 | 60.51 | 50.69 | 1.67 | 1.194 | | 2048 | 7168 | 1536 | 119.92 | 110.83 | 83.74 | 1.432 | 1.323 | | 8192 | 7168 | 1536 | 286.58 | 343.22 | 209.25 | 1.37 | 1.64 | | 32768 | 7168 | 1536 | 1044.66 | 1265.86 | 719.7 | 1.452 | 1.759 | | 128 | 7168 | 3072 | 52.37 | 48.27 | 43.38 | 1.207 | 1.113 | | 512 | 7168 | 3072 | 92.35 | 66 | 53.09 | 1.74 | 1.243 | | 2048 | 7168 | 3072 | 145.6 | 133.81 | 89.44 | 1.628 | 1.496 | | 8192 | 7168 | 3072 | 383.65 | 431.82 | 234.38 | 1.637 | 1.842 | | 32768 | 7168 | 3072 | 1402.05 | 1628.43 | 1001.49 | 1.4 | 1.626 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 1536 | 17.65 | 39.89 | 38.74 | | 512 | 7168 | 1536 | 21.52 | 50.94 | 50.69 | | 2048 | 7168 | 1536 | 36.54 | 90.37 | 83.74 | | 8192 | 7168 | 1536 | 104.14 | 191.68 | 209.25 | | 32768 | 7168 | 1536 | 401.31 | 652.77 | 719.7 | | 128 | 7168 | 3072 | 18.96 | 41.38 | 43.38 | | 512 | 7168 | 3072 | 28.21 | 50.48 | 53.09 | | 2048 | 7168 | 3072 | 59.25 | 90.56 | 89.44 | | 8192 | 7168 | 3072 | 195.92 | 196.91 | 234.38 | | 32768 | 7168 | 3072 | 763.71 | 650.18 | 1001.49 | ### TP8 Note: K=768 is shared expert down-proj, K=1536 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 768 | 49.47 | 44.93 | 40.08 | 1.234 | 1.121 | | 512 | 7168 | 768 | 70.29 | 56.58 | 47.46 | 1.481 | 1.192 | | 2048 | 7168 | 768 | 110.34 | 100.75 | 81.1 | 1.36 | 1.242 | | 8192 | 7168 | 768 | 264.43 | 300.98 | 202.27 | 1.307 | 1.488 | | 32768 | 7168 | 768 | 928.83 | 1109.26 | 697.5 | 1.332 | 1.59 | | 128 | 7168 | 1536 | 51.26 | 46.86 | 41.63 | 1.231 | 1.126 | | 512 | 7168 | 1536 | 72.22 | 62.64 | 49.82 | 1.45 | 1.257 | | 2048 | 7168 | 1536 | 121.71 | 109.84 | 82.08 | 1.483 | 1.338 | | 8192 | 7168 | 1536 | 306.91 | 347.92 | 209.04 | 1.468 | 1.664 | | 32768 | 7168 | 1536 | 1108.27 | 1289.57 | 706.02 | 1.57 | 1.827 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 768 | 17.3 | 39.2 | 40.08 | | 512 | 7168 | 768 | 18.61 | 49.44 | 47.46 | | 2048 | 7168 | 768 | 26.64 | 91.39 | 81.1 | | 8192 | 7168 | 768 | 60.18 | 209.14 | 202.27 | | 32768 | 7168 | 768 | 216.29 | 714.27 | 697.5 | | 128 | 7168 | 1536 | 17.22 | 39.66 | 41.63 | | 512 | 7168 | 1536 | 23.17 | 51.39 | 49.82 | | 2048 | 7168 | 1536 | 36.64 | 90.43 | 82.08 | | 8192 | 7168 | 1536 | 104.14 | 210.16 | 209.04 | | 32768 | 7168 | 1536 | 400.02 | 713.71 | 706.02 | ## E2E prefill-only benchmark All benchmarks were done with 8xGB300, TP8+EP+SP (DeepGEMM MegaMoE), `--max-num-batched-tokens 32768`, 8k input - 1 output requests. Baseline is 7aa248f Concurrency | Baseline TTFT (median) | GEMM-RS TTFT (median) | Baseline TPGS | GEMM-RS TPGS -- | -- | -- | -- | -- C1 | 312.81 ms | 298.44 ms (-4.59%) | 3,172.7 tok/GPU/s | 3,432.2 tok/GPU/s (+8.18%) C32 | 7,240.71 ms | 6,799.51 ms (-6.09%) | 4,500.5 tok/GPU/s | 4,794.0 tok/GPU/s (+6.52%) ## Test Plan Unit test (also added to distributed CI) ``` tests/kernels/test_kimi_k3_gemm_rs.py ``` E2E testing, TP8+EP+SP (DeepGEMM MegaMoE) - GSM8K: 96.82% - OCRBench: 88.40% ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/52079", "createdAt": "2026-08-13T01:15:53Z", "updatedAt": "2026-08-13T16:02:28Z", "timestamp": "2026-08-13T16:02:28Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "performance", "ci/build", "kimi", "k3" ], "author": "gau-nernst", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52080", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 AMD MLA: fuse the q-a and kv-a RMSNorms", "text": "## Purpose On the Kimi-K3 AMD MLA front-end, every token runs **two** separate RMSNorm launches — `q_a_layernorm(q_c)` then `kv_a_layernorm(kv_c)` — once per MLA layer. This PR collapses them into a single `fused_q_kv_rmsnorm` call (`models/common/ops/fused_qk_rmsnorm.py`), a **portable Triton** kernel already in the tree — no CUDA-only gate, and its one PDL branch is `tl.constexpr`-guarded on `current_platform.is_arch_support_pdl()`, which returns `False` on ROCm — so it runs on ROCm as-is with no new kernel. The front-end norm fires on every token across all MLA layers, so collapsing two launches into one removes a per-token kernel launch and its intermediate write. This is a **small, low-risk micro-optimization**: verified at the kernel level (see Test Result), throughput-neutral end-to-end on the workloads measured (the norms are a small, non-critical-path share of GPU time, and the saving is launch overhead that cudagraph capture already elides on the served decode path). It is offered as a clean reduction in launch count and intermediate writes with no accuracy risk, not as a headline speedup. ### How it works - **`MLAModules` gains an optional `q_kv_norm` callback** (`layers/mla.py`). When `None` (every other model/backend), the wrapper keeps its two separate norms — that path is byte-identical to before. - **The MLA wrapper's `forward`**, on the q-LoRA path (`q_lora_rank is not None`), defers `q_a_layernorm` and, when `q_kv_norm` is set, normalizes `q_c` and `kv_c` together at the kv-split point. It rebinds `q_c` to the normed tensor so both the q-projection and any later indexer see the normalized value — matching the separate-norm semantics exactly. - **Kimi-K3 AMD `KimiMLAAttention`** installs the callback, wrapping `fused_q_kv_rmsnorm` with the `q_a`/`kv_a` layernorm weights and `config.rms_norm_eps` (both norms share the same eps). Only wired on the q-LoRA path; the uncompressed path has no q norm and is unchanged. This mirrors the pattern established by #50664 (optional `MLAModules` callback for the K3 output gate; generic default, AMD installs the fused variant). The uncompressed path (no q norm) and every other model that leaves `q_kv_norm` unset are untouched. Relates to #50682 (ROCm/AMD Kimi-K3 gap and roadmap tracking → Performance Optimization → Mixed Attention / MLA). ### Not a duplicate No AMD adapter or open PR wires `fused_q_kv_rmsnorm` into the AMD K3 path (checked against #50682's PR list and open `rocm`+`k3` PRs). The nearby fusions are different: #50664 (MLA output gate), #50637 (AttnRes + RMSNorm), #50634 (KDA decode gate), #41095 (AITER dual-RMSNorm for DeepSeek MLA, not K3). #51772 (chunked-context K/V packing) does not touch `common/ops` or this norm. ### Files changed - `vllm/model_executor/layers/mla.py` — optional `q_kv_norm` field on `MLAModules`; wrapper `forward` uses it on the q-LoRA path. No behavior change when unset. - `vllm/models/kimi_k3/amd/linear.py` — `KimiMLAAttention` builds and passes the fused callback. - `tests/models/kimi_k3/test_amd_mla_qkv_norm.py` — parity + empty-batch test. ## Test Plan Unit — the fused op must equal two independent RMSNorm modules (ROCm GPU): ```bash .venv/bin/python -m pytest -q tests/models/kimi_k3/test_amd_mla_qkv_norm.py ``` Covers token counts `{1, 7, 64, 512, 4096}` (decode- through prefill-sized) plus an empty batch. Accuracy — this changes an activation path, so a full eval is required: ```bash # serve K3 on the AMD box (gfx942), then: make lm-eval-gsm8k # gsm8k parity vs main (must match within tolerance) ``` End-to-end prefill perf (long input) — baseline `main` vs this PR, same seeds: ```bash make bench- # 8k/1k profile; compare TTFT (prefill) before/after ``` ## Test Result Environment: 8× MI325X (`gfx942`), `moonshotai/Kimi-K3`, `int4_per_group_32`, TP=8, EP off. This is a **small but fair micro-optimization**: the kernel-level effect is real and verified, and end-to-end it is throughput-neutral (within run-to-run noise) on this prefill-heavy workload — it neither helps nor harms serving numbers, and it carries no accuracy risk. **Kernel evidence (torch profiler traces)** — the fused kernel does exactly what it should. Each `fused_q_kv_rmsnorm` launch replaces the two separate `q_a` / `kv_a` RMSNorm launches, and nothing else in the trace changes: | | candidate (fused) | baseline (two norms) | |---|---:|---:| | `fused_q_kv_rmsnorm` launches | 384 | 0 | | plain q-a + kv-a RMSNorm launches | 0 | 768 | | front-end norm GPU time | 1.61 ms | 3.16 ms | The plain-RMSNorm count drops by exactly `768 = 2 × 384` — every fused launch removes the two norms it replaces — for **~1.55 ms less GPU time and 384 fewer kernel launches** across the profiling window. All other kernels are unchanged. **End-to-end serving** (`vllm bench serve`, 8024/1024, conc 16). Baseline = `main` (two norms); Candidate = this PR (fused): | Metric | Baseline | Candidate | Δ | |---|---:|---:|---:| | Mean TTFT (ms) | 4615.0 | 4617.2 | +0.0% | | Median TTFT (ms) | 2559.4 | 2559.3 | ~0% | | Mean TPOT (ms) | 52.70 | 52.49 | −0.4% | | Output tok/s | 279.35 | 280.34 | +0.4% | | Total tok/s | 2468.3 | 2477.1 | +0.4% | All deltas are within run-to-run noise (≤0.4%). Expected: the front-end norms are ~4.4% of GPU time and are **not on the critical path** (the workload is bound by the projection GEMMs and the SiTU activation), and the saving here is kernel-launch overhead — which full cudagraph capture already elides on the served decode path. So the fusion is a clean win in eager mode and at the kernel level, but does not move aggregate serving throughput at this operating point. It is included as a correct, low-risk reduction in launch count and intermediate writes, not as a headline speedup. **Correctness** - Unit (`test_amd_mla_qkv_norm.py`): _TODO PASS/FAIL_ (`N passed` — the fused op equals two independent RMSNorm modules across token counts `{1, 7, 64, 512, 4096}`). - gsm8k parity vs `main` (1319 samples): _TODO — baseline acc vs candidate acc; the fused op is fp32-accurate, so parity is expected._ ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/52080", "createdAt": "2026-08-13T01:43:05Z", "updatedAt": "2026-08-13T10:23:02Z", "timestamp": "2026-08-13T10:23:02Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "mpashkovskii", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52081", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"Avoid long-blocking H2D copies in ViT\" (#51841)", "text": "Reverts the changes from #51841 (\"Avoid long-blocking H2D copies in ViT\"). ## Why `Arm CPU Test` has failed on every build since #51841 merged (nightly [#83608](https://buildkite.com/vllm/ci/builds/83608), plus per-commit postmerge builds 83603 and 83607 on the same commit). It passed on build 83539, the last run before the merge. The failure is at engine startup, during `profile_run`: ``` File \"vllm/model_executor/models/qwen3_vl.py\", line 709, in rot_pos_emb pinned = torch.empty( RuntimeError: pin_memory=True requires a CUDA or other accelerator backend; no pinned memory allocator is available on this system. ``` `Qwen3_VisionTransformer.rot_pos_emb` now allocates a staging buffer with `pin_memory=True` unconditionally. On CPU-only platforms there is no pinned-memory allocator, so `torch.empty(..., pin_memory=True)` raises and `EngineCore` init dies: ``` RuntimeError: Worker failed with error 'pin_memory=True requires a CUDA or other accelerator backend; no pinned memory allocator is available on this system.' ``` ## Alternative to reverting The H2D optimization is worth keeping. A narrower fix would be to gate the pinned allocation on accelerator availability and fall back to the plain `torch.cat` path otherwise, e.g. `pin_memory=current_platform.is_pin_memory_available()`. The same guard should be considered for the per-row `mrope_positions` copy in `gpu_model_runner.py`, which relies on the source being pinned to be genuinely async. Reverting here because it restores CPU platforms immediately with no correctness or accuracy loss; a re-land with the platform guard is preferable if the author can turn it around quickly. Auto-generated by CI failure analyzer.", "url": "https://github.com/vllm-project/vllm/pull/52081", "createdAt": "2026-08-13T02:06:56Z", "updatedAt": "2026-08-13T02:36:13Z", "timestamp": "2026-08-13T02:36:13Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "qwen" ], "author": "vllm-agent", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Fix memory pinning in Qwen3_VisionTransformer for non-gpu", "text": "Fix for CI failure introduced by https://github.com/vllm-project/vllm/pull/51841", "url": "https://github.com/vllm-project/vllm/pull/52082", "createdAt": "2026-08-13T02:35:39Z", "updatedAt": "2026-08-13T15:46:52Z", "timestamp": "2026-08-13T15:46:52Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready", "needs-rebase", "qwen" ], "author": "njhill", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52084", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][DSV4] Optimize sparse top-k metadata kernels for higher prefill throughput", "text": "## Purpose Optimize sparse top-k metadata kernels for higher prefill throughput ## Test ``` vllm bench serve \\ --backend vllm \\ --base-url http://localhost:8000 \\ --model deepseek-ai/DeepSeek-V4-Flash-0731 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 64 \\ --num-prompts 128 \\ --num-warmups 8 \\ --request-rate inf \\ --ignore-eos \\ --temperature 0 \\ --seed \"$seed\" \\ --save-result ``` ``` vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --enable-expert-parallel \\ --tensor-parallel-size 8 \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 \\ --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --no-enable-prefix-caching \\ --max-num-batched-tokens 16384 ``` ## Test Plan | Tokens | 128 Workers | 256 Workers | Relative Change | | ---: | ---: | ---: | ---: | | 1,024 | 9.75 us | 8.27 us | **15.2% faster** | | 4,096 | 22.40 us | 15.52 us | **30.7% faster** | | 16,384 | 74.76 us | 47.14 us | **36.9% faster** | ### Paired A/B/A Serving Results | Run | Workers | Mean Output Throughput | Mean TPOT | | --- | ---: | ---: | ---: | | A1 | 256 | 638.67 tokens/s | 102.26 ms | | B | 128 | 629.61 tokens/s | 104.80 ms | | A2 | 256 | 639.44 tokens/s | 101.88 ms | The second optimized run improved output throughput by **1.56%** and reduced mean TPOT by **2.79%** relative to the paired 128-worker baseline. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52084", "createdAt": "2026-08-13T02:57:54Z", "updatedAt": "2026-08-13T03:08:55Z", "timestamp": "2026-08-13T03:08:55Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "chaunceyjiang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52085", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Warn when DeepSeek-V4 silently defaults reasoning_effort to \"high\"", "text": "## Purpose Fixes #52083. #50580 changed the DeepSeek-V4 chat encoding (both the Python tokenizer wrapper `vllm/tokenizers/deepseek_v4.py` and the Rust renderer `rust/src/chat/src/renderer/deepseek_v4/encoding.rs`) so that any request which ends up with thinking mode enabled but no explicit `reasoning_effort` now defaults to `reasoning_effort=\"high\"`. That happens both when a request omits `thinking`/`enable_thinking` **and** `reasoning_effort` entirely (the implicit default), and when a caller explicitly passes `thinking=True`/`enable_thinking=True` without also setting `reasoning_effort` (thinking was requested, but the effort default still silently changed). Either way it silently inserts a reasoning-effort prompt prefix that was previously omitted by default. This is a shared code path used by every DeepSeek-V4 checkpoint (Pro and Flash-0731 alike) — there's no per-checkpoint differentiation in the encoder/renderer. The change was an intentional, tested alignment with the DeepSeek-V4-Flash-0731 hosted API contract, but it silently changed the default request behavior for other DeepSeek-V4 deployments (e.g. DSV4-Pro) that relied on the old \"no reasoning-effort blurb unless requested\" default, causing a spike in truncations/reasoning length as reported in the issue. Given the encoder has no way to know which DeepSeek-V4 checkpoint variant is being served, reverting or branching the default behavior isn't a safe minimal fix — it could just as easily reintroduce the original 0731 bug #50580 fixed for other deployments. Per the issue's own suggestion (\"At the very least, we should issue a warning that the default behaviour has been changed\"), this PR adds a one-time warning in both encoders when this implicit default kicks in, telling users how to restore the previous behavior explicitly with `reasoning_effort=\"low\"`. The warning is keyed off the same `thinking_enabled`/resolved `ThinkingMode::Thinking` condition that actually selects `reasoning_effort=\"high\"`, not off \"thinking/enable_thinking were both omitted\" — an earlier version of this PR used the narrower condition and missed the explicit-`thinking=True`-without-`reasoning_effort` path, which hits the exact same silent default. ## Test Plan - Added `test_deepseek_v4_warns_when_defaulting_reasoning_effort` and `test_deepseek_v4_warns_when_thinking_explicit_without_reasoning_effort` (parametrized over `thinking=True` and `enable_thinking=True`) to assert the warning fires for both trigger paths, and `test_deepseek_v4_does_not_warn_when_explicit` (parametrized over `reasoning_effort=\"high\"`, `thinking=True` with an explicit `reasoning_effort=\"low\"`, and `thinking=False`/`enable_thinking=False`) to assert it stays silent when `reasoning_effort` is set or thinking is off. All use the existing `disable_log_dedup` fixture to observe `logger.warning_once`. - Verified the new `test_deepseek_v4_warns_when_thinking_explicit_without_reasoning_effort` test fails without the source fix: `git checkout HEAD -- vllm/tokenizers/deepseek_v4.py` (reverting to the narrower `thinking_unset`-gated condition), ran the test (failed as expected with `assert False` for both parametrizations), then restored the `thinking_enabled`-gated fix. - The Rust renderer change mirrors the same fix by checking the resolved `thinking_mode == ThinkingMode::Thinking` instead of `enable_thinking.is_none()`, gated by the same `std::sync::Once`; the existing 14 tests in `rust/src/chat/src/renderer/deepseek_v4/tests.rs` (including `omitted_thinking_and_effort_default_to_high`) continue to pass unchanged, confirming rendered output is untouched. No dedicated Rust test was added for the warning itself since this crate has no existing pattern for capturing `tracing` output in tests, and adding that infra seemed out of proportion for a log-only change. ### Commands run ``` .venv/bin/python -m pytest tests/tokenizers_/test_deepseek_v4.py -v # 33 passed cargo test -p vllm-chat renderer::deepseek_v4 # 14 passed pre-commit run --files vllm/tokenizers/deepseek_v4.py tests/tokenizers_/test_deepseek_v4.py rust/src/chat/src/renderer/deepseek_v4/encoding.rs # ruff-check, ruff-format, mypy, rust-cargo-fmt, etc. all passed ``` ## Duplicate Check Searched for open PRs referencing #52083 or similar titles — none found. #50684 (\"Fix DeepSeek-V4 reasoning_effort \\\"high\\\" tier and message-level tools preservation\") addresses a related but distinct bug (the \"high\" tier text was mislabeled/missing, and message-level `tools` were dropped for `role == \"system\"`) in the Python encoder only; it does not touch the default-value regression described in #52083, and doesn't touch the Rust renderer. ## AI Assistance AI assistance (Claude) was used to bisect the regression to #50580, compare the Python tokenizer wrapper and Rust renderer default-value logic, implement the warning in both, and write/run the tests above. All changes were reviewed and the listed commands were run and their output verified by the submitting human.", "url": "https://github.com/vllm-project/vllm/pull/52085", "createdAt": "2026-08-13T03:16:29Z", "updatedAt": "2026-08-13T13:15:58Z", "timestamp": "2026-08-13T13:15:58Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "deepseek", "rust" ], "author": "chelsealong", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52086", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Gate cooperative top-k on CUDA 13", "text": "## Fixes - Build cooperative top-k kernels only on CUDA 13 or newer. - Let source builds skip bundled FA2/FA3 extensions while retaining Python-only FA4. - Treat an FA4-only installation as a valid FlashAttention installation. ## Verification - The patched source builds as vLLM 0.23.1rc1.dev1407+gdab991476 on CUDA 12.8. - The baked SkyRL image imports FlashAttention successfully.", "url": "https://github.com/vllm-project/vllm/pull/52086", "createdAt": "2026-08-13T03:17:05Z", "updatedAt": "2026-08-13T13:46:42Z", "timestamp": "2026-08-13T13:46:42Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "ci/build", "nvidia" ], "author": "j316chuck", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52087", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Preserve constant effective K schedule semantics", "text": "## Purpose `num_speculative_tokens_per_batch_size` is currently treated as runtime-variable whenever the schedule is present. Some schedules are syntactically non-uniform but resolve to one positive effective K after the configured runtime maximum is applied: ```text num_speculative_tokens = 3 schedule = [(1, 8, 5), (9, 16, 4)] effective K values = {min(5, 3), min(4, 3)} = {3} ``` Such a schedule cannot change the target verification width. The current behavior has three effects: 1. V1 downgrades `FULL_AND_PIECEWISE` to `PIECEWISE`. 2. Data parallelism disables the schedule and falls back to the raw maximum K, which changes the configured effective K. 3. Prefix-cache-hit requests entering their first decode step are not padded to the effective verification width. This PR: - classifies a schedule as constant only when every entry resolves to the same positive K after runtime clamping; - normalizes the runtime maximum to that effective K so the Scheduler, GPU Runner, metrics, and CUDA Graph descriptor use one width; - applies the V1 graph downgrade and DP fallback only when the effective verification width can vary; and - pads prefix-cache-hit requests with the constant effective K. Malformed schedules, genuinely variable schedules, and K=0 stay on the conservative path. Existing schedule validation remains in place. No CUDA kernel, drafter, verifier, or rejection-sampling rule is changed. ## Why this is not duplicate work Before implementation I checked the issue and searched open PRs by issue number and by the areas `constant effective K`, `speculative schedule`, `full cudagraph capture sizes`, `separate decode capture sizes`, and related terms. - #48944 adds a context-length axis to K selection; this change only classifies schedules that collapse to one runtime width. - #50488 ensures wide uniform-decode batches are represented in the capture grid; this change addresses an earlier configuration/shape classification boundary. - #52000 dispatches more uniform-decode batches to padded FULL graphs; it does not preserve a constant schedule's effective K under data parallelism. - #49652 handles fixed-width autoregressive draft-decode capture; it does not classify the target verification schedule. - #52070 covers always-K=0 behavior. K=0 is deliberately excluded here. ## Test plan ### Focused behavior tests ```bash HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \\ /vllm-workspace/.venv/bin/python -m pytest -q \\ tests/v1/spec_decode/test_dynamic_sd.py ``` ```text 26 passed, 15 warnings in 7.59s ``` Coverage includes: - no schedule, one entry, same-K entries, values clamped to one K, variable K, K=0, and malformed entries; - preserving effective K=2 under DP when the raw maximum is K=3; - prefix-cache-hit first-decode padding with the effective K; and - direct Dispatcher assertions that the normalized runtime shape is uniform and resolves to `CUDAGraphMode.FULL`. ### Repository hooks ```bash uv tool run pre-commit run --files \\ tests/v1/spec_decode/test_dynamic_sd.py \\ vllm/config/speculative.py \\ vllm/config/vllm.py \\ vllm/v1/core/sched/scheduler.py ``` All applicable hooks passed, including Ruff, mypy, typos, SPDX, forbidden-import, and configuration checks. `git diff --check` passed. ## GPU validation Environment: - NVIDIA RTX PRO 6000 Blackwell Server Edition; - Qwen3-8B BF16, V1 model runner; - N-gram speculation, raw maximum K=3, effective schedule K=2; - random input/output length 128/128. ### Single-GPU paired matrix Each concurrency level used five paired repetitions with alternating execution order. | Concurrency | Base PIECEWISE | Patched FULL | Paired change | 95% CI | |---:|---:|---:|---:|---:| | 1 | 94.12 | 92.45 | -1.78% | [-1.98%, -1.57%] | | 2 | 174.05 | 169.16 | -2.81% | [-2.91%, -2.71%] | | 4 | 344.90 | 367.47 | +6.55% | [+6.20%, +6.90%] | | 8 | 658.70 | 638.82 | -3.02% | [-3.31%, -2.73%] | A five-run isolation control with the patched build forced to PIECEWISE was approximately neutral at concurrency 1 (94.05 vs 94.12 output tok/s). This indicates that effective-K normalization itself is not the main regression source; FULL-versus-PIECEWISE performance crosses over by workload shape on this setup. These results do not support a universal performance claim. They show that shape safety and performance admission are separate decisions. A follow-up should select FULL only for empirically winning token/batch buckets, with PIECEWISE as the fail-closed default. ### Two-GPU data-parallel stress test Two DP ranks were tested at temperature 0 for three repetitions at concurrency 7 and 16: - 2304 total requests completed, 0 failed; - base logs show the schedule disabled and a fallback to K=3; - patched logs show both ranks initialized with effective K=2 and retained the schedule. The throughput numbers are intentionally not presented as a same-K speedup because the base and patched runs execute different K. This test validates configuration semantics, rank consistency, and absence of a DP deadlock. ## Current assessment The DP behavior is a concrete configuration-semantics fix: a schedule whose effective K is always 2 should not silently become K=3 merely because DP is enabled. The effective-width propagation and direct Dispatcher test also fix a gap where graphs could be captured without the runtime descriptor selecting the expected replay path. The FULL graph policy needs further reviewer discussion or a separate performance-admission change because the measured crossover is non-monotonic. This PR remains a draft while that scope is resolved. ## AI assistance disclosure OpenAI Codex assisted with prior-art checks, implementation and test drafting, GPU benchmark execution, and documentation. The human submitter must review every changed line, rerun the relevant tests, and be able to defend the change end-to-end before requesting merge.", "url": "https://github.com/vllm-project/vllm/pull/52087", "createdAt": "2026-08-13T03:17:52Z", "updatedAt": "2026-08-13T08:51:35Z", "timestamp": "2026-08-13T08:51:35Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "speculative-decoding", "nvidia" ], "author": "feednetinfra", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52090", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix: Deprecated pylance warnings on using AsyncIterator with asynccontextmanager", "text": "## Purpose Fix the deprecated pylance warnings ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52090", "createdAt": "2026-08-13T03:33:09Z", "updatedAt": "2026-08-13T03:39:10Z", "timestamp": "2026-08-13T03:39:10Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "frontend", "verified" ], "author": "elizabetht", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Auto-ping Cohere on related issues", "text": "## Purpose As discussed offline ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52091", "createdAt": "2026-08-13T03:37:59Z", "updatedAt": "2026-08-13T16:40:39Z", "timestamp": "2026-08-13T16:40:39Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "ci/build" ], "author": "DarkLight1337", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52092", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True", "text": "## Summary - Build and install a pre-built `triton-cpu` wheel in the CPU build/test images instead of `pip install`-ing it from source inside CI, unblocking the Triton topk-topp kernel to run as a normal (non-soft-fail) test. - Move the topk-topp Triton kernel test out of the soft-fail `CPU-ModelRunnerV2 Tests` suite into `CPU-Kernel Tests`, and the linear-attention chunked-prefill correctness test into `CPU-Language Generation and Pooling Model Tests`, then remove the now-empty `CPU-ModelRunnerV2 Tests` suite. - Guard the hardcoded `pin_memory=True` calls in `vllm/v1/worker/gpu/` (shared by the CPU V2 runner) and the multimodal model code it drives behind the existing `PIN_MEMORY` flag, since CUDA pinned memory isn't available on CPU. Also stub `torch.cuda.stream` in the CPU shim and log which model runner is active. ## Test plan - [x] `python3 -c \"import yaml; yaml.safe_load(open('.buildkite/hardware_tests/cpu.yaml'))\"` — confirms the reorganized pipeline YAML is still valid. - [ ] CPU CI (`CPU-Kernel Tests`, `CPU-Language Generation and Pooling Model Tests`, and CPU image build) — not run locally in this session; relying on CI to validate the Dockerfile and runtime changes on real Intel CPU hardware.", "url": "https://github.com/vllm-project/vllm/pull/52092", "createdAt": "2026-08-13T03:55:43Z", "updatedAt": "2026-08-13T06:40:57Z", "timestamp": "2026-08-13T06:40:57Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "qwen", "cpu", "mrv2" ], "author": "bigPYJ1151", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52093", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Perf] Narrow DeepSeek V4 eager CUDA graph region\" (#51430)", "text": "## Summary - Revert [#51430](https://github.com/vllm-project/vllm/pull/51430) and restore the wider DeepSeek-V4 eager attention region. - Preserve the later ROCm `_fused_wqa_wkv_gemm` override point while resolving the revert against current main. - This is the prerequisite for safely restoring MRV1 piecewise graphs in the stacked follow-up. ## Why On gfx950 TP8, the narrowed eager region corrupts output under the MRV1 piecewise graph path. MRV2 avoids that corruption but regresses 8k/1k concurrency-4 throughput. Restoring the wider eager region makes the legacy path correct and recovers the attention performance needed by the follow-up runner revert. This revisits closed PR [#51750](https://github.com/vllm-project/vllm/pull/51750) with new ROCm correctness and performance measurements. No open PR currently reverts [#51430](https://github.com/vllm-project/vllm/pull/51430). ## Test plan Configuration: DeepSeek-V4-Pro, gfx950, TP=8, DP=1, prefix caching disabled, `max_num_batched_tokens=8192`, AITER MoE, FP8 KV cache, `FULL_AND_PIECEWISE` graphs. Combined with the stacked [#51768](https://github.com/vllm-project/vllm/pull/51768) revert: - Correctness smoke: `1 + 1` returned `2`. - InferenceX 8k/1k, concurrency 4: 40/40 successful requests. - Output throughput: **185.66 tok/s**. - Mean TPOT: **20.53 ms**. - Median ITL: **19.47 ms**. - Mean TTFT: **466.34 ms**. - Historical reference: 186.99 tok/s, 20.36 ms TPOT, 19.23 ms median ITL. Pre-commit hooks run during commit passed, including Ruff, formatting, mypy, SPDX, and configuration validation. ## Contribution notes - Duplicate check: [#51750](https://github.com/vllm-project/vllm/pull/51750) is closed; no open PR reverting [#51430](https://github.com/vllm-project/vllm/pull/51430) was found. - AI assistance was used for regression isolation, implementation, testing, and PR preparation. - The human submitter must review every changed line and independently validate the results before merge.", "url": "https://github.com/vllm-project/vllm/pull/52093", "createdAt": "2026-08-13T04:06:16Z", "updatedAt": "2026-08-13T04:15:31Z", "timestamp": "2026-08-13T04:15:31Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "deepseek", "nvidia" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52094", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Bugfix] Guard DeepSeek V4 MRV1 piecewise CUDA graphs\" (#51768)", "text": "## Summary - Restore Model Runner V1 as the DeepSeek-V4 default. - Remove the MRV1 piecewise-graph rejection added by [#51768](https://github.com/vllm-project/vllm/pull/51768). - Stack this change on [#52093](https://github.com/vllm-project/vllm/pull/52093), which first restores the wider eager attention region required for correctness. ## Dependency Stacked on [#52093](https://github.com/vllm-project/vllm/pull/52093). Do not merge this PR first: reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) alone would re-enable the known-broken MRV1 piecewise path while [#51430](https://github.com/vllm-project/vllm/pull/51430) remains active. Until [#52093](https://github.com/vllm-project/vllm/pull/52093) merges, this PR intentionally contains both revert commits; afterward its diff reduces to the config/test revert. ## Why On gfx950 TP8, MRV2 preserved correctness but regressed steady decode performance. With the prerequisite eager-region revert, MRV1 piecewise execution is correct and restores long-context throughput. No open PR reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) was found. ## Test plan Configuration: DeepSeek-V4-Pro, gfx950, TP=8, DP=1, prefix caching disabled, `max_num_batched_tokens=8192`, AITER MoE, FP8 KV cache, `FULL_AND_PIECEWISE` graphs. | Configuration | Output throughput | Mean TPOT | Median ITL | | --- | ---: | ---: | ---: | | MRV2 before reverts | 154.81 tok/s | 24.70 ms | 23.65 ms | | MRV1 + [#52093](https://github.com/vllm-project/vllm/pull/52093) | **185.66 tok/s** | **20.53 ms** | **19.47 ms** | | Historical reference | 186.99 tok/s | 20.36 ms | 19.23 ms | Additional results: - 40/40 successful InferenceX 8k/1k concurrency-4 requests. - Mean TTFT: 466.34 ms. - Correctness smoke: `1 + 1` returned `2`. - Two selective ROCm runs measured 185.16 and 184.94 tok/s, corroborating the rollback result. Pre-commit hooks run during commit passed, including Ruff, formatting, mypy, SPDX, and configuration validation. ## Contribution notes - Duplicate check: no open PR reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) was found. - AI assistance was used for regression isolation, implementation, testing, and PR preparation. - The human submitter must review every changed line and independently validate the results before merge.", "url": "https://github.com/vllm-project/vllm/pull/52094", "createdAt": "2026-08-13T04:07:21Z", "updatedAt": "2026-08-13T04:08:00Z", "timestamp": "2026-08-13T04:08:00Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "deepseek", "nvidia" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52096", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Reduce AITER MLA FP8 BMM warmup sizes", "text": "## Purpose Reduce ROCm AITER MLA FP8 BMM startup work by precompiling only the real decode sizes that are covered by the configured CUDA graph limit. The previous path precompiled every FP8 BMM size from `1..1024` for every MLA layer. This patch keeps a contiguous warmup range, because `MLAAttention.forward_impl` strips CUDA-graph padding before this BMM path, but caps that range at the graph-covered decode size when CUDA graphs are enabled. Larger batches can still JIT later, but they are already outside the captured fast path. The AITER Triton FP8 BMM kernel specializes on the query-row count. Warming sizes that vLLM will not use for captured decode increases model loading time before the server can handle requests. For the tested DeepSeek-V2-Lite ROCm config, `max_cudagraph_capture_size` is `512`, so this changes each MLA layer's FP8 BMM warmup from `1..1024` to `1..512`. Small odd actual decode sizes, such as batch 7, remain covered because the warmup range is still contiguous. This PR was developed with AI assistance. I reviewed the changed code paths, ran the checks below, and verified the ROCm smoke tests on MI300X. ## Duplicate-work check No linked issue exists for this change. I checked for open duplicate work with: - `gh pr list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` - `gh pr list --repo vllm-project/vllm --state open --search \"rocm aiter mla fp8 bmm\"` - `gh pr list --repo vllm-project/vllm --state open --search \"triton_fp8_bmm mla warmup\"` - `gh issue list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` The only close related open PR I found is #52063, which is also mine. It is related but not a duplicate: #52063 changes the runtime K-side MLA projection path, while this PR changes startup precompile bounds for the AITER FP8 BMM path that remains. If #52063 lands first, this PR may need a small rebase in the same `process_weights_after_loading` area. ## Test Plan Local checks: - `pre-commit run --files vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `ruff check vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `python -m py_compile vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `git diff --check` - `pytest -q tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` RunPod MI300X smoke tests with `deepseek-ai/DeepSeek-V2-Lite`, ROCm AITER MLA, `max_model_len=1024`, `input_len=128`, `output_len=1`: - batch 1 latency smoke - batch 7 latency smoke, to verify an odd real decode size under the graph limit remains covered ## Test Result Local checks: - `pre-commit run --files ...`: passed - `ruff check ...`: passed - `python -m py_compile ...`: passed - `git diff --check`: passed - `PYTHONPATH=. uv run --no-sync --python 3.12 --with-requirements requirements/common.txt --with pytest --with tblib pytest -q tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py`: passed (`9 passed, 14 warnings`) RunPod MI300X smoke results: - batch 1: FP8 BMM precompile changed to `0/512`; model load `58.38s`; generation completed; avg latency `13.84 ms` - batch 7: FP8 BMM precompile changed to `0/512`; model load `57.30s`; generation completed; avg latency `63.4 ms` No JIT-monitor warnings appeared after inference monitor activation in either smoke test.", "url": "https://github.com/vllm-project/vllm/pull/52096", "createdAt": "2026-08-13T04:31:08Z", "updatedAt": "2026-08-13T05:44:49Z", "timestamp": "2026-08-13T05:44:49Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "rohash123", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52097", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Add suffix_gpu drafter for Async Scheduling", "text": "## Purpose Suffix decoding (`method=\"suffix\"`, #25784) has the best acceptance lengths among model-free drafters on repetitive/agentic traffic, but it is CPU-side and rejected by the async-scheduling whitelist, so it gives up CPU/GPU overlap exactly where spec decode helps most (high-concurrency TPOT). This PR adds `method=\"suffix_gpu\"`: a GPU-resident suffix drafter following the same device-state drafter contract as `ngram_gpu` (#29184) — state on device, drafts from the previous step's GPU sampled ids, no host sync on the draft path — which makes suffix decoding compatible with async scheduling. RFC: #51788. In-tree changes (~590 lines outside tests, ~300 lines of tests): - `config/speculative.py`: `suffix_gpu` method; reuses `suffix_decoding_*` knobs, adds `suffix_gpu_*` GPU knobs; `use_gpu_state_drafter()` groups ngram_gpu + suffix_gpu; async-scheduling whitelist admits `suffix_gpu`. - `v1/spec_decode/suffix_proposer_gpu.py`: `SuffixProposerGPU` wrapper — borrows `update_token_ids_ngram` from `NgramProposerGPU`, same `propose()` signature and valid-count async D2H; adds cross-request global-index ingest (side stream, off the critical path) and a bucketized CUDA graph for the draft chain, pre-captured at warmup. - `v1/worker/gpu_model_runner.py`: `ngram_gpu`-only gates generalized to `use_gpu_state_drafter()` (`is_ngram_gpu` local renamed to `is_gpu_state_drafter`); suffix-specific ingest hooks and draft-graph pre-capture. - `tests/v1/spec_decode/test_suffix_gpu.py`: proposer unit tests. - `tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py`: `test_suffix_gpu_with_async_scheduling` (GSM8K under async, mirroring the ngram_gpu async test) and `test_suffix_gpu_acceptance` (acceptance improves as the global index warms, same pattern as the suffix CPU acceptance test; pins `suffix_gpu_ingest_chunk=1`, `num_backoff=8`, `max_occurrences=128`). Both acceptance tests — including the pre-existing CPU one — now pin `enable_prefix_caching=False` and the V1 model runner: warm rounds replay identical prompts, and prefix-cache hits change decode batch composition enough to depress the measured rate below the 0.80 floor for the CPU method too - Metrics accounting for GPU-state drafters (dedicated merge): `update_scheduler_for_invalid_drafts` returns per-request padded-slot trim counts, the worker reports them via `ModelRunnerOutput.num_invalid_spec_tokens`, and `make_spec_decoding_stats` subtracts them (fully-padded drafts are skipped instead of being recorded as zero-token draft events). Without this, padded-k scheduling deflates the reported acceptance rate for ngram_gpu and suffix_gpu alike and makes GPU and CPU drafter stats incomparable. The drafter core lives in a standalone package (https://github.com/zip95297/SuffixGPU), lazily imported — same pattern as `suffix`'s `arctic-inference` dependency; vLLM works without it installed. All results in this PR were produced with suffix-gpu 0.1.1 (PyPI, tag `v0.1.1`; the runs used the equivalent package code at repo commit `1ac9238`, installed editable — the only delta is constructor defaults, which the vLLM config passes explicitly). Not duplicating existing work (searched 2026-08-12): no open PR wires suffix decoding into the async path — open-PR search for \"suffix\" in titles returns docs/loader/KV items only, \"suffix_gpu\" returns none; #25784 is CPU+sync-only, #29184 is ngram-only. The metrics accounting part overlaps in mechanism with open PR #44056 (no activity since May) and is required here for suffix_gpu's acceptance test and honest benchmark numbers; this PR uses method-neutral naming (`num_invalid_spec_tokens`), also fixes ngram_gpu, and I am happy to rebase onto #44056 if it lands first or split this part out if maintainers prefer. #50518 covers the scheduler-side view for uniform spec batches and is unaffected. ## Test Plan - Unit tests: `pytest tests/v1/spec_decode/test_suffix_gpu.py -v` (skips without CUDA / suffix_gpu package -> CI-safe) - Acceptance e2e (in-tree pattern; suffix_gpu + async and the pre-existing CPU suffix test as a regression check): `pytest tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance -v` - Greedy consistency (spec decode must not change greedy outputs): `VLLM_BATCH_INVARIANT=1 python benchmarks/greedy_consistency.py 16` async no-spec vs async suffix_gpu (+ suffix_gpu eager / ngram_gpu / sync suffix CPU legs), 16 prompts x 256 tokens, token-identical required. Batch-invariant kernels + enforce_eager are required for a bitwise gate: spec decode changes batch composition, and batch-dependent FP reduction order (FlashAttention split heuristics) flips near-tie greedy argmax even between two no-spec runs. - Serving benchmark mirroring #25784's setup (Spec-Bench, TPOT + drafted/ accepted totals, k x concurrency grid) and #29184's async-vs-sync comparison: Llama-3.1-8B-Instruct bf16 TP=1 on 1x L20 48GB, k in {5,16}, concurrency 1/4/16/64/128/192/256, prefix caching off, max_num_batched_tokens=8192 and max_num_seqs=320 for every variant (spec decode schedules (k+1) padded token slots per running request before invalid ones are trimmed, so the serve-context defaults silently cap effective concurrency at the top grid points). Script: benchmarks/bench_pr_matrix.sh (attached). - Drafter-level correctness/latency/VRAM vs the arctic CPU suffix tree (deterministic Spec-Bench replay, 221 tests): equivalence on unambiguous corpora; per-step drafting cost crosses over at B~64-128 — beyond that, CUDA-graph mode is 2.1-2.7x faster than the CPU tree's sequential walk (B=256: 1.85 vs 4.98 ms), which is what makes the drafter viable in the high-concurrency regime async scheduling targets. Full tables: https://github.com/zip95297/SuffixGPU/blob/main/RESULTS.md ## Test Result ### Unit tests ``` $ pytest tests/v1/spec_decode/test_suffix_gpu.py -v 6 passed in 5.93s (L20, torch 2.13.0+cu130, branch commit f646af65c) test_propose_drafts_repetition[False/True], test_graph_and_eager_agree, test_capture_draft_graph_at_warmup, test_capture_draft_graph_warms_up_without_graph, test_ingest_and_cross_request_draft ``` ### E2E GSM8K (in-tree pattern, mirrors ngram_gpu's async test) ``` $ pytest \"tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py::test_suffix_gpu_with_async_scheduling\" -v 1 passed — GSM8K accuracy: 0.762 (threshold 0.70) Llama-3.1-8B-Instruct, suffix_gpu k=16, async_scheduling=True, 1319 questions 5-shot ``` ### Acceptance e2e (in-tree pattern, mirrors the CPU suffix acceptance test) ``` $ pytest tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance -v 2 passed test_suffix_gpu_acceptance (suffix_gpu k=16, async scheduling) test_suffix_decoding_acceptance (CPU suffix, unchanged-behavior check) ``` suffix_gpu warm-up on the shared workload (10 rounds x 100 prompts, per-round rate/AL from an identical-config probe, prefix caching off): round 0 rate 0.428 / AL 2.03 -> round 9 rate 0.857 / AL 5.70. The final-round delta is only ~1.4k drafted tokens, so the end rate moves a few points between reruns (0.79-0.86 observed; greedy near-ties under varying async batch composition); it clears the same 0.80 floor the CPU test uses. ### Greedy consistency ``` $ VLLM_BATCH_INVARIANT=1 python benchmarks/greedy_consistency.py 16 16 prompts x 256 tokens, k=16, Llama-3.1-8B-Instruct, enforce_eager: suffix_gpu (graph) vs async no-spec : 0/16 mismatches suffix_gpu (eager) vs async no-spec : 0/16 suffix_gpu eager vs graph replay : 0/16 (graph does not change outputs) ngram_gpu vs async no-spec : 0/16 suffix CPU (sync) vs async no-spec : 0/16 identical: true ``` ### Spec-Bench serving (Llama-3.1-8B-Instruct, 1x L20) 480 Spec-Bench prompts per point, output len 256, prefix caching off, `--max-num-batched-tokens 8192 --max-num-seqs 320` for every variant, identical 64-prompt warmup per server before measurement. Full script attached below; raw results under `benchmarks/results/`. **k=5 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 58 (17.2) | 245 (15.4) | 754 (19.8) | 1297 (45.5) | 1343 (86.0) | 1375 (127.2) | 1373 (169.5) | | ngram_gpu_async | 46 (21.6) | 168 (23.3) | 515 (30.3) | 1126 (52.7) | 1341 (89.2) | 1389 (128.5) | 1396 (172.3) | | suffix_gpu_async | 57 (17.6) | 239 (15.6) | 744 (19.3) | 1316 (43.2) | 1410 (80.5) | 1461 (117.6) | 1467 (154.5) | **k=16 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 55 (18.2) | 239 (15.9) | 755 (19.5) | 1273 (46.3) | 1322 (86.0) | 1357 (125.8) | 1363 (167.2) | | ngram_gpu_async | 43 (23.0) | 161 (24.3) | 519 (29.9) | 1156 (52.2) | 1339 (89.6) | 1387 (131.0) | 1405 (170.6) | | suffix_gpu_async* | 52 (19.2) | 230 (16.2) | 736 (19.5) | 1276 (44.8) | 1349 (84.3) | 1402 (121.5) | 1417 (154.6) | **Drafted / accepted tokens per point (AL, acceptance rate)** | variant | k | c1 | c16 | c64 | c256 | |---|---|---|---|---|---| | suffix_sync | 5 | 79832/22918 (1.54, 28.7%) | 86405/33544 (1.94, 38.8%) | 86516/32697 (1.89, 37.8%) | 86240/31708 (1.85, 36.8%) | | suffix_gpu_async | 5 | 79742/22332 (1.52, 28.0%) | 87458/33453 (1.93, 38.3%) | 88350/35551 (2.04, 40.2%) | 88444/38485 (2.23, 43.5%) | | ngram_gpu_async | 5 | 33913/9467 (2.40, 27.9%) | 34863/9738 (2.40, 27.9%) | 34527/9332 (2.35, 27.0%) | 34358/9667 (2.41, 28.1%) | | suffix_sync | 16 | 83232/22867 (1.54, 27.5%) | 95070/34453 (1.98, 36.2%) | 93340/33505 (1.94, 35.9%) | 92211/32536 (1.89, 35.3%) | | suffix_gpu_async | 16 | 82552/22582 (1.53, 27.4%) | 95486/34166 (2.00, 35.8%) | 98323/36571 (2.13, 37.2%) | 105420/40513 (2.33, 38.4%) | | ngram_gpu_async | 16 | 35468/9978 (2.41, 28.1%) | 34844/9586 (2.38, 27.5%) | 33685/9643 (2.43, 28.6%) | 34316/9778 (2.42, 28.5%) | Findings: 1. **suffix_gpu + async vs suffix + sync** (the conflict this PR resolves): parity at c1-c16, ahead from c64 up — +7% at k=5 saturation (1467 vs 1373 tok/s), +4% at k=16 (1417 vs 1363), with lower TPOT (154.5 vs 169.5 ms at k=5 c256). 2. **vs ngram_gpu + async** (in-tree GPU drafter): ahead at every grid point, up to +44% (k=5 c16: 744 vs 515 tok/s). 3. **vs async no-spec**: ahead through c64 (TPOT -8% at c64, -27% at c16, -31% at c4), behind at c128+ on this mixed workload (1467 vs 1654 at k=5 c256) — at saturation, verification compute outweighs acceptance on Spec-Bench's non-repetitive categories. Same regime behavior as the other drafters (suffix_sync 1373, ngram_gpu 1396). 4. **Acceptance scales with concurrency for suffix_gpu only**: AL 1.52 -> 2.23 (k=5) and 1.53 -> 2.33 (k=16) from c1 to c256 while the CPU suffix tree stays flat (1.94 -> 1.85, 1.98 -> 1.89) — the cross-request global index compounds under concurrent traffic. 5. Per-position acceptance at c=1 is essentially identical to the CPU suffix tree (k=16: 0.328/0.113/0.045... vs 0.329/0.115/0.046...) — the GPU drafter reproduces the CPU method's draft quality; full table in the RFC. --- - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) Documentation update — planned (see RFC \"Docs plan\"): `suffix_gpu` subsection in `docs/features/spec_decode.md` (when to pick it over suffix/ngram_gpu, `suffix_gpu_*` knobs, async-scheduling note, large-k memory-headroom guidance, install hint); lands with this PR or as an immediate follow-up once packaging (in-tree vs external) is settled. AI assistance was used for code analysis, benchmark scripts; all changed lines were reviewed and validated by the author, and all tests above were run by the author. Signed-off-by: zip95297 ---
Benchmark reproduction script (bench_pr_matrix.sh) ```bash #!/bin/bash # PR benchmark matrix for suffix_gpu (mirrors vllm-project/vllm#25784 setup). # # Grid: 4 variants x k {5,16} x concurrency {1,4,16,64,128,192,256} # async_nospec : async scheduling, no spec (baseline, k-independent) # suffix_sync : method=suffix (CPU tree, arctic), sync scheduling # (async rejects it -- that is the point of this PR) # ngram_gpu_async : method=ngram_gpu, async scheduling (in-tree GPU drafter) # suffix_gpu_async : method=suffix_gpu, async scheduling (this PR) # # Model: meta-llama/Llama-3.1-8B-Instruct (bf16, TP=1), 1x NVIDIA L20 48GB # Dataset: Spec-Bench question.jsonl (md5 0c39ae23e6f213549c66d6d691c99034) # Suffix knobs shared with the CPU suffix method: max_tree_depth=24, # max_cached_requests=1000. ngram uses prompt_lookup 3-5 (the stronger # baseline config from #25784). set -x export HF_HUB_OFFLINE=1 # pyarrow's embedded jemalloc background thread SIGSEGVs in the EngineCore # subprocess on this box; disable it. export JE_ARROW_MALLOC_CONF=background_thread:false # All variants forced onto model runner v1 for a fair comparison # (suffix/ngram_gpu/suffix_gpu fall back to v1 anyway). export VLLM_USE_V2_MODEL_RUNNER=0 BASE=/home/admin/workspace/aop_lab/app_source/zjb/suffix_async/pr_to_vllm_final REPO=${BASE}/vllm MODEL=meta-llama/Llama-3.1-8B-Instruct DATA=${BASE}/benchmarks/question.jsonl OUT=${BASE}/benchmarks/results PATH=\"${REPO}/.venv/bin:${PATH}\" # libcuda 580 shim (kernel driver 580 / userland libcuda 12.2 mismatch) export LD_LIBRARY_PATH=${BASE}/.cudafix:${LD_LIBRARY_PATH:-} export HF_HUB_CACHE=${BASE}/hf_cache mkdir -p \"${OUT}\" cd \"${BASE}/benchmarks\" [ -f \"${DATA}\" ] || cp /home/admin/question.jsonl \"${DATA}\" echo \"$(cd ${REPO} && git rev-parse HEAD) $(cd ${REPO} && git status --short | wc -l) dirty-files\" \\ > \"${OUT}/run_manifest.txt\" echo \"suffix-gpu $(git -C ${BASE}/../SuffixGPU rev-parse --short HEAD) $(git -C ${BASE}/../SuffixGPU status --short | wc -l) dirty-files\" \\ >> \"${OUT}/run_manifest.txt\" date >> \"${OUT}/run_manifest.txt\" wait_server() { local pid=$1 i=0 until curl -s http://localhost:8000/v1/models >/dev/null; do kill -0 \"$pid\" 2>/dev/null || { echo \"SERVER PROCESS DIED\"; return 1; } sleep 5 i=$((i + 5)) if [ \"$i\" -ge 900 ]; then echo \"SERVER START TIMEOUT\"; return 1; fi done } for k in 5 16; do suffix_cfg=$(cat < \"${OUT}/server_${name}_k${k}.log\" 2>&1 & pid=$! if ! wait_server ${pid}; then echo \"SKIP ${name} k${k}: server failed to start\" | tee -a \"${OUT}/failures.txt\" kill ${pid} 2>/dev/null; wait ${pid} 2>/dev/null; sleep 10 continue fi # Same warmup for every variant before measuring: model/drafter CUDA # graphs, Triton JIT and the suffix global index all start cold. # Snapshot the cumulative spec counters afterwards so the first # measured point's delta excludes warmup traffic. vllm bench serve \\ --model ${MODEL} \\ --dataset-name spec_bench \\ --dataset-path \"${DATA}\" \\ --spec-bench-output-len 256 \\ --num-prompts 64 \\ --max-concurrency 16 \\ --no-oversample curl -s http://localhost:8000/metrics \\ | grep -E \"spec_decode\" > \"${OUT}/metrics_${name}_k${k}_warmup.txt\" || true for concurrency in 1 4 16 64 128 192 256; do vllm bench serve \\ --model ${MODEL} \\ --dataset-name spec_bench \\ --dataset-path \"${DATA}\" \\ --spec-bench-output-len 256 \\ --max-concurrency ${concurrency} \\ --no-oversample \\ --save-result \\ --result-filename \"${OUT}/bench_${name}_k${k}_c${concurrency}.json\" # Per-concurrency spec metrics delta comes from the cumulative # counters; snapshot after each client run. curl -s http://localhost:8000/metrics \\ | grep -E \"spec_decode\" > \"${OUT}/metrics_${name}_k${k}_c${concurrency}.txt\" || true done kill ${pid}; wait ${pid} 2>/dev/null sleep 10 done done echo \"BENCHMARK ALL DONE\" ```
Greedy-consistency gate (greedy_consistency.py) ```python # Greedy-consistency gate for suffix_gpu. # # Speculative decoding must not change greedy outputs: run the same # prompts with (a) async scheduling + no spec and (b) async scheduling + # suffix_gpu, and require token-identical outputs. Additional legs # (suffix_gpu eager, ngram_gpu, CPU suffix) localize any divergence. # # Bitwise comparability requires batch-invariant kernels: spec decode # changes batch composition, and FlashAttention's split heuristics make # FP reduction order batch-dependent, flipping near-tie greedy argmax # (verified: without this, async no-spec disagrees with itself across # duplicated prompts in one batch). So we set VLLM_BATCH_INVARIANT=1 and # enforce_eager=True (batch-invariant aten overrides must not be # bypassed by compile/cudagraphs; on SM<90 vLLM cannot guarantee them # under compiled paths). # # Each engine runs in its own subprocess so GPU memory is fully # released between runs (spawn context; worker must be module-level). import json import os import sys os.environ.setdefault(\"VLLM_BATCH_INVARIANT\", \"1\") from vllm import LLM, SamplingParams MODEL = \"meta-llama/Llama-3.1-8B-Instruct\" K = int(sys.argv[1]) if len(sys.argv) > 1 else 16 SUFFIX_GPU_SPEC = { \"method\": \"suffix_gpu\", \"num_speculative_tokens\": K, \"suffix_decoding_max_cached_requests\": 1000, \"suffix_decoding_max_tree_depth\": 24, } SUFFIX_CPU_SPEC = { \"method\": \"suffix\", \"num_speculative_tokens\": K, \"suffix_decoding_max_cached_requests\": 1000, \"suffix_decoding_max_tree_depth\": 24, } # High-repetition prompts (exercise the accept path) + regular prompts # (exercise the empty-draft path), repeated for multi-request scheduling. PROMPTS = [ \"Repeat the following sentence 20 times: the quick brown fox jumps over the lazy dog.\", \"Count from 1 to 100, separated by commas.\", \"Explain the theory of relativity in one paragraph.\", 'List the JSON object {\"a\": 1, \"b\": 2} fifteen times, one per line.', ] * 4 SP = SamplingParams(temperature=0.0, max_tokens=256, ignore_eos=True) def run(mode: str) -> list[list[int]]: kwargs = { \"model\": MODEL, \"enable_prefix_caching\": False, \"gpu_memory_utilization\": 0.8, # Batch-invariant aten overrides must not be bypassed by # compile/cudagraphs (SM8.9 cannot guarantee them otherwise). \"enforce_eager\": True, } if mode == \"async_nospec\": kwargs[\"async_scheduling\"] = True elif mode == \"async_suffix_gpu\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = SUFFIX_GPU_SPEC elif mode == \"async_suffix_gpu_eager\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = { **SUFFIX_GPU_SPEC, \"suffix_gpu_use_cuda_graph\": False, } elif mode == \"async_ngram_gpu\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = { \"method\": \"ngram_gpu\", \"num_speculative_tokens\": K, \"prompt_lookup_min\": 3, \"prompt_lookup_max\": 5, } elif mode == \"sync_suffix_cpu\": kwargs[\"async_scheduling\"] = False kwargs[\"speculative_config\"] = SUFFIX_CPU_SPEC else: raise ValueError(mode) llm = LLM(**kwargs) outs = llm.generate(PROMPTS, SP) token_ids = [list(o.outputs[0].token_ids) for o in outs] del llm return token_ids def _worker(q, mode): q.put(run(mode)) def run_in_subprocess(mode: str) -> list[list[int]]: import multiprocessing as mp ctx = mp.get_context(\"spawn\") q = ctx.Queue() for _attempt in range(3): p = ctx.Process(target=_worker, args=(q, mode)) p.start() try: ids = q.get(timeout=1800) p.join() return ids except Exception: p.kill() p.join() import time time.sleep(20) raise RuntimeError(f\"engine failed 3 times for mode={mode}\") def diff(ref: list[list[int]], got: list[list[int]]) -> list[dict]: mismatches = [] for i, (r, g) in enumerate(zip(ref, got)): if r != g: j = next( k for k in range(min(len(r), len(g)) + 1) if k >= min(len(r), len(g)) or r[k] != g[k] ) mismatches.append( { \"prompt_idx\": i, \"first_diff_pos\": j, \"ref_len\": len(r), \"got_len\": len(g), } ) return mismatches if __name__ == \"__main__\": import os prev = f\"greedy_consistency_k{K}.json\" if os.path.exists(prev): cached = json.load(open(prev)) nospec_ids = cached[\"nospec_ids\"] sgpu_ids = cached[\"suffix_gpu_ids\"] scpu_ids = cached[\"suffix_cpu_ids\"] print(\"reusing cached nospec/suffix_gpu/suffix_cpu ids\") else: nospec_ids = run_in_subprocess(\"async_nospec\") sgpu_ids = run_in_subprocess(\"async_suffix_gpu\") scpu_ids = run_in_subprocess(\"sync_suffix_cpu\") sgpu_eager_ids = run_in_subprocess(\"async_suffix_gpu_eager\") ngram_ids = run_in_subprocess(\"async_ngram_gpu\") result = { \"k\": K, \"num_prompts\": len(PROMPTS), \"suffix_gpu_vs_nospec\": diff(nospec_ids, sgpu_ids), \"suffix_gpu_eager_vs_nospec\": diff(nospec_ids, sgpu_eager_ids), \"suffix_gpu_eager_vs_graph\": diff(sgpu_ids, sgpu_eager_ids), \"ngram_gpu_vs_nospec\": diff(nospec_ids, ngram_ids), \"suffix_cpu_vs_nospec\": diff(nospec_ids, scpu_ids), } result[\"identical\"] = not result[\"suffix_gpu_vs_nospec\"] print(json.dumps(result, indent=2)) with open(f\"greedy_consistency_k{K}_full.json\", \"w\") as f: json.dump( { \"result\": result, \"nospec_ids\": nospec_ids, \"suffix_gpu_ids\": sgpu_ids, \"suffix_gpu_eager_ids\": sgpu_eager_ids, \"ngram_gpu_ids\": ngram_ids, \"suffix_cpu_ids\": scpu_ids, }, f, ) ```
", "url": "https://github.com/vllm-project/vllm/pull/52097", "createdAt": "2026-08-13T04:34:29Z", "updatedAt": "2026-08-13T04:35:07Z", "timestamp": "2026-08-13T04:35:07Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "speculative-decoding" ], "author": "zip95297", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52098", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Log output token IDs at DEBUG level", "text": "## Purpose Allow operators to keep human-readable generated output logs without emitting output token IDs at the default INFO level. Following maintainer feedback, this now mirrors the existing request-input logging split instead of adding a new CLI flag: - INFO keeps generated text and the finish reason. - DEBUG additionally logs output token IDs. - `--max-log-len` continues to truncate both output text and token IDs at their respective levels. - Streaming delta, streaming-complete, and non-streaming output logs share the same behavior. Fixes #51912. Duplicate check: I rechecked the issue discussion and open PRs by issue number and output-token-ID logging keywords. No other open PR implements this behavior. No model behavior or evaluation output changes. Model evals are not applicable because this only changes request log formatting and CLI help text. AI assistance disclosure: OpenAI Codex assisted with implementation, review-feedback handling, and test execution. The human submitter must review every changed line, understand the end-to-end change, and take responsibility for the PR before merge. ## Test Plan - Execute the existing `tests/entrypoints/serve/utils/test_request_logger.py` unit tests, covering INFO/DEBUG separation, streaming and non-streaming logs, truncation, and edge cases. - Run Ruff check and format validation on all three net-changed files. - Run the repository Python 3.10 mypy hook on all three net-changed files. - Run `validate-config`, applicable commit hooks, and `git diff --check`. ## Test Result - Request logger unit tests: 9 passed in 0.12s. The local Windows environment did not contain the full vLLM runtime/test dependency set, so the existing test file was run with `--noconftest` and minimal import stubs; the real `RequestLogger` implementation and all tests in that file were executed. - Ruff check and format: passed. - Python 3.10 mypy on the three net-changed files: passed. - `validate-config`: passed with UTF-8 console mode. - Applicable commit hooks passed, including typos, SPDX, forbidden imports, configuration validation, and DCO sign-off. - `git diff --check`: passed. - `actionlint` and `update-dockerfile-graph` were skipped because no workflow or Dockerfile changed. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test commands. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] Documentation impact considered. CLI help text now describes the INFO/DEBUG split; no separate documentation page is needed.
", "url": "https://github.com/vllm-project/vllm/pull/52098", "createdAt": "2026-08-13T04:51:35Z", "updatedAt": "2026-08-13T13:11:52Z", "timestamp": "2026-08-13T13:11:52Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "frontend", "ready" ], "author": "ruirui6946", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52099", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][MoE] Add explicit expert-parallel topology", "text": "## Summary - add an experimental `--expert-parallel-size` independent of tensor parallel size - construct fixed-TP-lane EP groups, e.g. TP4/DP2 gives `[0,4]`, `[1,5]`, `[2,6]`, `[3,7]` - preserve existing flattened EP behavior when the option is unset - fail closed for PP/PCP, EPLB, elastic EP, non-AG/RS backends, and EP sizes other than DP This is the topology foundation for experts that remain tensor-sharded while ownership is distributed across DP ranks. MoE execution support is stacked in #52100. ## Test plan - [x] `pytest -q tests/config/test_explicit_expert_parallel.py` — 9 passed - [x] Ruff check and format check on changed files - [x] `git diff --check`", "url": "https://github.com/vllm-project/vllm/pull/52099", "createdAt": "2026-08-13T04:52:10Z", "updatedAt": "2026-08-13T05:00:13Z", "timestamp": "2026-08-13T05:00:13Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52100", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Preserve TP sharding with explicit EP", "text": "## Summary - preserve physical TP size/rank inside experts when explicit EP is configured - use the fixed-lane EP group for expert mapping and pre-I/O weight filtering - keep AG/RS dispatch/combine over EP and the existing late TP all-reduce - add coverage for TP4/EP2 config, Kimi expert ownership, loader filtering, final TP reduction, and the ROCm latent tail Stacked on #52099. The topology commit disappears once #52099 merges. Kimi-K3 has 896 routed experts. With EP2, each process owns 448 experts with TP4-sharded weights; four ranks in a TP group own distinct shards of the same 448-expert set. No Kimi production-code special case is required: the existing ROCm latent runner consumes the preserved physical TP size. ## 8-GPU validation - topology: TP4/DP2/EP2, AG/RS, target-only, 100K input + 1K output, concurrency 1 - startup log: `Local/global number of experts: 448/896`; weight filter loads `448/896` - execution log: `MoEPrepareAndFinalizeNaiveDPEPModular` + `AiterExperts` - numerical gate: GSM8K **100/100 strict**, **100/100 flexible**, zero errors/malformed - TPOT over 3 runs: **27.296 ms mean** (27.107–27.447 ms); correctness passes, but this topology is not the C1 performance winner - non-MTP performance gate is met by TP8/DP1: **18.059 ms mean TPOT** over 3 runs (18.056–18.062 ms) ## Test plan - [x] topology + generic execution suite — 13 passed - [x] Kimi ROCm latent runner + hybrid MoE suite — 19 passed on 8 GPUs - [x] Ruff check and format check on changed files - [x] `git diff --check` - [x] 8-GPU TP4/DP2/EP2 Kimi-K3 serving and GSM8K validation", "url": "https://github.com/vllm-project/vllm/pull/52100", "createdAt": "2026-08-13T04:54:49Z", "updatedAt": "2026-08-13T05:00:15Z", "timestamp": "2026-08-13T05:00:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52101", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Distributed][MoonEP] BF16 PoC integration of MoonEP balanced EP backend", "text": "## Purpose First item of the MoonEP integration roadmap RFC #52095: a correctness-first BF16 proof of concept of MoonEP (https://github.com/MoonshotAI/MoonEP) as a vLLM all2all backend. Included: - `\"moonep\"` `All2AllBackend` registration and `has_moonep()` probe - `MoonEPAll2AllManager` owning the cached `moonep.Buffer` - `MoonEPPrepareAndFinalize` conforming to the modular prepare/finalize interface: pad to static capacity → `dispatch` → `prefetch_weight` for the planned redundant experts (the `MoonEPCommPlan` is stashed between prepare and finalize, the same pattern DeepEP-HT uses for its handle) → `combine` in finalize - Replicated `[E+B, …]` BF16 weight-layout helper and a reference segment-loop expert runner over `cu_seqlens` - `tools/moonep/validate_moonep_bf16_poc.py`: distributed validation of the full dispatch/prefetch/compute/combine path against a dense per-token reference MoE Known PoC limitations, tracked as follow-up items in #52095: BF16/unquantized only; eager only; reference segment loop rather than a grouped GEMM; expert weights replicated per rank rather than MoonEP's sharded symmetric-memory ownership; no FusedMoE layer/engine wiring yet (engine use of the backend is not supported by this PR). ## Test Plan On an NVLink symmetric-memory capable node with the `moonep` package installed: ```bash torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py --skew 8.0 torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py \\ --num-tokens 512 --capacity 512 --hidden-size 2048 \\ --intermediate-size 768 --topk 8 --num-experts 64 --skew 4.0 ``` ## Test Result All three configurations pass on 4× GB300 (EP=4, NVLink), every rank `global_ok=True`: | Config | max rel_err | |---|---| | Default (S=128, H=512, top-4 of 32 experts) | 7.7e-3 | | Heavy router skew (`--skew 8`, exercises the redundant-expert planner + prefetch slots) | 5.7e-3 | | Larger shapes (S=512, H=2048, top-8 of 64 experts, skew 4) | 1.2e-2 | All within BF16 tolerance (2e-2) against the dense per-token reference. No model evaluation is included: this backend is opt-in, is not wired into the engine by this PR, and does not affect any existing serving path or model output. ## Notes - Not a duplicate: no open PR or issue in this repo implements a MoonEP backend (searched \"MoonEP\" across PRs/issues; no implementation PRs exist). - I have reviewed and tested every changed line.", "url": "https://github.com/vllm-project/vllm/pull/52101", "createdAt": "2026-08-13T04:57:26Z", "updatedAt": "2026-08-13T14:20:27Z", "timestamp": "2026-08-13T14:20:27Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "nvidia" ], "author": "kaijunli-infr", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52102", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Multimodal] Guard read_frames against empty frame indices", "text": "## Summary `Qwen3VLVideoBackend.compute_frames_index_to_sample()` returns an empty list for a 0-frame video (it lacks the base class's `max(1, ...)` floor), which then crashed `VideoBackend.read_frames()`: - **frame-recovery path** (`_read_frames_with_recovery`): `IndexError: list index out of range` at `next_target_map[frame_indices[-1]] = total_frames` — the same line's neighbor `max_frame_idx = frame_indices[-1] if frame_indices else 0` is guarded a few lines above, but this one is not. - **no-recovery path**: `ValueError: max() iterable argument is empty`. `read_frames` is the single entry point shared by **every** video backend (OpenCV / PyAV / TorchCodec / DeepStream), so the guard protects all of them. This adds an early return for empty frame lists that matches the empty-shaped frames (`np.empty((0, h, w, 3))`) the recovery path already produces for empty input — the same state downstream code already receives today for 0-frame videos through the base-class path. ## Test ```bash .venv/bin/python -m pytest \\ \"tests/multimodal/test_video.py::test_read_frames_handles_empty_frame_indices\" \\ \"tests/multimodal/test_video.py::test_qwen3vl_zero_frame_video_does_not_crash\" -v # 3 passed ``` - `ruff check` / `ruff format` on both changed files: clean. - Full `tests/multimodal/test_video.py` run: 40 passed, 5 skipped; the 25 failures are all `ModuleNotFoundError: No module named 'av'/'torchcodec'` (video decode deps absent in this CPU environment), unrelated to this change. **Model evaluation:** not applicable — this is a crash-only fix for degenerate (0-frame) input; it does not change output, accuracy, or serving behavior for any valid input. ## Duplicate / overlap check No open PR references an empty-frame crash in `read_frames` or the Qwen3VL `compute_frames_index_to_sample` path. - #50990 *\"Validate dynamic video sampling metadata\"* is related in theme but does **not** overlap: it adds validation to `DynamicVideoBackend` only, and does not touch `read_frames`, the Qwen3VL backend, or any backend's `read_frames` boundary. - `bad_words` / YAML-config / JSON-schema empty-input fixes are unrelated code paths. ## AI assistance This change was developed with AI assistance (Claude Code): bug identification, fix, and tests were AI-drafted and reviewed by a human before submission.", "url": "https://github.com/vllm-project/vllm/pull/52102", "createdAt": "2026-08-13T04:59:11Z", "updatedAt": "2026-08-13T05:11:45Z", "timestamp": "2026-08-13T05:11:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "multi-modality" ], "author": "zuver-lab", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52103", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Carry request provenance through stored events", "text": "## Purpose Carry request provenance through asynchronous KV-offload completions so locally initiated stores and promotions can emit full `BlockStored` events. The current tracker snapshots full payloads at scheduler lookup/store time and keeps them in a global map. This PR instead keeps a lazy event locator with the request and builds the payload when the raw stored event arrives. ```text Request + KV-group geometry | ReqContext event state | existing async job paths | CPU / FS / OBJ completion | OffloadingEvent(key, ReqContext) | full BlockStored ``` ## Changes - Attach request-scoped event state to `ReqContext`. - Lazily resolve a stored key to its full-chunk or partial-tail boundary. - Carry the exact `ReqContext` through CPU, FS, OBJ, cascade, and promotion completion. - Use context identity, not request ID, when an old completion is finalized. - Generate GPU-equivalent per-block `extra_keys`, including multimodal and prompt-embedding inputs. - Keep placeholders for external work, unsupported cache shapes, and unsafe token-mutating requests. - Keep only detached chunk hashes for the existing CPU removal contract; #49413 PR 2 removes that table. - Preserve queued CPU removals and their detached expansion metadata across reset. - Move context ownership from active work to the raw stored event, then release it after translation; failed submissions and shutdown unwind their owners. ## Scope This PR does not change: - `OffloadKey = (block_hash, group_idx)`; - `prepare_store` admission or batching; - worker transfer metadata; - the legacy expanded-removal contract; or - the public KV-event schema. Single-key admission, k-way ordering, and #44865 remain separate work. This PR is based on the KV-group block-granularity fix merged in #51614. Token-mutating paths affected by #49377 keep placeholder payloads until their hash and offload state are safe. ## Duplicate-work check - #49506 is superseded by this narrower request-scoped design. - #51614 is merged and provides the block-granularity base. - #50087 changes request-level store policy/admission. - #52022 fixes store-threshold accounting. - #51646 only updates event-medium documentation. No open PR above carries request provenance through async stored-event completion. ## Tests All validation ran on the workstation against `3d204dfdaa`: ```text .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_events.py -q 32 passed .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py -q 122 passed .venv/bin/python -m pytest \\ tests/v1/kv_offload/cpu/test_manager.py \\ tests/v1/kv_offload/tiering/test_async_lookup.py \\ tests/v1/kv_offload/tiering/test_fs_tier.py \\ tests/v1/kv_offload/tiering/test_obj_tier.py \\ tests/v1/kv_offload/tiering/test_tiering_offloading.py -q 182 passed pre-commit run ruff-format --files \\ $(git diff --name-only origin/main...HEAD -- '*.py') Passed pre-commit run ruff-check --files \\ $(git diff --name-only origin/main...HEAD -- '*.py') Passed pre-commit run mypy-3.12 --hook-stage manual \\ --files $(git diff --name-only origin/main...HEAD -- vllm) Passed ``` Model evaluation: not applicable. This changes event metadata and lifecycle only, not model execution or output. ## AI assistance AI assistance was used to trace the event lifecycle, implement the change, and draft tests. The human submitter must review every changed line and these test results before submission.", "url": "https://github.com/vllm-project/vllm/pull/52103", "createdAt": "2026-08-13T05:44:02Z", "updatedAt": "2026-08-13T05:44:38Z", "timestamp": "2026-08-13T05:44:38Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "kv-connector" ], "author": "Change72", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52106", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Core] Restore incremental thinking-budget marker scans", "text": "## Purpose PR #46425 made thinking-budget marker searches incremental by advancing per-search cursors after unsuccessful scans. PR #45984 later removed those cursors while fixing natural `` re-entry and replaced them with a shared `scan_offset` that only advances when a thinking section exits. As a result, while a single long thinking section remains open and the end marker has not been generated, every decode step scans the complete growing output again. The cumulative marker-search work therefore regressed from linear to quadratic. This PR advances `scan_offset` after every marker-search update while retaining `max_marker_len - 1` tokens of overlap. The overlap preserves detection of multi-token markers split across decode steps. Search results are still converted to absolute output indices, and the existing natural-end, forced-end, re-entry, and speculative-decoding state transitions are unchanged. This restores the incremental marker-search behavior introduced by #46425 and regressed by #45984. ## Test Plan - Add a deterministic complexity regression test that instruments `_find_last_sequence_index()` and counts the total number of tokens passed to marker searches. - Assert a linear upper bound based on output length and maximum marker length instead of using a wall-clock threshold. - Run all thinking-budget tests and the complete logits-processor correctness test file. - Run Ruff checks on the modified files. ```text .venv/bin/python -m pytest tests/v1/logits_processors/test_correctness.py -k thinking_budget -q .venv/bin/python -m pytest tests/v1/logits_processors/test_correctness.py -q uvx ruff check vllm/v1/sample/thinking_budget_state.py tests/v1/logits_processors/test_correctness.py uvx ruff format --check vllm/v1/sample/thinking_budget_state.py tests/v1/logits_processors/test_correctness.py ``` ## Test Result ### Complexity regression The table below reports the cumulative number of tokens passed to marker searches while repeatedly appending one token to a long open thinking section. | Revision | 500 steps | 1,000 steps | Growth ratio | |---|---:|---:|---:| | #46425 merge (`7f99e80c3`) | 1,504 | 3,004 | 2.00x | | #45984 merge (`ed908cf0a`) | 126,254 | 502,504 | 3.98x | | Upstream main before this patch | 126,254 | 502,504 | 3.98x | | This patch | 1,504 | 3,004 | 2.00x | The regressed implementation grows approximately fourfold when the sequence length doubles, while the patched implementation grows approximately twofold. ### CPU micro-benchmark The CPU micro-benchmark follows the per-decode-step setup used by #46425: append one token to the full output list on every step and search for an end marker that is not present. Each result is the median of three runs. | Steps | Upstream main | This patch | Speedup | |---:|---:|---:|---:| | 1,000 | 0.021085 s | 0.000700 s | 30.1x | | 2,000 | 0.089897 s | 0.001439 s | 62.5x | | 4,000 | 0.359857 s | 0.002841 s | 126.6x | | 8,000 | 1.462388 s | 0.005671 s | 257.9x | ### Correctness and lint ```text 15 passed, 26 deselected 41 passed All Ruff checks passed 2 files already formatted ``` ## Duplicate-work Check As of August 13, 2026, searches for open PRs containing `thinking_token_budget`, `scan_offset`, and thinking-budget performance did not find another PR addressing this regression. ## AI Assistance AI assistance was used for source-history analysis, implementation, and test preparation. I reviewed the complete diff, verified the algorithmic reasoning, and ran the tests listed above.", "url": "https://github.com/vllm-project/vllm/pull/52106", "createdAt": "2026-08-13T06:24:52Z", "updatedAt": "2026-08-13T06:25:28Z", "timestamp": "2026-08-13T06:25:28Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "Xilluill", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52108", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][CI/Release][3/N] Add xpu wheel release to release pipeline", "text": "# Summary This PR adds XPU wheel building and publishing support to the Buildkite release pipeline, enabling pre-built XPU wheels to be distributed via wheels.vllm.ai. # Changes Release pipeline (release-pipeline.yaml) - Add a new `Build wheel - x86_64 - XPU` step that builds the XPU wheel using `Dockerfile.xpu` and uploads it to S3. - Rename the existing `Publish XPU Triton shim index` step to `Publish and stage XPU Triton shim`, and extend it to also stage the Triton shim wheel into the per-commit S3 directory so that the nightly index generator can pick it up. Nightly index generation (generate-and-upload-nightly-index.sh, generate-nightly-index.py) - Filter for `vllm-*` wheels when detecting the version from the commit directory, avoiding false matches on the Triton shim wheel. - Recognize xpu as a known variant suffix in wheel filename parsing. - Include `triton-*+xpu-*` wheels in release (non-nightly) index generation so the XPU Triton shim is co-published alongside the vLLM XPU wheel. Triton shim publish script (publish-triton-shim.sh) - Stage the Triton shim wheel into the per-commit S3 prefix (s3://vllm-wheels/$COMMIT/) so it is available alongside the vLLM XPU wheel. - Add BUILDKITE_COMMIT validation and fix --wheel-dir / --output-dir paths. Dockerfile.xpu (Dockerfile.xpu) - Add a vllm-build multi-stage target that produces a standalone dist/*.whl artifact for CI wheel builds. - Remove the manual triton/triton-xpu uninstall-reinstall workaround from the runtime stage (no longer needed with the Triton shim). Documentation (gpu.xpu.inc.md) - Replace \"no pre-built XPU wheels\" with installation instructions for nightly and per-commit wheels via wheels.vllm.ai. - Document the triton==3.7.2+xpu shim and remove the manual triton-xpu swap instructions. ## Test Plan trigger release pipeline for this branch, https://buildkite.com/vllm/release-v2/builds/5120 ## Test Result wheels can be found https://wheels.vllm.ai/9b7e36805bf42c0a23a58971fd72ba598fc1e984/xpu or install by uv pip install vllm --pre --extra-index-url=https://wheels.vllm.ai/9b7e36805bf42c0a23a58971fd72ba598fc1e984/xpu/ --extra-index-url=https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52108", "createdAt": "2026-08-13T06:41:24Z", "updatedAt": "2026-08-13T09:45:19Z", "timestamp": "2026-08-13T09:45:19Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation", "intel-gpu", "ci/build" ], "author": "jikunshang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52110", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[sharded state loader] support pp in sharded state loader", "text": "## Purpose For now, sharded state loader does not support pipeline parallel. Add pipeline parallel support for sharded state loader and keep backward compatibility, i.e., tensor parallel only. ## Test Plan Add some ut to cover newly added functionalities. 1. TP only, backward compatibility 2. PP only, save sharded state + load sharded state + generate 3. TP+PP, save sharded state + load sharded state + generate ## Test Result All pass ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52110", "createdAt": "2026-08-13T06:42:26Z", "updatedAt": "2026-08-13T07:16:26Z", "timestamp": "2026-08-13T07:16:26Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation" ], "author": "andyxning", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52112", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Fix a few int4/int8 quantization errors", "text": "## Purpose Fix a few quantization bugs on ROCm introduced by #44120 to enable int4/int8 quantized models like cyankiwi/MiniMax-M3-AWQ-INT4, QuantTrio/Qwen3-235B-A22B-GPTQ-Int8. 1. Add asym quantization support for TRITON moe backend; 2. Add SWIGLUOAI_UNINTERLEAVE activation for TRITON moe backend; 3. Remove wrong assertion (GROUP_SIZE==-1) from MoeWNA16Method; ## Test Plan 1. VLLM_USE_BREAKABLE_CUDAGRAPH=0 vllm serve cyankiwi/MiniMax-M3-AWQ-INT4/ --block-size 128 -tp 4 --attention-backend TRITON_ATTN --tool-call-parser minimax_m3 --enable-auto-tool-choice --reasoning-parser minimax_m3 --moe-backend triton --max-model-len 32768 2. vllm serve QuantTrio/Qwen3-235B-A22B-GPTQ-Int8/ -tp 8 --moe-backend triton Both generate correct answers.", "url": "https://github.com/vllm-project/vllm/pull/52112", "createdAt": "2026-08-13T06:44:17Z", "updatedAt": "2026-08-13T14:20:57Z", "timestamp": "2026-08-13T14:20:57Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm", "quantization" ], "author": "qli88", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52114", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] [Quantization] Add Ling hybrid MXFP4 routed experts support", "text": "## Purpose Add support for Ling checkpoints that use hybrid quantization: block FP8 for dense and shared-expert projections, and MXFP4 for routed experts. This change reads Ling-specific quantization metadata and remaps routed-expert scale names to the convention expected by `Mxfp4MoEMethod` for both the main and MTP models", "url": "https://github.com/vllm-project/vllm/pull/52114", "createdAt": "2026-08-13T06:59:24Z", "updatedAt": "2026-08-13T13:46:53Z", "timestamp": "2026-08-13T13:46:53Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "quantization" ], "author": "zexplorerhj", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52115", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Preserve Step3-VL zero-patch CUDA graph outputs", "text": "## Summary - honor the encoder postprocessor's `clone` contract when a Step3-VL image has zero local patches - keep the no-clone path as an alias while preserving cloned outputs across later CUDA graph replays - add a focused regression test for both ownership modes Without this, the single-part fast path returns the CUDA graph output buffer directly even when `clone=True`. A later replay can therefore overwrite a previously cached image embedding and make it represent another image. ## Validation - Focused leased RTX 2080 Ti (SM75, FP16) model-level CUDA graph validation: **BASE RED / PATCH GREEN** - Two distinct 512×512 images, `num_patches=[0, 0]`, real CUDA graph capture and production postprocessor - Base: the first retained output changed after replaying the second image - Patch: both replay-time outputs matched eager execution and the first retained output stayed unchanged - `git diff --check` - `python3 -m py_compile vllm/model_executor/models/step3_vl.py tests/models/multimodal/processing/test_step3_vl_image_embeds.py` A local focused pytest was not run because this isolated worktree has no project virtual environment; the GPU model-level harness exercised the target production path directly. ## Duplicate check I searched current open vLLM issues and PRs for Step3-VL, encoder CUDA graph, zero-patch, aliasing, and clone/output ownership. I found no open change addressing this ownership bug. The current `main` implementation still has the faulty single-part fast path. ## Model evaluation This is an ownership/lifetime correction rather than a numerical model change. The leased GPU validation compared both images against eager output at replay time and confirmed exact preservation after the second replay. ## AI assistance AI assistance was used for source investigation, implementation, and test-harness preparation. I reviewed every changed line and independently verified the diff, duplicate status, and GPU RED/GREEN evidence.", "url": "https://github.com/vllm-project/vllm/pull/52115", "createdAt": "2026-08-13T07:10:24Z", "updatedAt": "2026-08-13T07:12:02Z", "timestamp": "2026-08-13T07:12:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "multi-modality", "nvidia" ], "author": "qdivan", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52116", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fix DualChunkRotaryEmbedding hard-coded cuda device crash", "text": "## Purpose `DualChunkRotaryEmbedding.__init__` hard-codes `self.device = torch.device(f\"cuda:{device_idx}\")` alongside the portable `torch.accelerator.current_device_index()`. On any non-CUDA backend the subsequent `.to(device=self.device)` inside `_compute_cos_sin_cache` triggers CUDA lazy-init and crashes model loading with \"Torch not compiled with CUDA enabled\" for every Dual Chunk Attention model (e.g. Qwen2.5-*-1M). Reproduced on Ascend NPU when serving Qwen2.5-7B-Instruct-1M: ``` (EngineCore) Starting to load model .../Qwen2.5-7B-Instruct-1M... (EngineCore) EngineCore failed to start. Traceback (most recent call last): File \".../vllm/v1/engine/core.py\", line 1200, in run_engine_core engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) File \".../vllm/v1/engine/core.py\", line 123, in __init__ self.model_executor = executor_class(vllm_config) File \".../vllm/v1/executor/uniproc_executor.py\", line 68, in _init_executor self.driver_worker.load_model() File \".../vllm/model_executor/model_loader/__init__.py\", line 140, in get_model return loader.load_model(...) File \".../vllm/model_executor/model_loader/utils.py\", line 63, in initialize_model model = model_class(vllm_config=vllm_config, prefix=prefix) File \".../vllm/model_executor/models/qwen2.py\", line 458, in __init__ self.model = Qwen2Model(...) File \".../vllm/model_executor/models/qwen2.py\", line 375, in __init__ self.start_layer, self.end_layer, self.layers = make_layers(...) File \".../vllm/model_executor/models/utils.py\", line 711, in make_layers + get_offloader().wrap_modules(...) File \".../vllm/model_executor/offloader/base.py\", line 104, in wrap_modules return list(modules_generator) File \".../vllm/model_executor/models/qwen2.py\", line 263, in __init__ self.self_attn = Qwen2Attention(...) File \".../vllm/model_executor/models/qwen2.py\", line 178, in __init__ self.rotary_emb = get_rope(...) File \".../vllm/model_executor/layers/rotary_embedding/__init__.py\", line 92, in get_rope rotary_emb = DualChunkRotaryEmbedding(...) File \".../vllm/model_executor/layers/rotary_embedding/dual_chunk_rope.py\", line 42, in __init__ self._compute_cos_sin_cache() File \".../vllm/model_executor/layers/rotary_embedding/dual_chunk_rope.py\", line 105, in _compute_cos_sin_cache q_cache = torch.cat((q_cos, q_sin), dim=-1).to(...) File \".../torch/cuda/__init__.py\", line 417, in _lazy_init raise AssertionError(\"Torch not compiled with CUDA enabled\") AssertionError: Torch not compiled with CUDA enabled ``` Fix: use `current_platform.device_type` instead of the \"cuda\" literal - the same portable pattern already used across vllm (config, compilation passes, other layers). The device index still comes from `torch.accelerator.current_device_index()`. No cache values change; only the device the cos/sin caches are placed on. ## Test Plan - New unit test `tests/model_executor/layers/test_dual_chunk_rope.py`: ```bash python -m pytest tests/model_executor/layers/test_dual_chunk_rope.py -v ``` - E2E: `vllm serve` a Dual Chunk Attention model (e.g. Qwen2.5-7B-Instruct-1M) on a non-CUDA backend (Ascend NPU). ## Test Result **Before** (Ascend NPU): `vllm serve Qwen2.5-7B-Instruct-1M` crashes during model loading (traceback above), `AssertionError: Torch not compiled with CUDA enabled`. **After**: - UT: ``` tests/model_executor/layers/test_dual_chunk_rope.py::test_device_follows_platform_type PASSED tests/model_executor/layers/test_dual_chunk_rope.py::test_device_matches_real_platform PASSED ``` - `vllm serve Qwen2.5-7B-Instruct-1M` on Ascend NPU starts and serves normally. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52116", "createdAt": "2026-08-13T07:19:20Z", "updatedAt": "2026-08-13T07:19:56Z", "timestamp": "2026-08-13T07:19:56Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia" ], "author": "danziheng1024", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52117", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Do not require SupportsPP of draft models under pipeline parallelism", "text": "Fixes #52069. `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so `_verify_args` demands `SupportsPP` from the draft model. No head in `MTPModelTypes` implements it, so every MTP speculative config dies in `create_engine_config` under PP > 1. A draft model always runs whole on a single PP rank (the V2 runner builds it on the last rank only), so this verifies it as single-stage, on a shallow copy of the config. **Test plan:** without the change, `--pipeline-parallel-size 4 --speculative-config '{\"method\":\"mtp\",...}'` fails at config time on GLM-4.5-Air; with it, the server loads and serves (8× RTX 3090, three machines, nightlies dev678–dev693). Complementary to #46994, which does not touch this file. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52117", "createdAt": "2026-08-13T07:22:01Z", "updatedAt": "2026-08-13T18:01:09Z", "timestamp": "2026-08-13T18:01:09Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52118", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] [Bugfix] process ragged weights in xpu linear backend", "text": "python examples/basic/offline_inference/generate.py --model gaunernst/DeepSeek-V2-Lite-Chat-FP8 --enforce-eager --max-model-len 2048 --trust-remote-code Before: \"image\" After: \"image\"", "url": "https://github.com/vllm-project/vllm/pull/52118", "createdAt": "2026-08-13T07:29:01Z", "updatedAt": "2026-08-13T09:46:42Z", "timestamp": "2026-08-13T09:46:42Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "intel-gpu" ], "author": "zufangzhu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52119", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Kill regex compilation subprocess on timeout", "text": "Replace the thread-based compile_regex_with_timeout() with a fork-based subprocess that is SIGKILL'd on deadline expiry. The previous ThreadPoolExecutor approach could not cancel already-running callables, allowing adversarial regex patterns to accumulate CPU/memory-consuming compiler workers after each timeout error. - Use multiprocessing fork context for fast subprocess creation - SIGKILL + join the child before returning ValueError on timeout - Add global semaphore (VLLM_REGEX_COMPILATION_MAX_CONCURRENT, default 1) to bound concurrent compilation processes - Adapt xgrammar/outlines/lm-format-enforcer call sites to use picklable top-level workers with serialize/deserialize round-trips - Add regression tests asserting zero lingering processes after timeout", "url": "https://github.com/vllm-project/vllm/pull/52119", "createdAt": "2026-08-13T07:34:38Z", "updatedAt": "2026-08-13T07:35:14Z", "timestamp": "2026-08-13T07:35:14Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "structured-output" ], "author": "jperezdealgaba", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52120", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Fix ReDoS in validation error path sanitizer", "text": "## Summary - Fix CWE-1333 regex denial-of-service in `sanitize_message()` where a 100KB slash-delimited input (`\"/a\" * 50000`) could keep an API worker busy for 3+ seconds per rejected request. - Add a dot-character pre-check to skip the generic path regex entirely when no filename extension is possible, and use an atomic group `(?>...)` to prevent backtracking within repeated slash-segments for messages that do contain dots. - Addresses GHSA-f2g9-pmwr-xwc7. ## Test plan - [x] `pytest tests/entrypoints/serve/utils/test_api_utils.py -v` — 22 tests pass (4 new ReDoS regression tests) - [x] `pytest tests/entrypoints/serve/utils/test_error_sanitization.py -v` — 9 tests pass - [x] `pre-commit run --files ` — all hooks pass - [x] Reproduced advisory PoC shape (`\"/a\" * 50000`): completes in <0.1ms after fix (was 3+ second hang) - [x] Verified legitimate path redaction still works (`/app/server.py`, `/usr/lib/...` etc.) Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/52120", "createdAt": "2026-08-13T07:43:58Z", "updatedAt": "2026-08-13T07:59:25Z", "timestamp": "2026-08-13T07:59:25Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "ready" ], "author": "jperezdealgaba", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52121", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fall back to native rotary when flash-attn is installed but fails to load", "text": "## Purpose `ApplyRotaryEmb.__init__` guards the optional flash-attn rotary import with `suppress(ModuleNotFoundError)`. That covers a *missing* module, but not an *installed but unloadable* one: when the `flash_attn` C extension cannot be loaded, Python raises `ImportError` — the parent class — which escapes the narrower guard and aborts engine startup during model loading. `ApplyRotaryEmb` is constructed by `RotaryEmbedding.__init__` (`rotary_embedding/base.py`), so this guard runs when loading essentially any model that uses RoPE — not just the VL models that call it directly. A broken optional dependency therefore takes down startup for everything. The common trigger is mundane: rebuild or upgrade PyTorch without rebuilding flash-attn. `flash_attn.ops.triton.rotary` resolves fine, then loading `flash_attn_2_cuda` fails on a moved/removed `c10` symbol: ``` ImportError: .../flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE ``` This PR distinguishes the two cases rather than widening the guard blindly: - `ModuleNotFoundError` — flash-attn simply is not installed. Stay silent; this is the normal, expected path and its behaviour is unchanged. - any other `ImportError` — flash-attn is installed but broken. Fall back to native **and** emit `logger.warning_once`, so the user learns their flash-attn is unusable instead of silently running the slow path. A plain `suppress(ImportError)` would fix the crash but hide a genuinely broken install, which I believe is why the narrow exception type was requested in #42679. ## Why this is not a duplicate - **#42679** (merged) introduced the current `suppress(ModuleNotFoundError)`. It fixed FA4 environments where `flash_attn.ops.triton.rotary` is *absent*. It does not cover an import that fails during extension loading. - **#36900** (open since March) targets the same lines, but the failure it describes — `ModuleNotFoundError: No module named 'flash_attn.ops'` with `flash-attn-4` installed — is already handled by #42679's `suppress(ModuleNotFoundError)`. Its diff is also based on the older `find_spec` code that #42679 replaced, so it no longer applies. I believe #36900 can be closed as obsolete; the `ImportError` case is the only part still unaddressed. - **#43684** (merged) added a separate native fallback in this same method for a ROCm grid-limit issue, so falling back to native here is established precedent rather than new policy. I searched open/closed/merged PRs for `rotary_embedding common.py`, `flash_attn.ops.triton.rotary`, `ApplyRotaryEmb`, `suppress(ModuleNotFoundError)` and `undefined symbol` before opening this. ## Test plan and results Hardware: RTX PRO 5000 Blackwell (sm_120), CUDA 13.2, hand-built PyTorch 2.13.0, vLLM built from source. **1. Branch behaviour** — all three paths verified by patching `import_module` in the module under test (needs `set_current_vllm_config`, or the `default_vllm_config` fixture in `tests/conftest.py`): | case | expected | result | |---|---|---| | flash-attn working | fast path selected | `apply_rotary` | | `ImportError` (ABI mismatch) | warn once + fall back | fell back to `None`, one warning | | `ModuleNotFoundError` (absent) | **silent** + fall back | fell back to `None`, no warning | | repeated construction | no log spam | deduplicated by `warning_once` | The third row is the behaviour #42679 deliberately preserved; it is unchanged here. **2. Original failure** — reproduced on this machine with a genuinely ABI-mismatched flash-attn build (2.8.3.post1 compiled against an earlier torch). On unpatched code `vllm serve` aborts during model loading with the `undefined symbol` traceback above and never becomes ready. With the guard widened, the engine starts and serves normally through the native rotary path — verified with a real workload (Qwen3.6-27B-FP8 + DFlash speculative decoding, `--max-num-seqs 3`, `--max-model-len 112000`): 122.7 tok/s and 7.91 ms TPOT single stream, 230.5 tok/s aggregate at concurrency 3. **3. End-to-end with this exact commit** — after rebuilding flash-attn against the current torch, `vllm serve` starts cleanly, `/health` returns 200, no warning is emitted (fast path taken, as expected), and generation is correct with speculative decoding active (accept rate 18.2% at `num_speculative_tokens=16`). No numerical change: this only selects which rotary implementation is used when flash-attn cannot load, and the fallback is the same native path used on every machine without flash-attn. **4. Lint** ```bash pre-commit run --files vllm/model_executor/layers/rotary_embedding/common.py ``` All hooks pass (ruff check, ruff format, typos, mypy 3.10, SPDX, forbidden imports). ## AI assistance disclosure AI assistance (Claude Code) was used to diagnose the failure, confirm the exception hierarchy and the `nm`-level symbol analysis, run the verification above, and draft this description. I reviewed every changed line, reproduced both the failure and the fix on my own hardware, and can defend the change end to end.", "url": "https://github.com/vllm-project/vllm/pull/52121", "createdAt": "2026-08-13T07:46:09Z", "updatedAt": "2026-08-13T07:46:45Z", "timestamp": "2026-08-13T07:46:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "bheu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52122", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MiniCPM-V] Fix AssertionError in get_dummy_mm_data when passing VideoDummyOptions to _get_dummy_images", "text": "**Issue**: pytest tests/lora/test_minicpmv_tp.py::test_minicpmv_lora raises AssertionError on non-CUDA platforms (e.g., XPU). **Root Cause**: Commit 9a276d6375 added a runtime assertion to _get_dummy_images in dummy_inputs.py: assert overrides is None or isinstance(overrides, ImageDummyOptions) However, MiniCPMVDummyInputsBuilder.get_dummy_mm_data in minicpmv.py had always been passing video_overrides (type VideoDummyOptions) directly to _get_dummy_images when constructing per-frame dummy images for video. This type mismatch was silently ignored before the assertion was introduced. **Why upstream CI missed it**: test_minicpmv_lora is decorated with @pytest.mark.skipif(current_platform.is_cuda_alike(), ...), so it is skipped entirely on CUDA CI. **Fix**: In minicpmv.py, convert VideoDummyOptions before passing to _get_dummy_images: - Apply num_frames override to num_video_frames - Convert width/height into an ImageDummyOptions instance", "url": "https://github.com/vllm-project/vllm/pull/52122", "createdAt": "2026-08-13T08:06:44Z", "updatedAt": "2026-08-13T15:44:04Z", "timestamp": "2026-08-13T15:44:04Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "mayuyuace", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52123", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Update CODEOWNERS", "text": "## Purpose As discussed offline with @Isotr0py ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52123", "createdAt": "2026-08-13T08:21:34Z", "updatedAt": "2026-08-13T08:44:34Z", "timestamp": "2026-08-13T08:44:34Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "ci/build" ], "author": "DarkLight1337", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52124", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Renderer] Shutdown the renderer properly.", "text": "## Purpose Currently, during online serving, the renderer is shutdown twice, while it is not shutdown during offline inference. ``` offline: python examples/basic/offline_inference/embed.py this pr: INFO 08-13 16:11:10 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1340] [shutdown] EngineCore: exiting busy loop INFO 08-13 16:11:11 [base.py:305] [shutdown] BaseRenderer main: INFO 08-13 16:19:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop online: vllm serve intfloat/e5-small this pr: (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=9032) INFO 08-13 22:56:06 [contextlib.py:478] [shutdown] BaseRenderer (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=9032) INFO 08-13 22:56:06 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=9032) WARNING 08-13 22:56:06 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=9032) WARNING 08-13 22:56:06 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 9092 (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:141] [shutdown] API server: engine client stopped (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:144] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:165] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=9032) INFO: Shutting down (APIServer pid=9032) INFO: Waiting for application shutdown. (APIServer pid=9032) INFO: Application shutdown complete. main: (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=20533) INFO 08-13 16:21:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 20593 (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:139] [shutdown] API server: engine client stopped (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:142] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:163] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO: Shutting down (APIServer pid=20533) INFO: Waiting for application shutdown. (APIServer pid=20533) INFO: Application shutdown complete. ``` ## Best practices for shutdown? Using ExitStack and weakref.finalize provides a nearly perfect shutdown method, and running it a second time will have no effect. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52124", "createdAt": "2026-08-13T08:22:32Z", "updatedAt": "2026-08-13T15:25:08Z", "timestamp": "2026-08-13T15:25:08Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend" ], "author": "noooop", "state": "open", "assignees": [ "sangstar" ] }, { "id": "github:vllm-project/vllm:pull_request:52126", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: prevent PyNvVideoCodec decoder slot limit bypass via ClassVar shadowing", "text": "Move mutable decoder-pool state (_decoder_slots, _active_decoder_slots, _decoder_slot_cond, _max_decoder_slots) out of ClassVar attributes on PyNvVideoCodecVideoBackendMixin into a module-level _PyNvDecoderPool singleton. This prevents subclass augmented assignment (cls._active += 1) from creating independent shadow counters per concrete class, which allowed multiple subclasses to independently exceed the configured process-wide hw_decoders limit.", "url": "https://github.com/vllm-project/vllm/pull/52126", "createdAt": "2026-08-13T08:29:00Z", "updatedAt": "2026-08-13T08:33:06Z", "timestamp": "2026-08-13T08:33:06Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "multi-modality" ], "author": "jperezdealgaba", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52127", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build][CPU] Shrink triton-cpu-build layer by dropping build artifacts", "text": "## Summary - The `vllm-triton-cpu-build` stage in `docker/Dockerfile.cpu` clones and builds `triton-lang/triton-cpu`, but only the resulting wheel is needed by later stages/the final image. - The cloned `triton-cpu` source/build tree and Triton's downloaded LLVM/MLIR toolchain (under `/root/.triton`) were being retained in the final layer, growing the stage to 3.46GB. - This PR removes the `triton-cpu` source tree after the wheel is built, and moves `/root/.triton` into a cache mount so it never lands in the image layer (while still persisting across builds for faster rebuilds). - As a side effect, this also addresses why this stage in CI (e.g. https://buildkite.com/vllm/ci/builds/83692) can look \"not cached\": BuildKit's remote cache import is lazy — a cached step's actual layer blobs are still pulled on demand by downstream stages that bind to it, so a large cached layer can still take a long time to materialize even though the step itself reports `CACHED`. ## Test plan - [x] `docker build --target vllm-triton-cpu-build -f docker/Dockerfile.cpu .` on both base branch and this branch; compared final stage image sizes via `docker history`/`docker save`: **3.46GB → 1.44GB**. - [x] `docker run --rm ls -la /vllm-workspace/dist` on the new image: wheel is present and unchanged; `triton-cpu` source dir and `/root/.triton` are both absent from the layer. - [x] Confirmed via `git diff` that no other stage is touched.", "url": "https://github.com/vllm-project/vllm/pull/52127", "createdAt": "2026-08-13T08:34:50Z", "updatedAt": "2026-08-13T09:17:05Z", "timestamp": "2026-08-13T09:17:05Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "cpu" ], "author": "bigPYJ1151", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52128", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Include MM limits predicate in model compute_hash to prevent compile-cache poisoning", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52128", "createdAt": "2026-08-13T08:35:41Z", "updatedAt": "2026-08-13T08:57:18Z", "timestamp": "2026-08-13T08:57:18Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "vschandramourya", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52129", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Demote xgrammar accept_tokens rejection log to DEBUG (spec decode replay spam)", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52129", "createdAt": "2026-08-13T08:36:00Z", "updatedAt": "2026-08-13T08:56:46Z", "timestamp": "2026-08-13T08:56:46Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "structured-output" ], "author": "vschandramourya", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52130", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Don't use the live processor cache for V1 multimodal profiling", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52130", "createdAt": "2026-08-13T08:36:20Z", "updatedAt": "2026-08-13T08:56:24Z", "timestamp": "2026-08-13T08:56:24Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "vschandramourya", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52131", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Move api_server.py out openai folder", "text": "## Purpose Following https://github.com/vllm-project/vllm/pull/41907 > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. We'd better move api_server.py out openai folder ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52131", "createdAt": "2026-08-13T08:39:15Z", "updatedAt": "2026-08-13T15:12:24Z", "timestamp": "2026-08-13T15:12:24Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "documentation", "performance", "frontend", "multi-modality" ], "author": "noooop", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52132", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Enhance engine snapshot management and API lifecycle", "text": "## Purpose Add an opt-in Engine Snapshot deep-sleep path that keeps the API frontend, HTTP listener, and launcher alive while snapshotting and exiting the `EngineCoreProc` subtree. On wake, vLLM validates the snapshot and runtime environment, restores the EngineCore and CUDA state, adopts the restored process, reconnects the frontend, and reopens request admission only after health and semantic checks pass. Design, roadmap, compatibility constraints, and complete experimental results are documented in [RFC #52125: vLLM Native Reusable Engine Snapshot Sleep (L3)](https://github.com/vllm-project/vllm/issues/52125). This work overlaps with #51360 in its use of CRIU and CUDA Checkpoint/Restore for an initialized engine. The snapshot boundary and user-facing lifecycle differ: - #51360 captures the complete initialized service process tree before the HTTP listener is bound and restores it during instance startup. - This PR implements runtime deep sleep. The frontend and HTTP listener remain available for status and wake requests while only the EngineCore subtree is captured, terminated, restored, and reattached. ## Change Summary - Add the Engine Snapshot lifecycle: `READY → DRAINING → PREPARING → SNAPSHOTTING → HIBERNATED` and `HIBERNATED → RESTORING → ATTACHING → VERIFYING → READY`. - Add idle-only request admission control and readiness handling around snapshot transitions. - Add EngineCore detach, expected-exit handling, restored-process adoption, and frontend reconnection. - Add `fake` and `criu_cuda` snapshot providers. - Add atomic snapshot artifact commit, manifest validation, diagnostics, rollback, and cleanup. - Expose two L3 resource policies: - `full`: preserve weights, KV cache, and runtime state in the CUDA image. - `minimized`: discard weights and KV cache before capture, then reload weights from model files and rebuild KV cache after restore. - Add development-mode CLI configuration and the Level 3 sleep, wake, and status endpoints. - Add allocator, worker, process-lifecycle, provider, manager, API, middleware, CLI, and serving tests. - Document prerequisites, supported scope, endpoints, snapshot contents, and deployment restrictions. ## Current Scope The initial implementation supports: - Same-machine restore. - One host and one GPU. - `TP=1`, `PP=1`, and `DP=1`. - One Python API server using `UniProcExecutor`. - Idle-only capture with no in-flight request preservation. - `fake` and CRIU/CUDA providers. - Development-mode enablement through `VLLM_SERVER_DEV_MODE=1`. The feature is opt-in and does not change the default `vllm serve` process topology or request path. Cross-machine restore, 1→N restore, multi-GPU snapshot groups, distributed topologies, and in-flight request preservation remain follow-up work described in the RFC. ## Test Plan - Run Ruff on the changed Python files. - Run Python 3.10 mypy on the changed typed modules. - Run the snapshot unit-test suite: `pytest -q tests/snapshot` - Run the Engine Snapshot CLI parser tests: `pytest -q tests/entrypoints/openai/test_cli_args.py -k engine_snapshot` - Exercise the lifecycle and failure paths with the `fake` provider. - Run repeated real CRIU/CUDA capture and restore cycles for the `full` and `minimized` policies. - Verify restored inference output, process cleanup, physical HBM release, snapshot artifacts, and phase-level timing. ## Test Result Test environment: - NVIDIA H20, single GPU. - Qwen3-8B BF16. - `TP=1`, `PP=1`, `DP=1`, one API server. - Non-eager execution. - Model, compilation cache, CRIU images, and snapshots stored on tmpfs. - Six runs per group: one warmup followed by five measured runs. - `minimized` was repeated across three groups for 15 measured runs. Median results: | L3 policy | Measured runs | Checkpoint time | Restore time | Artifact size | | --- | ---: | ---: | ---: | ---: | | `minimized` | 15 | 2.837 s | 5.948 s | 4.251 GiB | | `full` | 5 | 27.242 s | 22.990 s | 49.659 GiB | Restore time is measured from the `/wake_up` request to the first non-empty token from a fixed streaming request, with the final response content also checked. Correctness and resource results: - All reported `full` and `minimized` runs restored successfully and passed the fixed semantic checks. - Every pre-restore model page-residency check reported 100%. - Engine `read_bytes` delta was 0 in all reported runs. - Every reported `HIBERNATED` check showed 0 MiB for both project-process HBM and full-GPU HBM. - Ruff passed. - Python 3.10 mypy passed. - Snapshot tests: `75 passed`. - CLI parser tests: `3 passed, 30 deselected`. These validation results predate the final public-policy simplification commit and are not presented as a rerun of the current tree. ## AI Assistance AI assistance was used for implementation, testing, analysis, and drafting. The submitter reviewed the changes and owns the final submission.", "url": "https://github.com/vllm-project/vllm/pull/52132", "createdAt": "2026-08-13T08:52:39Z", "updatedAt": "2026-08-13T14:43:13Z", "timestamp": "2026-08-13T14:43:13Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "frontend" ], "author": "fuhefei", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52133", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend] Add Hunyuan A13B tool parser", "text": "## Purpose The Rust frontend exposes parser selection and automatic model matching, but it did not provide the Python frontend's `hunyuan_a13b` tool parser. Serving `tencent/Hunyuan-A13B-Instruct` through the Rust frontend therefore could not parse the model's `[...]` JSON-array format into OpenAI tool-call deltas. This PR adds a Hunyuan A13B parser using the existing shared incremental JSON tool-call core. It supports parallel calls, empty call arrays, nested argument objects, non-ASCII text, arbitrary streaming chunk boundaries, whitespace between `` and the JSON array, and argument properties named `name` without treating them as new call headers. To match the Python parser, it also strips the leading chat-template `助手：` prefix from natural-language responses and discards text after a completed tool-call block, including an empty block. The parser is registered under the Python-compatible `hunyuan_a13b` name, with model-name matching for the hyphenated and underscored Hunyuan A13B forms. This contributes to the Rust frontend parity roadmap in #44280. Duplicate-work check: I reviewed #44280's claims and searched current open PRs for `hunyuan rust`, `hunyuan_a13b`, `Hunyuan A13B Rust Frontend`, `HunyuanA13BToolParser`, and `tool_calls JSON array Rust Frontend`. No open PR adds this parser. Related PRs cover HY-V3 reasoning (#48800), Python Hunyuan parser fixes (#49535 and #47954), and a Python Hermes parser-engine migration (#51937), so they do not overlap this Rust implementation. AI assistance was used to research, implement, test, and review this change. The commit includes the required attribution trailer. This is intentionally a draft so the account owner can complete the final human review before marking it ready. ## Test Plan ```bash cargo fmt --all --check cargo nextest run -p vllm-parser cargo nextest run -p vllm-chat factory_new_resolves_default_patterns cargo nextest run -p vllm-chat validate_parser_overrides_rejects_unknown_tool_parser cargo clippy -p vllm-parser -p vllm-chat --all-targets -- -D warnings git diff --check origin/main...HEAD ``` The parser unit tests cover: - ordinary text without a tool call; - chat-template prefix removal across streaming chunk boundaries; - preservation of a non-leading `助手：` occurrence in ordinary content; - parallel calls and deeply nested arguments; - prefix text and non-ASCII arguments; - whitespace and newlines before the JSON array; - empty arrays in complete and character-by-character streaming input; - discarded suffix text after complete and empty tool-call blocks; - rejection of an invalid trailing comma; - arbitrary streaming chunk boundaries; - nested argument keys named `name`; - a truncated tool call failing closed. ## Test Result - `cargo fmt --all --check`: passed - Full `vllm-parser` nextest suite: 431 passed (including 15 Hunyuan tests) - factory model-pattern test: 1 passed - unknown-parser registry snapshot test: 1 passed - Clippy with warnings denied for `vllm-parser` and `vllm-chat`: passed - `git diff --check`: passed Clippy emitted only a dependency-level future-incompatibility notice for `proc-macro-error2`; it emitted no lint failure for this change. An additional full `vllm-chat` run passed 268 tests before eight network-transport integration tests hit the local sandbox's `Operation not permitted` restriction and fail-fast cancelled the remainder; the two chat tests affected by this change passed independently. No model evaluation is required because this adds frontend parsing parity and does not change model inference or generated tokens. ---
Essential Elements of an Effective PR Description Checklist - [x] Purpose and related roadmap are documented. - [x] Exact test commands are provided. - [x] Test results are reported. - [x] No model documentation update is required; this adds a frontend parser for an already supported model/parser format.
", "url": "https://github.com/vllm-project/vllm/pull/52133", "createdAt": "2026-08-13T08:56:42Z", "updatedAt": "2026-08-13T09:23:29Z", "timestamp": "2026-08-13T09:23:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rust" ], "author": "Sunbeam23333", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Fix `WhisperEncoderLayer.forward` docstring in `dots3_note`", "text": "## Purpose The docs build emits griffe warnings for `vllm/models/dots3_note/nvidia/audio_encoder.py`: ``` WARNING - griffe: audio_encoder.py:351: Parameter 'attention_mask' does not appear in the function signature WARNING - griffe: audio_encoder.py:353: Parameter 'layer_head_mask' does not appear in the function signature ``` The `WhisperEncoderLayer.forward` docstring was inherited from the upstream HF Whisper implementation and never updated for this layer's signature, which takes packed variable-length inputs (`cu_seqlens_*`, `max_seqlen_*`) and rotary embeddings instead of `attention_mask`/`layer_head_mask`. This documents the parameters the method actually takes. It also corrects the return annotation, which claimed `torch.Tensor` while the method returns `tuple[Any, ...]` (hidden states, plus attention weights when `output_attentions=True`). Docstring only, plus one annotation. No runtime behaviour change. ## Not a duplicate - `gh pr list --state open --search \"griffe docstring parameter\"` returns only #51342, which fixes an unrelated annotation in `vllm/benchmarks/throughput.py`. - `gh pr list --state open --search \"dots3_note audio_encoder\"` returns nothing. ## Test Plan Run griffe's Google-style docstring parser over the module and check for warnings. ## Test Result Before: ``` WARNING: module.py:351: Parameter 'attention_mask' does not appear in the function signature WARNING: module.py:353: Parameter 'layer_head_mask' does not appear in the function signature ``` After: no warnings. `pre-commit run --files vllm/models/dots3_note/nvidia/audio_encoder.py` passes, including `mypy` and `ruff`. No model evaluation is included because the change touches no executed code. ## Note AI assistance (Claude Code) was used for this change. I have reviewed every changed line. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52134", "createdAt": "2026-08-13T08:58:58Z", "updatedAt": "2026-08-13T09:33:25Z", "timestamp": "2026-08-13T09:33:25Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ready", "build-docs" ], "author": "hmellor", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52135", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(missing_documentation): Class `RequestOutputKind` in vllm/sampling_params.py has no", "text": "## TLDR **Gap:** Class `RequestOutputKind` in vllm/sampling_params.py has no docstring — add parameter docs, return type, and example [BLOCKED:Tests failed after retry: ==================================== ERRORS ==================================== ___________________ ERROR collecting vllm/sampling_params.py _____ **Wedge type:** `missing_documentation` **Issue:** https://github.com/vllm-project/vllm/blob/main/vllm/sampling_params.py#L182 ## Changes - `vllm/sampling_params.py` **Diff size:** 43 lines across 1 file(s) ## Pre-submission checklist - [x] Minimal change — touches at most 3 files - [x] Tests updated (if test suite present) - [x] No CI/CD, Dockerfile, or lock file modifications - [x] Diff reviewed for secrets ## AI Assistance Disclosure This contribution was AI-assisted using Hermes Agent (Nous Research). Co-authored-by: Hermes Agent ", "url": "https://github.com/vllm-project/vllm/pull/52135", "createdAt": "2026-08-13T09:07:35Z", "updatedAt": "2026-08-13T09:07:45Z", "timestamp": "2026-08-13T09:07:45Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "nickhac", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52136", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add `pydocstyle` to the `ruff` rules", "text": "Enables the `D` ruleset in ruff and fixes the resulting violations across the codebase. ### Config `D100`–`D107` are ignored, so this does **not** require a docstring on anything that lacks one today. Six more rules are ignored because their remaining violations were either impossible to autofix or actively wrong to autofix: | Rule | Why skipped | |---|---| | `D205`, `D400`, `D415` | All treat a summary sentence that merely *wraps* as summary-plus-body. The autofix appends periods to list items. | | `D401`, `D404` | Flag idiomatic noun-phrase docstrings on properties and attributes. | | `D301` | Churn-only (`r\"\"\"` prefixes). | Plus `ignore-var-parameters = true` (no `*args`/`**kwargs` descriptions required) and a `tests/**` ignore for `D417`, since test params are mostly pytest fixtures. ### Changes The bulk is autofix output for the mechanical rules (`D209`, `D212`, `D200`, `D202`, `D413`, and friends). On top of that, by hand: - **~290 `Args:` entries** written across 88 files for `D417`, mostly in `fused_moe`, attention backends, and multimodal models. Several were format bugs rather than missing docs: numpy-style `Parameters`/`----------` blocks that ruff's Google parser can't read at all. - **807 trailing periods** for `D400`/`D415`, applied only where `D205` doesn't co-fire, so no list items were punctuated. - **37 `E501` regressions** fixed. These were introduced by the `D212` autofix collapsing wrapped summaries onto line 1 and pushing them past 88 chars. - A handful of one-offs: `D210`, `D402`, `D418`, `D419`. Incidental fixes picked up along the way: typos (`videoand`, `output_dytpe`, `the m odel`) and stale docs naming parameters that no longer exist.", "url": "https://github.com/vllm-project/vllm/pull/52136", "createdAt": "2026-08-13T09:16:21Z", "updatedAt": "2026-08-13T17:07:32Z", "timestamp": "2026-08-13T17:07:32Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "build-docs", "kimi", "k3", "cohere" ], "author": "hmellor", "state": "open", "assignees": [ "sangstar" ] }, { "id": "github:vllm-project/vllm:pull_request:52138", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU]bump up vllm_xpu_kernels to 0.1.13.1", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52138", "createdAt": "2026-08-13T09:46:44Z", "updatedAt": "2026-08-13T09:47:29Z", "timestamp": "2026-08-13T09:47:29Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "intel-gpu", "ci/build" ], "author": "jikunshang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52139", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][CI] Give the AITER MLA decode metadata stub its MLA dims", "text": "# Purpose `tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py::test_persistent_decode_metadata_matches_fp8_golden` fails on main with `AttributeError: 'types.SimpleNamespace' object has no attribute 'q_lora_rank'`. Two jobs report it, the dedicated AITER MLA job and the sharded `kernels/attention` job, but it is the same test. \"[Model] Add native Dots3 NOTE multimodal support\" (#51255), changed `MLACommonMetadataBuilder.__init__` to read the MLA latent dimensions off the layer in `static_forward_context` instead of calling `get_mla_dims(self.model_config)`, so that hybrid MLA models can use different dimensions per KV cache group. The builder now also sizes its chunked-prefill workspace from `kv_lora_rank + qk_rope_head_dim` rather than from `model_config.get_head_size()`. This test stubs that layer with a `SimpleNamespace` holding only `prefill_backend`, so the builder raises on the first dimension it asks for. #51255 updated the same stub in the sibling test `test_rocm_aiter_mla_causal_verify_mask.py` and in the MLA backend tests, but missed this file, which uses the identical pattern. # Test Plan The change adds the five MLA dimensions to the stub. They are taken from `get_mla_dims(vllm_config.model_config)`, which is what the builder itself used before #51255, so the stub cannot drift from the `deepseek-ai/DeepSeek-R1` configuration the rest of the test is built around. The sibling test hardcodes the same values as module constants because it also builds tensors from them; this one does not need them anywhere else. On gfx950 (MI355X), with AITER: ``` pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py \\ tests/kernels/attention/test_rocm_aiter_mla_causal_verify_mask.py ``` The repository was also searched for any other test stubbing a layer the same way; these two are the only ones. # Test Result Before the change the decode metadata test fails with the `AttributeError` above. After it, both tests pass. The assertion the test exists for, that the builder's persistent decode metadata matches the golden recomputed with explicit fp8 dtypes, is reached and holds; nothing about the coverage changes. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52139", "createdAt": "2026-08-13T09:47:40Z", "updatedAt": "2026-08-13T17:06:16Z", "timestamp": "2026-08-13T17:06:16Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "stefankoncarevic", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52140", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Quantization] Add Ling 3.0 Flash FP4 support", "text": "## Purpose Add loading and serving support for [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4). Ling FP4 uses mixed FP8/MXFP4 quantization metadata: ```json { \"quant_method\": \"fp8\", \"routed_experts_quant_method\": \"mxfp4\" } ``` This PR: - adds a model-specific configuration hook that keeps `Fp8Config` while selecting MXFP4 storage for routed experts; - maps routed-expert `*_weight_scale_inv` checkpoint names to the internal MXFP4 `*_weight_scale` parameters; - applies the loading behavior to both the main and MTP models. The change is gated on Ling's FP4 metadata, leaving BF16, FP8, and INT4 checkpoints unchanged. No open PR currently adds Ling FP4 checkpoint loading. ## Test Plan Run the full in-tree GSM8K evaluation in no-thinking mode. The evaluator uses the raw completions endpoint without applying Ling's chat template or enabling native thinking: ```bash TMPDIR=/path/to/gsm8k PYTHONPATH=. .venv/bin/python -c ' from tests.evals.gsm8k.gsm8k_eval import evaluate_gsm8k print(evaluate_gsm8k( port=8001, num_questions=1319, num_shots=5, max_tokens=256, temperature=0, seed=42, use_chat_completions=False, max_concurrency=16, request_timeout_seconds=3600, )) ' ``` ## Test Result NVIDIA GB10, CUDA 13.0, PyTorch 2.13.0+cu130: | Checkpoint | Mode | Accuracy | Invalid responses | Output tokens/s | | --- | --- | ---: | ---: | ---: | | Ling 3.0 Flash FP4 | No thinking | 84.08% | 0.00% | 102.75 | | Ling 3.0 Flash INT4 | No thinking | 86.28% | 0.00% | 109.67 | Both runs used the same vLLM revision, 1,319 questions, 5-shot prompts, temperature 0, `max_tokens=256`, concurrency 16, and the raw completions endpoint. Ling's native thinking mode was not enabled. ## AI assistance disclosure OpenAI Codex assisted with investigation, implementation, deployment, testing, evaluation analysis, and drafting this description. The human submitter reviewed the changes and understands the implementation.", "url": "https://github.com/vllm-project/vllm/pull/52140", "createdAt": "2026-08-13T09:48:47Z", "updatedAt": "2026-08-13T09:58:58Z", "timestamp": "2026-08-13T09:58:58Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "FENP", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52141", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Observability] add model initializing duration log", "text": "## Purpose Add model initializing duration log ## Test Plan NA ## Test Result NA ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52141", "createdAt": "2026-08-13T09:54:03Z", "updatedAt": "2026-08-13T09:55:29Z", "timestamp": "2026-08-13T09:55:29Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "andyxning", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52142", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix standalone torch.compile cache loading after relocation", "text": "## Purpose The standalone Inductor compile cache persists artifact handles as `(key, absolute_path)`. vLLM supports reusing its `torch.compile` cache after copying it to another machine or baking it into a container image. If the cache directory changes location in that process, the artifact exists under the new cache directory, but `InductorStandaloneAdaptor.load()` still uses the persisted absolute path. If the original location is unavailable, loading fails with `FileNotFoundError`. This change reconstructs the artifact path from the adaptor's current `self.cache_dir` and the persisted artifact key (`handle[0]`) instead of loading from the stale path in `handle[1]`. The persisted handle format remains unchanged, so this fix does not require changing existing cache metadata. Fixes #52154. ## Test Plan Added a regression test covering both standalone artifact save formats: - `binary` - `unpacked` The test initializes the adaptor with a relocated cache directory while using a handle that contains the old absolute path, and verifies that `CompiledArtifact.load()` resolves the artifact from the current cache directory. ```bash python -m pytest -q \\ --confcutdir=tests/compile \\ tests/compile/test_compiler_interface.py ``` ## Test Result Before this change: ```text 2 failed Expected: .../new/artifact_shape_None_subgraph_0 Actual: .../old/artifact_shape_None_subgraph_0 ``` With this change: ```text 2 passed ``` Also verified with `git diff --check` and pre-commit on the modified files.", "url": "https://github.com/vllm-project/vllm/pull/52142", "createdAt": "2026-08-13T09:55:15Z", "updatedAt": "2026-08-13T10:54:12Z", "timestamp": "2026-08-13T10:54:12Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "jungjiyu", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52143", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Inkling structured output stop tokens", "text": " ## Purpose Fixes #51693. Inkling can emit structural response tokens after completing a JSON-schema structured output. The reported failure token `200028` is `<|begin_of_text|>` in the public `thinkingmachines/Inkling-Small-NVFP4` tokenizer metadata. Because these Inkling structural tokens are not declared through generation config, xgrammar did not treat them as stop-token overrides and could reject them after the JSON payload was complete. This PR resolves that path by: - Adding tokenizer-resolved Inkling structural terminal tokens to the structured-output grammar stop-token override set when the Inkling reasoning parser is active. - Leaving `SamplingParams.stop_token_ids` unchanged, so Inkling reasoning block terminators such as `<|end_message|>` do not prematurely stop the request before structured output begins. - Marking the request stopped once the structured-output grammar terminates after accepting one of its stop overrides. Duplicate-work checks run before opening this PR: - `gh issue view 51693 --repo vllm-project/vllm --comments`: issue open, unassigned, no comments. - `gh pr list --repo vllm-project/vllm --state open --search \"51693 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"Inkling structured output stop tokens grammar rejected\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched files. - Focused unit tests for Inkling structural stop-token injection and scheduler stop-on-grammar-termination behavior. - Ruff format and ruff check on touched files. No model evals were run; this change affects request stopping / structured-output grammar handling, not model quality or accuracy. ## Test Result - `./.venv/bin/python -m py_compile vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/python -c \"... StructuredOutputManager._create_grammar Inkling stop-token check ...\"`: passed; grammar compile received `{199999, 200006, 200010, 200028}` while `SamplingParams.all_stop_token_ids` and `SamplingParams.stop_token_ids` remained `[199999]`. - `./.venv/bin/pre-commit run ruff-format --files vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/python -m pytest tests/v1/structured_output/test_reasoning_structured_output.py -k 'inkling_structural_tokens_are_stop_overrides' -v`: selected test reported `PASSED`, then local macOS test process exited with code 139 during global teardown in `torch.accelerator.memory.empty_host_cache()` via `tests/conftest.py::cleanup_fixture`. - `./.venv/bin/python -m pytest tests/v1/core/test_scheduler.py -k 'stop_request_when_structured_output_fsm_terminates' -v`: selected test reported `PASSED`, then local macOS test process exited with code 139 during the same global teardown path. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/52143", "createdAt": "2026-08-13T10:00:35Z", "updatedAt": "2026-08-13T10:15:40Z", "timestamp": "2026-08-13T10:15:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "structured-output" ], "author": "danielpolimac", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52144", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] Add pause/resume E2E tests", "text": "## Purpose One part of the RL CI Matrix for vLLM proposed by #45585. This PR adds end-to-end tests for the pause/resume API used in RL rollout, covering: 1. is_paused state & idempotency: repeated pause/resume, multiple cycles 2. Invalid mode behavior — return 400; state unchanged 3. Behavior of keep/wait/abort on in-flight and new requests, and behavior after resume 4. clear_cache=true/false behavior — golden output comparison ## Test Plan ### Test Matrix Priority | Test Case | Main Coverage -- | -- | -- P0 | test_pause_resume_state_is_idempotent_across_cycles | is_paused, repeated calls, multiple cycles P0 | test_invalid_pause_mode_preserves_state | Status code and unchanged state for illegal mode P0 | test_pause_mode_request_lifecycle[mode] | abort/wait/keep for in-flight, new requests, and resume P1 | test_clear_cache_preserves_output_and_controls_prefix_cache | Golden output and cache retention/clearing ### Request Lifecycle Across the Three Modes Expected behavior matrix: | mode | in-flight request at pause return | new request submitted after pause | after resume | | ----- | ------------------------------------------------------------ | --------------------------------- | ----------------------------------------------------------- | | abort | already finished, `finish_reason == \"abort\"` | accepted but stays pending | new request completes normally; old request does not resume | | wait | already finished normally, typically `finish_reason == \"length\"` | stays pending | new request completes normally | | keep | not yet finished, generation progress frozen | stays pending | both old and new requests complete normally | ### Speed & Stability Use a class/module-scoped `server_url` fixture to start the server once and reuse it across all tests, calling `resume()` before and after each test to restore state. The cache test should perform its own `clear_cache=True` at the start to avoid cache pollution from shared server state. abort/wait/keep and multi-request behavior already have fairly comprehensive coverage in lower-level tests. The HTTP E2E layer just needs to preserve the key mappings above. ## Test Result H100 single card: ```sh pytest --durations=0 -sv test_pause_resume.py =============================== test session starts =============================== platform linux -- Python 3.11.15, pytest-9.1.1, pluggy-1.6.0 -- /opt/miniconda/envs/y00806874/bin/python3.11 cachedir: .pytest_cache rootdir: /home/c00944022/tests plugins: typeguard-4.5.2, anyio-4.14.1 collected 6 items test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles PASSED test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] PASSED test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache PASSED ================================ slowest durations ================================ 36.12s setup test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles 10.06s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] 5.56s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 1.02s teardown test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache 0.73s call test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache 0.64s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.60s call test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles 0.10s call test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 0.01s teardown test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.01s teardown test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.01s teardown test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s setup test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s teardown test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles (2 durations < 0.005s hidden. Use -vv to show these durations.) =============================== 6 passed in 54.96s ================================ ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52144", "createdAt": "2026-08-13T10:03:59Z", "updatedAt": "2026-08-13T12:09:50Z", "timestamp": "2026-08-13T12:09:50Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "floatlibai", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52145", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add missing return type annotations in outputs.py", "text": "### Purpose Add missing return type annotations to from_base() static methods and PoolingRequestOutput.__repr__() in vllm/outputs.py. All from_base() static methods on EmbeddingOutput, ClassificationOutput, ScoringOutput, EmbeddingRequestOutput, ClassificationRequestOutput, and ScoringRequestOutput lacked return type annotations. PoolingRequestOutput.__repr__() was missing -> str, making it the only __repr__ in the file without one — every other __repr__ in the same file already has it. No logic changes. ### Test Plan pre-commit run ruff-check --files vllm/outputs.py pre-commit run mypy-3.10 --files vllm/outputs.py --hook-stage manual ### Test Result ruff check...........Passed Run mypy for Python 3.10...........Passed", "url": "https://github.com/vllm-project/vllm/pull/52145", "createdAt": "2026-08-13T10:04:56Z", "updatedAt": "2026-08-13T12:58:46Z", "timestamp": "2026-08-13T12:58:46Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "verified" ], "author": "vineetatiwari27", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52147", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Standardise weight tying on `ParallelLMHead.tie_weights`", "text": "## Purpose vLLM expresses tied word embeddings in three different ways. Only one of them, `self.lm_head = self.lm_head.tie_weights(embed_tokens)`, dispatches through `quant_method.tie_weights`. This PR converts the other two so tying is expressed one way everywhere. - `self.lm_head.weight = embed_tokens.weight` (33 sites) bypassed the quant method entirely, so it was wrong for quant methods that repack. The `ParallelLMHead` already exists at that point, so this is a one-line swap with no structural change. - `self.lm_head = embed_tokens` (23 sites) built no `ParallelLMHead` at all in the tied branch. These now always build one and tie it. Two consequences of the second bullet are worth review: 1. A transient vocab x hidden allocation before the head is tied. 2. The quant config is now consulted for the `lm_head` prefix where it previously never was. Five models hardcode tying with `assert config.tie_word_embeddings` (`gemma`, `gemma2`, `commandr`, `cohere2_moe`, `mpt`) and have no real `lm_head` to build. They are left alone here. ## Relationship to #51665 This is not a duplicate. It is the tying-standardisation half of my own #51665, split out so it can be reviewed on its own. Everything else in that PR is deliberately excluded: - `ModelConfig.maybe_untie_word_embeddings` and the checkpoint-metadata plumbing. - The `AutoWeightsLoader` aliased-parameter skip and the ~50 hand-rolled `skip_prefixes=[\"lm_head.\"]` removals that depend on it. **Every existing skip is left untouched here**, so weight-loading behaviour is unchanged. - The `assert config.tie_word_embeddings` removals, which only exist because #51665 can untie at config time. I searched open PRs for `tie_weights`, `tie_word_embeddings` and `weight tying`; nothing else overlaps. ## Test Plan Structural change with no new behaviour, so this leans on construction and load-path checks plus existing suites. 1. Construct tied models on CPU and assert `lm_head` is a `ParallelLMHead` whose `weight` is the same `nn.Parameter` object as the embedding. Covers both conversion forms. 2. Load a real checkpoint through `load_weights` and confirm the tied head ends up populated. 3. `pre-commit run --all-files` on the changed files (ruff, ruff-format, mypy). ## Test Result Models constructed on CPU (Bloom, OPT and Mamba are all conversions of the second, structural form): ``` Qwen/Qwen3-0.6B: tie=True ParallelLMHead=True shared=True -> OK Qwen/Qwen2-0.5B: tie=True ParallelLMHead=True shared=True -> OK facebook/opt-125m: tie=True ParallelLMHead=True shared=True -> OK state-spaces/mamba-130m-hf: tie=True ParallelLMHead=True shared=True -> OK bigscience/bloom-560m: tie=True ParallelLMHead=True shared=True -> OK ``` End-to-end `load_weights` with the real `Qwen/Qwen3-0.6B` checkpoint: ``` checkpoint has lm_head.weight: True num loaded params: 226 lm_head.weight in loaded: False # still skipped by the existing skip_prefixes shared storage: True matches checkpoint: True head is nonzero: True ``` Existing suite touching the quantised tied-head path: ``` $ pytest tests/model_executor/test_qwen3_5_quantization.py -q 2 passed, 14 warnings in 0.65s ``` Linters: ``` $ pre-commit run --all-files # on the changed files ruff check ... Passed ruff format ... Passed Run mypy for Python 3.10 ... Passed ``` No model evals yet, they need a GPU box. Happy to run whatever the reviewers consider load-bearing here. ## Note for reviewers In `ernie45_moe.py` and `ernie45_vl_moe.py` the tie runs outside the `is_last_rank` check, so with PP > 1 `self.lm_head` is a `PPMissingLayer`, which has no `tie_weights`. That path already raised `AttributeError` on `PPMissingLayer.weight` before this change, so it is not a regression, but the failure mode changes. Worth fixing separately. --- AI assistance was used to produce this change. I have reviewed every changed line and stand behind it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52147", "createdAt": "2026-08-13T10:12:28Z", "updatedAt": "2026-08-13T15:10:11Z", "timestamp": "2026-08-13T15:10:11Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "qwen", "mistral" ], "author": "hmellor", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52148", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Fix FlashInfer SM12x prefill with sinks", "text": "## Summary Use FlashInfer's sink-aware paged prefill wrapper on SM12x when attention sinks are enabled. The generic FA2 prefill path accepts a `sinks` argument but does not apply it, so #49718 can use XQA for decode while producing incorrect prefill output. The wrapper is specialized with the active dtypes, head dimensions, sliding window, and softmax scale. DCP, NVFP4, SM90/SM100, and sink-free paths are unchanged. This is not a duplicate of the automatic revert #51987: that PR removes SM12x XQA support, while this change preserves XQA and fixes the prefill path for sink-attention models on SM12x . ## Validation - ` pytest tests/v1/attention/test_attention_backends.py::test_flashinfer_native_prefill_with_sinks -q` passing - SM12x GPT-OSS model eval fixed and passing - Nemotron and Qwen models both working as expected", "url": "https://github.com/vllm-project/vllm/pull/52148", "createdAt": "2026-08-13T10:33:31Z", "updatedAt": "2026-08-13T15:44:01Z", "timestamp": "2026-08-13T15:44:01Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "ready", "nvidia" ], "author": "askliar", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52149", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle persistent top-k candidate overflow", "text": "## Purpose Fixes #51782. `persistent_topk` uses fixed-size candidate buffers in several optimized dispatch paths. When many scores map to the same coarse histogram bin, those buffers can overflow. The counters keep increasing while excess candidates are dropped, so the kernel returns valid, unique indices that are not the true top-k values. This change: - detects candidate-buffer overflow in the short, persistent-decode, persistent-medium, and filtered-large paths; - falls back to a bounded-memory exact FP32 radix rescan of the full row; - leaves the existing fast paths and the 1 MiB workspace contract unchanged; - adds a regression matrix covering all four dispatch paths and `k = 512, 1024, 2048`. The fallback performs four full-row radix passes and is intentionally entered only after an overflow is detected. ## Test Plan 1. Reproduce the bug with the vLLM v0.27.0 installed kernel on narrow score distributions across all persistent-top-k dispatch paths. 2. Compile the patched headers into a CUDA extension against the v0.27.0 image. 3. For each case, compare the selected-value multiset exactly with `torch.topk`, and also validate index range and uniqueness. 4. Run the 87-case matrix with 20 kernel repetitions per case on an NVIDIA B300 (SM 10.3, CUDA 13.0, PyTorch 2.13.0+cu130). ## Test Result | Implementation | Cases | Failing cases | Bad rows | Invalid indices | Duplicate-index rows | | --- | ---: | ---: | ---: | ---: | ---: | | vLLM v0.27.0 baseline | 87 | 65 | 2,339 | 0 | 0 | | Patched | 87 | 0 | 0 | 0 | 0 | The patched run produced zero value mismatches (`rtol=0`, `atol=0`) across all 87 cases. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, including the issue it resolves. - [x] The test plan and environment. - [x] Before/after correctness results. - [x] No documentation update is needed; this is an internal CUDA kernel fix.
", "url": "https://github.com/vllm-project/vllm/pull/52149", "createdAt": "2026-08-13T10:34:44Z", "updatedAt": "2026-08-13T10:35:20Z", "timestamp": "2026-08-13T10:35:20Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug" ], "author": "xijiaat", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52151", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix fused add RMSNorm residual-sum variance", "text": "## Purpose Fixes #52104. The CUDA `vllm_c` fused-add RMSNorm kernel was computing variance from the low-precision rounded `input + residual` value. The native IR implementation keeps `input.float() + residual.float()` in FP32 through variance calculation, then rounds only the published residual output. This PR aligns the vectorized and generic `fused_add_rms_norm` CUDA kernel paths with that native arithmetic contract: - First pass computes variance from the FP32 residual sum without mutating `residual`. - Second pass recomputes the FP32 sum, writes the rounded residual output, and normalizes from the FP32 sum. - Adds a focused `vllm_c` regression test that checks the output is closer to the FP32-sum contract than to the old rounded-sum contract. Duplicate-work checks run before opening this PR: - `gh issue view 52104 --repo vllm-project/vllm --comments`: issue open; one comment suggesting regression coverage. - `gh pr list --repo vllm-project/vllm --state open --search \"52104 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"fused_add_rms_norm residual sum variance BF16 FP32 rounding\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched Python test file. - Focused pytest selection for the new regression test. - Ruff format/check on the touched Python test file. - Clang-format on the touched CUDA file. - Whitespace diff check. No model evals were run; this is a kernel arithmetic consistency fix, not a model-output-quality change. Full CUDA validation should be run in CI because the local host has no GPGPU. ## Test Result - `./.venv/bin/python -m py_compile tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/python -m pytest tests/kernels/ir/test_layernorm.py -k fp32_sum_for_variance -v`: selected 2 tests; both skipped locally because this macOS host has no GPGPU. - `./.venv/bin/pre-commit run ruff-format --files tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/pre-commit run clang-format --files csrc/libtorch_stable/layernorm_kernels.cu`: passed. - `git diff --check`: passed. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52151", "createdAt": "2026-08-13T10:42:29Z", "updatedAt": "2026-08-13T10:42:39Z", "timestamp": "2026-08-13T10:42:39Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "danielpolimac", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52152", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Add shared-disk weight transfer backend", "text": "## Purpose Add an inference-side `disk` weight-transfer backend for deployments where a trainer writes a complete safetensors checkpoint to shared storage and an orchestrator reloads it into a running vLLM engine through the existing start/update/finish control plane. The backend: - accepts one absolute local checkpoint directory per update session, visible at the same path on every inference worker; - loads only the requested primary safetensors checkpoint (not retained secondary weight sources); - preserves the standard model loader's weight mapping, expert-parallel filtering, shard-index validation, and strict missing-weight tracking; - registers lazily in the existing weight-transfer factory and documents the HTTP client flow; - always clears worker/session state when finishing raises, while preserving the original exception. Related use-case context: #48644. This PR does not close that broader request: it is inference-side only and intentionally does not add a trainer data plane, Hub downloads, fastsafetensors/GDS, or transactional rollback. > AI assistance disclosure: this change was researched and implemented with OpenAI Codex assistance. It is opened as a Draft and must receive final human-owner review before being marked ready. ## Test Plan ```bash uv run --no-project --active python -m pytest \\ tests/distributed/test_weight_transfer.py -k disk -q uv run --no-project --active python -m pytest \\ tests/v1/worker/test_gpu_worker_weight_transfer.py \\ -k finish_error_resets_target_and_allows_next_session -q uv run --no-project --active pre-commit run \\ --from-ref origin/main --to-ref HEAD ``` ## Test Result - Shared-disk backend tests: **11 passed** - Worker finish-error cleanup (normal and draft sessions): **2 passed** - All applicable pre-commit hooks passed, including Ruff, formatting, typos, Markdown, mypy, SPDX, lazy-import, and configuration validation. - Re-run after rebasing onto the then-current upstream `main`. ---
Essential Elements of an Effective PR Description Checklist - [x] Purpose and related context are documented. - [x] Reproducible test commands are included. - [x] Test results are included. - [x] Backend documentation and an end-to-end client example are included.
", "url": "https://github.com/vllm-project/vllm/pull/52152", "createdAt": "2026-08-13T10:44:21Z", "updatedAt": "2026-08-13T11:32:29Z", "timestamp": "2026-08-13T11:32:29Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "documentation" ], "author": "Sunbeam23333", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52153", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Pass model EOS ids to xgrammar tokenizer info", "text": "## Purpose Fixes #52146. `XgrammarBackend.__post_init__` built xgrammar `TokenizerInfo` from the tokenizer's single `eos_token_id`. Models can declare multiple EOS ids in `generation_config.json`, and vLLM's sampling path can stop on those ids. If xgrammar does not know about the full EOS set at tokenizer-info construction time, one of those model EOS tokens can remain samplable inside a structured-output string and truncate otherwise valid JSON. This PR unions the tokenizer EOS id with `model_config.try_get_generation_config()[\"eos_token_id\"]` and passes that set into both xgrammar tokenizer-info construction paths: - `xgr.TokenizerInfo(...)` for Mistral/Tekken tokenizers. - `xgr.TokenizerInfo.from_huggingface(...)` for Hugging Face tokenizers. The helper tolerates absent, duplicate, and malformed generation-config EOS values and preserves the tokenizer EOS fallback. Duplicate-work / coordination checks run before opening this PR: - `gh issue view 52146 --repo vllm-project/vllm --comments`: issue open; I posted a coordination comment because the issue body mentions a patch branch. - `gh pr list --repo vllm-project/vllm --state open --search \"52146 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"xgrammar generation_config eos_token_id stop_token_ids TokenizerInfo\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched Python files. - Focused helper assertions for multi-id, single-id, duplicate, absent, and malformed generation-config EOS values. - Focused pytest selection for the new helper tests. - Ruff format/check on touched files. - Whitespace diff check. No model evals were run; this changes stop-token metadata used by structured-output grammar masking, not model quality or accuracy. ## Test Result - `./.venv/bin/python -m py_compile vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `./.venv/bin/python -c \"... _model_stop_token_ids helper assertions ...\"`: passed. - `./.venv/bin/pre-commit run ruff-format --files vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `git diff --check`: passed. - `./.venv/bin/python -m pytest tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py -k model_stop_token_ids -v`: first selected case reported `PASSED`, then local macOS test process exited with code 139 during global teardown in `torch.accelerator.memory.empty_host_cache()` via `tests/conftest.py::cleanup_fixture`. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52153", "createdAt": "2026-08-13T10:46:42Z", "updatedAt": "2026-08-13T10:47:20Z", "timestamp": "2026-08-13T10:47:20Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "structured-output" ], "author": "danielpolimac", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52156", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Apply attention sinks in the Transformers backend", "text": "## Purpose vLLM PR #48270 adds native `GraniteSWAForCausalLM` / `GraniteMoeSWAForCausalLM` because \"the transformers backend does NOT handle these models correctly. Sink tokens get dropped silently and so the model gives wrong output.\" That is accurate, and it is a bug in the Transformers modeling backend rather than something inherent to those models, so this PR fixes it there and every current and future sink model works on the backend. Transformers models with learnable attention sinks (GPT-OSS, GraniteSWA, GraniteMoeSWA, DeepSeek V4, MiMo-V2-Flash, ...) hand the per-head sink to the attention interface as `s_aux`. Only the attention impl can fold a sink into the softmax denominator, so vLLM has to pass it to `Attention` at construction time. `vllm_attention_forward` accepted it in `**kwargs` and dropped it, so every softmax in the model was wrong. Reproduced with `ibm-granite/granite-swash-2b` (greedy, transformers backend): | prompt | before | after | | --- | --- | --- | | `The capital of France is` | `''` | `' Paris.\\nThe capital of France is the largest city...'` | | `def fibonacci(n):` | `'::::::::::::::::::::::::'` | a correct fibonacci body | | `The boiling point of water at sea level is` | `'\\n\\n\\n at at at at at'` | `' 100 degrees Celsius (212 degrees Fahrenheit).'` | Two defects were involved: 1. **Sinks were dropped.** `Base.find_sinks` now locates the parameter Transformers keeps the sinks in, `Base.init_sink` materializes it as this rank's slice of the heads with a sharded weight loader, and `create_attention_instances` passes it as `Attention(sinks=...)`. `vllm_attention_forward` now raises instead of silently ignoring `s_aux` if a model applies sinks that we failed to find. 2. **`Attention` instances were invisible to `named_modules()`.** They are only stored in a plain dict, so `process_weights_after_loading` never ran for them, which is where FlashInfer upcasts sinks to the fp32 tensor its kernel requires. They are now registered in an `nn.ModuleList`, which also replaces the narrower `_vllm_mla_attn` workaround that existed for MLA layers. ## Test plan New tests in `tests/models/transformers/test_backend.py`: - `test_sinks` loads `tiny-random/gpt-oss-bf16` on the backend and asserts every attention layer has a sink and that the tensor it was handed matches the checkpoint. Skipped below SM90, where no backend supports sinks. - `check_implementation` now also asserts that attention instances are registered submodules, covering defect 2 for every model it is used with. Commands and results (1x GB200, transformers 5.15.0): ``` pytest tests/models/transformers/test_backend.py -k \"sinks\" # 1 passed pytest tests/models/transformers/test_backend.py -k \"test_mla or hybrid\" # 2 passed ``` ## Model evaluation Prompt logprobs for a 55-token paragraph, `ibm-granite/granite-swash-2b`, bf16, against HF eager as the reference (mean logprob -1.5943): | build | mean logprob | mean abs delta | max abs delta | | --- | --- | --- | --- | | main | -74.0868 | 72.4925 | 175.4590 | | this PR | -7.8748 | 6.8942 | 81.0792 | Greedy continuations become token-identical to HF with this PR. The residual logprob gap is a second, unrelated bug in the backend - Granite's `logits_scaling` divisor is ignored - fixed separately in #52158, which takes the same measurement to a mean absolute delta of 0.0165. ## Notes - Not a duplicate: no open PR addresses sinks in the Transformers backend (checked `is:pr is:open` for \"transformers backend\" and \"sinks\"). #48270 adds native implementations for two Granite variants; this fixes the backend for all sink models. The two changes are compatible. - Tensor parallel sink sharding follows the existing `sharded_weight_loader(0)` pattern (as in `gpt_oss.py`) but was not exercised, since only one GPU was available. Worth running `test_distributed`-style coverage before merge. - AI assistance was used for this change (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/52156", "createdAt": "2026-08-13T11:09:08Z", "updatedAt": "2026-08-13T13:01:02Z", "timestamp": "2026-08-13T13:01:02Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "tdoublep", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52157", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][Spec Decode] Support varlen trtllm-gen decode for adaptive verification", "text": "## Purpose - Resolves #51871: adaptive verification needs decode batches with per-request query lengths - Wire FlashInfer trtllm-gen decode to `cum_seq_lens_q`/`max_q_len`; `q_len_per_req` must be `None` for varlen - Flip `supports_device_cpu_query_lens_mismatch` on SM100: device `qo_indptr` is the source of truth, CPU lengths only an upper bound - Guard: adaptive verification without the trtllm-gen kernel now fails fast at init - Not a duplicate: no open PR touches trtllm-gen varlen decode ## Test Plan - `pytest tests/kernels/attention/test_flashinfer_trtllm_attention.py -k varlen` (B200) - Uniform decode regression subset on the same file - E2E: Gemma4-12B NVFP4 + DSpark (q=8) through the varlen path, GSM8K 50q - Kernel perf: varlen vs uniform-padded, profiler device time (B200) ## Test Result - 4 varlen + 16 uniform regression passed, 0 failed - E2E serves correctly: GSM8K 98%, 0% invalid, 17.6 q/s | shape | rows ragged/padded | varlen | padded | |---|---|---|---| | batch=8, q~U[1,8] | 38/64 | 28.0us | 22.2us | | batch=32, q~U[1,8] | 118/256 | 70.5us | 65.4us | | batch=32, q~U[1,4] | 74/128 | 61.4us | 54.2us | - Varlen carries a small fixed kernel overhead; it enables ragged batches rather than speeding uniform ones - Adaptive E2E is blocked on a public Gemma4 drafter with a confidence head; kernel tests cover the device-CPU mismatch semantics ---", "url": "https://github.com/vllm-project/vllm/pull/52157", "createdAt": "2026-08-13T11:12:00Z", "updatedAt": "2026-08-13T13:14:51Z", "timestamp": "2026-08-13T13:14:51Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "nvidia" ], "author": "guan404ming", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52158", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Apply logits scaling in the Transformers backend", "text": "## Purpose Transformers scales the logits in the `ForCausalLM` wrapper. This backend only loads the decoder and provides its own head and logits processor, so whatever the wrapper does after the decoder has to be reproduced here. Only `logit_scale` was, which means: - Granite-family and MiniCPM3 models returned logprobs `logits_scaling`x too confident: 9x for `ibm/PowerLM-3b`, 10x for `ibm-granite/granite-swash-2b`. - HyperCLOVAX silently dropped its muP multiplier. Scaling does not reorder the logits, so the generated tokens are unaffected. That is why this went unnoticed, and it means the impact is on consumers of logprob values: sampling with a temperature or penalties, logprob-based evals, speculative decoding acceptance, and anything ranking sequences by score. There is no config field or API to read the scale from, and no single convention for it either — `logit_scale` multiplies (Cohere), `logits_scaling` divides (Granite) except under muP where it multiplies (HyperCLOVAX), `logits_mup_width_multiplier` divides the hidden states (Inkling), `dim_model_base` implies a divisor (MiniCPM3), `lm_head_multiplier` multiplies (FalconH1). Rather than enumerate that, this PR measures it: at model init, build the model's own `ForCausalLM` class with its decoder and head replaced by stubs (no weights, no checkpoint), run its `forward` over known hidden states, and take the ratio. Whatever the wrapper does to the logits is then the only thing left in the output, and the value is inherited from Transformers instead of transcribed from it. If the ratio is not constant across the hidden states, the transform is not a scale (final logit softcapping) and cannot be folded into a logits processor, so we warn and leave the logits alone — as today. Same if the probe cannot be run at all. Neither is a regression, but both are now visible in the log rather than silent. Found while fixing attention sinks on the same backend (#52156), but independent of it. ## Test plan Three new tests in `tests/models/transformers/test_backend.py`: - `test_get_logit_scale` pins the measured scale for one config per convention (Granite, HyperCLOVAX, Cohere, MiniCPM3, Inkling), plus Llama for no scaling and Gemma2 to assert softcapping is *not* folded in as a scale. Pure CPU, no weights, runs in seconds. - `test_probe_logit_scale_runs_for_scaled_models` is the canary for the approach: it sweeps `MODEL_FOR_CAUSAL_LM_MAPPING`, sets every float config field that looks like a logit multiplier to a non-default value, and asserts the probe can still be run for each. If a Transformers refactor breaks the probe for a model that does scale its logits, we would silently fall back to 1.0; this fails CI instead. - `test_logits_scaling` compares the spread of the top-5 logprobs between the native and Transformers implementations of `ibm/PowerLM-3b`. Logprob *values* have to be compared because `check_logprobs_close` only compares token ids, which a missing division cannot change. Commands and results (1x GB200, transformers 5.15.0): ``` pytest tests/models/transformers/test_backend.py -k \"logit_scale\" # 8 passed pytest tests/models/transformers/test_backend.py -k \"logits_scaling\" # 1 passed ``` Mutation checks that the new tests bite: - Reverting the fix entirely: `test_logits_scaling` fails on 8/8 positions, max difference 48.06 against a 0.2 tolerance. - Dropping the config scalars from the decoder stub: the sweep fails on FalconH1, which reads its multiplier off the decoder. - Dropping `weight` from the head stub: the sweep fails on xLSTM, which reads the head dtype off it. What the probe finds across the 173 entries of `MODEL_FOR_CAUSAL_LM_MAPPING`, with default configs: 135 apply no scaling, 5 scale (Cohere/Cohere2/Cohere2Moe 0.0625, Inkling 1/24, MiniCPM3 0.1), 7 are nonlinear (Gemma2, Gemma3n, VaultGemma, RecurrentGemma, NanoChat, xLSTM — all final logit softcapping), 2 cannot have a default config built (Musicgen), and 24 wrappers cannot be run by the probe. Those 24 are encoder, seq2seq and legacy families (BERT variants, Whisper, TrOCR, XLNet, ...) whose head is not `lm_head`, plus Llama4/Mllama; none of them declares a logit scale, and they behave exactly as they do today. ## Model evaluation Prompt logprobs for a 55-token paragraph, `ibm-granite/granite-swash-2b`, bf16, against HF eager as the reference (mean logprob -1.5943). Measured on top of #52156, since that model also needs the sink fix to produce sane output at all: | build | mean logprob | mean abs delta | max abs delta | | --- | --- | --- | --- | | #52156 alone | -7.8748 | 6.8942 | 81.0792 | | #52156 + this PR | **-1.5990** | **0.0165** | **0.0863** | The remaining 0.0165 is bf16/kernel-level noise. ## Notes - Not a duplicate: no open PR touches logits scaling in the Transformers backend. - Final logit softcapping is still dropped by this backend, as it was before. It is not a scale, so it does not belong in the logits processor; `Attention(..., soft_cap=)` handles the attention-logit variant but the final one needs handling in `compute_logits`. Left for a separate PR, and the probe now warns when it is hit. - HyperCLOVAX's muP direction is covered by the unit test only. The smallest registered checkpoint is `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B`, too big for an e2e test here. - AI assistance was used for this change (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/52158", "createdAt": "2026-08-13T11:15:14Z", "updatedAt": "2026-08-13T11:57:20Z", "timestamp": "2026-08-13T11:57:20Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "tdoublep", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Hardware][Intel] Disable compiled grouped_topk on XPU", "text": "## Purpose Unconditional `@torch.compile` on `grouped_topk` emits an XPU Inductor kernel that gathers `scores[topk_ids]` with **out-of-range expert ids**: ``` Assertion `index out of bounds: 0 <= tmp0 < 128` failed. ``` This fires at engine-init warmup for any path that uses the Python grouped-topk router on XPU (NemotronH / Nemotron-3.5 Lightning MTP draft MoE, and any fallback that is not fused inside `XPUExpertsWNA16`). The fused base WNA16 MoE kernel does **not** execute this Python router, which is why the backbone can load while the draft / compiled router dies. This is the same class of compiled-reduction bug vLLM already documents for other backends (see the PowerPC argmax note in `topk_topp_sampler.py`). **Fix:** decorate `grouped_topk` with `torch.compiler.disable` **only when** `current_platform.is_xpu()`. CUDA keeps `@torch.compile` + the existing fused / fallback path. `torch.compiler.disable` is a graph break at the router only; the rest of the model can stay compiled / XPU-graph-captured. This is complementary to https://github.com/vllm-project/vllm/pull/39145 (fused XPU `grouped_topk` registration). That PR is the right long-term kernel path. This PR unblocks the eager/native router that still runs when that kernel is not selected (draft MoE, fused-flag off, or registration not yet landed). It does **not** add a second routing algorithm, expert-id clamps, or a CUDA-breaking top-level XPU import. Related kernels PR: https://github.com/vllm-project/vllm-xpu-kernels/pull/524 — zero the Xe2 grouped-GEMM scheduler counter (`at::empty` → `at::zeros`). Without that, graph-mode MoE can still be non-deterministic even if this router is correct. ## Test Plan ```bash pytest tests/kernels/moe/test_grouped_topk_xpu.py ``` The new file is skipped on non-XPU. On XPU it: - calls `grouped_topk` for softmax/sigmoid × bias/no-bias with `num_experts=128`, `num_expert_group=8`, `topk=8`, `topk_group=4`; - asserts every id is in `[0, num_experts)` and weights are finite; - wraps the same call in `torch.compile` and asserts it does **not** device-assert. Existing CUDA `tests/kernels/moe/test_grouped_topk.py` is unchanged. Manual serve repro (Intel XPU, NemotronH / Nemotron-3.5 Lightning with a draft that hits Python `grouped_topk`): ``` vllm serve --quantization gptq ... ``` Pre-fix: EngineCore dies during warmup with `0 <= tmp0 < 128`. Post-fix: router no longer asserts. ## Test Result On Intel Arc Pro B70: - Pre-fix: compiled `grouped_topk` device-asserts `0 <= tmp0 < 128` at engine-init warmup for NemotronH MTP draft MoE. - Post-fix: same serve path no longer asserts in the router. `torch.compile(grouped_topk)` on-device returns in-range ids. - CUDA path is a no-op (decorator still `@torch.compile`). - Full-eager + this router is deterministic at temp-0. Compiled/graph mode can still diverge in *other* XPU compiled kernels (sampler / GEMM without the kernels `at::zeros` fix). This PR does **not** claim “XPU is now deterministic.” No MTP prenorm, V1-bypass, W4A8, converters, or expert-id clamps in this PR.", "url": "https://github.com/vllm-project/vllm/pull/52159", "createdAt": "2026-08-13T11:20:36Z", "updatedAt": "2026-08-13T16:16:51Z", "timestamp": "2026-08-13T16:16:51Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "SergiioB", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52160", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Fix group numbering in Case 3 of hybrid_kv_cache_manager.md", "text": "## Purpose Fix a self-inconsistent group numbering in `docs/design/hybrid_kv_cache_manager.md` Case 3 (lines 111-124). The doc says Gemma-3-27b has 52 sliding-window layers, but lists: - Group 6: 10 sliding window attention layers (sw.40 - sw.49) - Group 7: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers With group_size=10, 52 sw layers end at Group 6 (sw.50-51 + 8 padding); there is no Group 7. sw.40-49 belongs to Group 5. Renumbered to: Group 5 = sw.40-49, Group 6 = sw.50-51 + 8 padding (7 groups total, Group 0-6). ## Why not duplicating an existing PR Searched `gh issue list` / `gh pr list` for \"hybrid_kv_cache_manager\": no open issue or PR. Pure documentation fix. ## Test Plan - pre-commit run markdownlint-cli2 --files docs/design/hybrid_kv_cache_manager.md ## Test Result Local environment has no pre-commit/Python; relying on CI check. --- ## AI Assistance This PR was drafted with AI assistance; the human submitter reviewed every changed line. Co-authored-by: deepseek-v4-flash", "url": "https://github.com/vllm-project/vllm/pull/52160", "createdAt": "2026-08-13T11:30:35Z", "updatedAt": "2026-08-13T11:31:11Z", "timestamp": "2026-08-13T11:31:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation" ], "author": "qwerqwerqwe8688-jpg", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52161", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Detect all attention-spelling variants in ModelConfig.is_hybrid", "text": "## Purpose `ModelConfig.is_hybrid`'s carve-out for granite-4.0-micro (a hybrid-capable architecture with zero actual mamba layers) only matched `layer_types` entries spelled `\"attention\"`. Transformers >= 5.13 normalizes pure-attention entries to `\"full_attention\"` instead (the model's own `granitemoehybrid.py` already handles both spellings in `ALL_DECODER_LAYER_TYPES` for layer dispatch, but `is_hybrid` never learned the new spelling). As a result `is_hybrid` stayed `True` for granite-4.0-micro. Combined with prefix caching now defaulting `mamba_cache_mode` to `\"align\"` for any hybrid-flagged model, `_get_mamba_bufs()` runs unconditionally and hits `AssertionError: no mamba layers in the model` on the first inference request — the engine starts and loads fine, then dies on the first completion. Fix: match both spellings in the carve-out. AI assistance was used to root-cause and author this fix; I reviewed every changed line and reproduced/verified the behavior end-to-end myself before opening this PR. ## Test Plan Reproduced on vLLM HEAD `37c3bdf5a` (2026-08-13): ``` vllm serve ibm-granite/granite-4.0-micro --max-model-len 8192 # then, once the server is up: curl http://localhost:8000/v1/completions \\ -H 'Content-Type: application/json' \\ -d '{\"model\":\"ibm-granite/granite-4.0-micro\",\"prompt\":\"The capital of France is\",\"max_tokens\":10}' ``` Then applied the fix and repeated the same two steps, plus a direct property check: ```python from vllm.config import ModelConfig mc = ModelConfig(model=\"ibm-granite/granite-4.0-micro\") print(mc.is_hybrid) ``` ## Test Result **Before fix:** server starts and loads the model successfully, then the completion request fails the engine: ``` AssertionError: no mamba layers in the model File \".../vllm/v1/worker/gpu_model_runner.py\", line 4437, in execute_model mamba_bufs = self._get_mamba_bufs() File \".../vllm/v1/worker/mamba_utils.py\", line 554, in get_mamba_groups assert len(mamba_group_ids) > 0, \"no mamba layers in the model\" vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` `ModelConfig(\"ibm-granite/granite-4.0-micro\").is_hybrid` → `True` (incorrect). **After fix:** `is_hybrid` → `False` (correct — this model has no mamba layers). Server starts and the same completion request succeeds: ```json {\"id\":\"cmpl-...\",\"choices\":[{\"text\":\" Paris.\\n\\n**Output:**\\n...\"}], ...} ``` No crash, no assertion. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52161", "createdAt": "2026-08-13T11:34:51Z", "updatedAt": "2026-08-13T13:32:59Z", "timestamp": "2026-08-13T13:32:59Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "ready" ], "author": "mganczarenko", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52162", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][PCP] Shard decode requests across PCP ranks", "text": "## Purpose PCP currently replicates every decode request on every PCP rank. For PCP-only deployments (`DCP == 1`), every rank already has a complete KV-cache replica, so this repeats the same decode computation `pcp_size` times. This PR: - assigns decode-only rows round-robin across PCP ranks on every scheduler step; - keeps the existing DualChunkSwap prefill partitioning unchanged; - gathers the owner-produced MLA and sparse-indexer KV updates so every PCP rank retains a complete cache replica; - restores hidden states in global batch order through the existing PCP gather; - preserves replicated decode when `DCP > 1`, because each DCP rank must consume its local KV shard. There is no new user-facing flag or API. Documentation is unchanged because this is an internal execution optimization with an automatic `DCP == 1` safety gate. ### Duplicate-work check I searched open PCP/decode issues and PRs before submission. This does not duplicate: - #49564, which adds FlashAttention/GQA PCP and PCP+DCP prefill support but does not shard PCP-only decode requests; - #49517, which replaces replicated KV-update collectives with opt-in, model-specific direct peer stores but does not reduce replicated decode computation; - #49741, which owner-shards persistent KV history for capacity rather than sharding decode compute. Related performance tracking: #31473. ### AI assistance OpenAI Codex was used to analyze, implement, test, benchmark, rebase, and prepare this draft. The commit includes an AI co-author trailer. This is intentionally a draft so the human submitter can review every changed line and the evidence before marking it ready. ## Test Plan Focused checks: ```bash uvx ruff==0.14.0 check \\ tests/v1/worker/test_gpu_pcp_manager.py \\ vllm/model_executor/layers/attention/mla_attention.py \\ vllm/model_executor/layers/attention/pcp.py \\ vllm/model_executor/layers/sparse_attn_indexer.py \\ vllm/v1/worker/gpu/pcp_manager.py uvx ruff==0.14.0 format --check \\ tests/v1/worker/test_gpu_pcp_manager.py \\ vllm/model_executor/layers/attention/mla_attention.py \\ vllm/model_executor/layers/attention/pcp.py \\ vllm/model_executor/layers/sparse_attn_indexer.py \\ vllm/v1/worker/gpu/pcp_manager.py .venv/bin/python -m pytest -q \\ --confcutdir=tests/v1/worker \\ tests/v1/worker/test_gpu_pcp_manager.py ``` The focused tests cover: - unique, balanced PCP-only decode ownership; - DCP fallback to replicated decode; - unchanged prefill partitioning in mixed batches; - gathered KV/slot mapping and hidden-state restoration order. End-to-end validation used 8x RTX 5090 with DeepSeek-V2-Lite-Chat: - functionality: concurrency 256, 4,096 output tokens, two rounds; - accuracy: full GSM8K test set, 1,319 examples, 5-shot, temperature 0, seed 42; - performance: 100K input / 1K output, 99% prefix-cache hit, concurrency 32, 128 requests, three retained PCP baseline/candidate runs. ## Test Result ### Current rebased HEAD - Rebased cleanly onto `upstream/main@5fee0a872d`. - `ruff check`: passed. - `ruff format --check`: 5 files already formatted. - `git diff --check upstream/main...HEAD`: passed. - Diff scope: 5 files, 219 insertions, 8 deletions. - The focused pytest suite passed `4 passed` on the pre-rebase candidate. It has not yet been rerun after the final rebase. ### Functionality and model evaluation All TP8, baseline PCP8, and candidate PCP8 functional runs completed: - `512/512` requests per state; - every request generated 4,096 tokens; - 2,097,152 output tokens per state; - zero errors and zero aborted requests. GSM8K over three runs: | State | Correct answers | Mean accuracy | Std. dev. | |---|---:|---:|---:| | Baseline PCP8 | 852, 852, 851 | 0.645691 | 0.000438 | | Candidate PCP8 | 844, 853, 869 | 0.648471 | 0.009600 | There is no stable accuracy decrease, but the candidate variance is materially higher. Accuracy is therefore classified as **inconclusive / nondeterministic**, not passed. ### PCP decode performance Each retained measurement completed 128/128 requests with exactly 1,024 output tokens and a measured prefix-cache token hit rate of 98.993%. | Metric | Baseline PCP8 | Candidate PCP8 | Change | |---|---:|---:|---:| | Request throughput | 0.3810 req/s | 0.5102 req/s | +33.91% | | Output throughput | 390.15 tok/s | 522.45 tok/s | +33.91% | | Total-token throughput | 39,407.53 tok/s | 52,771.39 tok/s | +33.91% | | Mean TTFT | 11,153 ms | 11,328 ms | +1.57% | | Mean TPOT | 71.16 ms | 50.20 ms | -29.46% | | Mean ITL | 71.62 ms | 50.54 ms | -29.42% | Throughput CV was 0.25% for baseline and 0.34% for candidate. A candidate-only 100K/1-token guardrail measured TP4+EP4 at 4,887.26 ms and PCP4+EP4 at 5,019.78 ms (+2.71% PCP TTFT). The multi-GPU results above were collected before the final rebase. The runtime diff applied cleanly to current main and current-head static checks pass, but the full multi-GPU matrix has not yet been repeated on `0b478f031e`. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52162", "createdAt": "2026-08-13T12:29:43Z", "updatedAt": "2026-08-13T12:30:22Z", "timestamp": "2026-08-13T12:30:22Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "mrv2" ], "author": "pisceskkk", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52163", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend][Rust] Bound chat-template evaluation to prevent DoS (#52025)", "text": "## Purpose Fixes #52025. The Rust frontend renders a caller-supplied (or model-supplied) Jinja `chat_template` from `POST /v1/chat/completions` with **no evaluation budget**. Rendering cost is `O(N^depth)` in caller-controlled loop bounds, and minijinja's per-`range()` element limit is trivially walked around by nesting three individually-legal `range()` calls. A **116-byte** request body can occupy a request-runtime worker thread for **tens of seconds**: ```jinja {% for x0 in range(999) %}{% for x1 in range(999) %}{% for x2 in range(999) %}{% endfor %}{% endfor %}{% endfor %}ok ``` The request runtime is a bounded pool (`worker_threads = min(available_parallelism, 32)`), so a small number of such requests can stall the chat-completions plane while `/health` stays green. The request-supplied gate (`trust_request_chat_template`, default off in Python vLLM) is not yet implemented in the Rust frontend, and the **model-supplied** template path has no gate at all. This wires up minijinja's existing `fuel` feature and sets a total evaluation budget in `build_environment` (`rust/src/chat/src/renderer/hf/template.rs`). The budget bounds total work regardless of who supplied the template, so a hostile template is rejected quickly while legitimate templates — which only iterate over the request's messages/tools/documents — are unaffected. For reference, minijinja itself already refuses a single `range()` above ~1M elements; the budget here (20M fuel units) is ~20× that, well above any real chat template. This is the primary remediation (item 1) from the issue. Implementing `trust_request_chat_template` and `spawn_blocking`-ing the render (items 2 and 4) are follow-ups and out of scope here. ## Test Plan `cargo test -p vllm-chat` — added two tests in `template.rs`: - `test_chat_template_evaluation_is_bounded`: the nested-`range()` bomb from the issue is now rejected (and returns quickly instead of running for ~55s). - `test_chat_template_within_budget_renders`: a legitimate template doing non-trivial work (`range(50000)`) still renders. ## Test Result ``` $ cargo test -p vllm-chat test result: ok. 267 passed; 0 failed test result: ok. 17 passed; 0 failed test result: ok. 16 passed; 0 failed ``` All existing `vllm-chat` tests pass (the budget does not affect any real template); `cargo fmt --check` and `cargo clippy -p vllm-chat` are clean.", "url": "https://github.com/vllm-project/vllm/pull/52163", "createdAt": "2026-08-13T12:43:07Z", "updatedAt": "2026-08-13T12:47:18Z", "timestamp": "2026-08-13T12:47:18Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "rust" ], "author": "jaideeppyne", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52164", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][DSA] Take the native decode path for MTP=3 on SM90", "text": "# [Attention][DSA] Take the native decode path for MTP=3 on SM90 ## Purpose Closes #35878. The DSA indexer flattens a spec-decode batch into one single-token row per query whenever `next_n` falls outside `{1, 2}`, so with MTP=3 (`next_n = 4`) each request's KV tile is read four times instead of once. DeepGEMM's `nv_dev` branch, which vLLM already pins (`cmake/external_projects/deepgemm.cmake`), implements `next_n = 4` on SM90 through a 2-CTA multicast cluster, so Hopper no longer needs that expansion. #45322 did the equivalent for SM100; this is the SM90 case the issue asks for. Behavior matrix (\"native\" = kernel sees the real batch `(B, next_n)`; \"flatten\" = requests expanded into `B × next_n` single-token pseudo-requests): | Platform | `next_n` (= spec tokens + 1) | Before | After | Changed? | |---|---|---|---|---| | SM100 (B200/GB200) | any | native | native | — | | **SM90 (H100/H200)** | **4 (MTP=3)** | **flatten** | **native** | ✅ **this PR** | | SM90 | 1, 2 | native | native | — | | SM90 | 3, ≥ 5 | flatten | flatten | — (no kernel) | | SM12x and others | 1, 2 | native | native | — | | SM12x and others | > 2 | flatten | flatten | — (unvalidated) | Three pieces: 1. The hardcoded gate becomes `_supports_native_decode(next_n)`, which asks `native_next_n_supported()` per architecture. SM90 implements `{1, 2, 4}` — note `3` is absent, so the gate is not a simple `>=` threshold. 2. `get_paged_mqa_logits_metadata` now sizes its own scheduler slots. `fp8_fp4_paged_mqa_logits` asserts the metadata against `num_sms / num_kv_multicast`, and SM90 `next_n = 4` runs one task per 2-CTA cluster rather than per SM, so the count must be halved. The wrapper derives `next_n` from `context_lens.shape[1]` and divides internally, so `num_sms` keeps its literal meaning and callers cannot get it wrong. The metadata buffer stays sized for one slot per SM; `build()` narrows it to a prefix view using the returned tensor's own shape. 3. Legality is a property of the step, not of the configuration: the kernel is handed `max_decode_len` Q rows, not the configured `next_n`, so a batch that happens to be uniformly 3 tokens deep still flattens on SM90. ## Test Plan Unit and kernel tests: ```bash pytest tests/v1/attention/test_indexer_native_next_n.py -v pytest tests/kernels/attention/test_deepgemm_attention.py -v ``` A standalone microbenchmark (not included in this PR) timed `fp8_fp4_paged_mqa_logits` on the same inputs fed natively as `(B, next_n)` and flattened to `B × next_n` single-token rows, to size the effect the decode path is trading on. End-to-end, DeepSeek-V3.2 on 8×H200, TP8 + EP, MTP=3. The \"before\" arm is this same tree with only `vllm/v1/attention/backends/mla/indexer.py` reverted to the flattening gate, so nothing else differs between arms. Every run was checked against the startup log line to confirm which path it took: ``` before: DSA indexer decode path: use_flattening=True supports_varlen=False (next_n=4, ...) after: DSA indexer decode path: use_flattening=False supports_varlen=False (next_n=4, ...) ``` Accuracy: ```bash cd tests/evals/gsm8k pytest -s -v test_gsm8k_correctness.py \\ --config-list-file=configs/models-h200.txt -k DeepSeek-V3.2-TP ``` `DeepSeek-V3.2-TP.yaml` only reports acceptance length when `min_acceptance_length` is set, so a local copy with `min_acceptance_length: 1.0` was used to make `get_acceptance_length()` print it. Its `startup_max_wait_seconds: 1200` is also not enough on a cold JIT cache — the first start on this machine took about 24 minutes. gsm8k runs at 4096 context while `index_topk` is 2048, so its top-k is close to \"select everything\" and it is not sensitive to the indexer's numerics. MRCR was run as a long-context check where the selection is genuinely sparse: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 32768 -tp 8 \\ --enable-expert-parallel --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' python tests/evals/mrcr/mrcr_eval.py --port 8000 --num-samples 24 --max-tokens 1024 ``` Serving performance: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 40960 -tp 8 \\ --enable-expert-parallel --trust-remote-code --no-enable-prefix-caching \\ --num-gpu-blocks-override 7000 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' vllm bench serve --model deepseek-ai/DeepSeek-V3.2 --dataset-name random \\ --random-input-len 32000 --random-output-len 512 --random-range-ratio 0 \\ --num-prompts $C --max-concurrency $C --ignore-eos --seed 1234 # C in 1 4 8 12 ``` Two methodology notes, because a naive sweep gives misleading numbers here: * `--num-gpu-blocks-override` pins both arms to the same KV cache. Left to the memory profiler the two arms landed on 506k vs 483k tokens on one pair of runs (and the other way round on another), which by itself changes queueing. * `--num-prompts == --max-concurrency` keeps the run to a single wave, so it is decode-bound. At 32K a 143GB×8 node holds roughly 12 concurrent requests, so higher concurrency measures the scheduler queueing, not this kernel. Every configuration was run twice so the run-to-run spread is visible next to the effect. ## Test Result ### Unit and kernel tests ``` tests/v1/attention/test_indexer_native_next_n.py ......... 9 passed tests/kernels/attention/test_deepgemm_attention.py ..... 5 passed ``` `test_deepgemm_fp8_fp4_paged_mqa_logits[2-4]` covers `next_n = 4` against the reference implementation. ### Kernel-level effect (H200, `next_n = 4`) Native `(B, next_n)` vs flattened, speedup of the paged MQA logits call alone: \"image\" | batch | 4K ctx | 16K ctx | 32K ctx | |---|---|---|---| | 1 | 0.97x | 1.03x | 1.03x | | 4 | 1.02x | 1.10x | 1.15x | | 8 | 1.02x | 1.19x | 1.33x | | 16 | 1.07x | 1.25x | 1.33x | | 32 | 1.12x | 1.27x | 1.36x | | 64 | 1.27x | 1.45x | 1.46x | The gain grows with both batch and context, which is what sharing the KV tile predicts. At batch 1 / 4K the native path is 3% slower — flattening one request into four rows gives the kernel more parallelism than it can otherwise use. That case does not appear end-to-end (see below), so the gate is left unconditional. ### End-to-end serving (32K input, 512 output) \"image\" | concurrency | mean TPOT before | after | Δ | output tok/s before | after | Δ | |---|---|---|---|---|---|---| | 1 | 16.44 ms | 16.23 ms | −1.2% | 44.61 | 45.00 | +0.9% | | 4 | 28.02 ms | 26.00 ms | −7.2% | 88.85 | 89.81 | +1.1% | | 8 | 43.43 ms | 42.54 ms | −2.0% | 108.90 | 109.84 | +0.9% | | 12 | 60.99 ms | 58.88 ms | −3.5% | 117.19 | 118.46 | +1.1% | Values are the mean of two runs. Throughput improves by 0.9–1.1% at every concurrency, and at 1, 8 and 12 that clears each arm's own run-to-run range; the concurrency-4 cell does not (its TPOT spread is ±9%, larger than the effect). There is no regression at concurrency 1, so the batch-1 kernel result above does not surface in serving. The end-to-end gain is far smaller than the kernel speedup because the paged MQA logits kernel is a small share of a 671B MoE decode step.
Raw output, concurrency 12 Before (`use_flattening=True`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 52.52 Output token throughput (tok/s): 116.99 Total token throughput (tok/s): 7428.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 61.62 Median TPOT (ms): 60.08 P99 TPOT (ms): 93.81 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.12 Median ITL (ms): 33.42 ``` After (`use_flattening=False`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 51.88 Output token throughput (tok/s): 118.43 Total token throughput (tok/s): 7520.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 59.60 Median TPOT (ms): 60.80 P99 TPOT (ms): 92.96 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.36 Median ITL (ms): 32.64 ```
### Accuracy \"image\" gsm8k, 1319 questions, invalid rate 0.000 on every run: | run | before | after | |---|---|---| | 1 | 0.9560 | 0.9538 | | 2 | 0.9522 | 0.9553 | | 3 | — | 0.9530 | | **mean** | **0.9541** | **0.9540** | Mean acceptance length, which is the control that matters for comparing the performance numbers at all: | run | before | after | |---|---|---| | 1 | 3.074 | 3.069 | | 2 | 3.081 | 3.079 | | 3 | — | 3.078 | | **mean** | **3.077** | **3.075** | The arms overlap on both measures. At n = 1319 one standard error on accuracy is about 0.006, and the arms differ by 0.0001. MRCR at 32K context, where the indexer's top-k is genuinely selective: | | before | after | |---|---|---| | match_ratio | 0.4873 | 0.5637 | | prefix_hit_rate | 1.0000 | 0.9583 | | n=2 / n=4 / n=8 | 0.6494 / 0.4745 / 0.3380 | 0.8091 / 0.5510 / 0.3310 | One run per arm at 24 samples over three needle buckets, so this shows no degradation rather than a gain. ### Path coverage The change adds a per-step fallback for the case where `max_decode_len` is 3, which has no SM90 kernel. Instrumented runs across gsm8k, MRCR and the serving sweep recorded over 140,000 decode steps and saw only `max_decode_len` 1 (the draft model's own decode) and 4 (the verify step) — never 3 — so the fallback stays cold in steady-state MTP decoding. Both cudagraph modes were exercised: gsm8k under `--enforce-eager`, and the serving sweep with `CUDAGraphMode.FULL_AND_PIECEWISE` capturing all 49 decode sizes. --- Not duplicating existing work: no open PR references #35878, and none of the open sparse-indexer PRs touch the SM90 `next_n` gate — #47469 is SM100 varlen, #51555 and #43327 are ROCm, #47629 and #38476 are SM8x/SM12x backends. #45322 is merged and covers SM100 only. AI assistance was used for this change and its validation.", "url": "https://github.com/vllm-project/vllm/pull/52164", "createdAt": "2026-08-13T13:06:39Z", "updatedAt": "2026-08-13T13:42:03Z", "timestamp": "2026-08-13T13:42:03Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "verified" ], "author": "zobinHuang", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52165", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc][Spec Decode] Detect DeepSeek-V4 DSpark checkpoints from config", "text": "## Purpose To close https://github.com/vllm-project/vllm/issues/52111 ## Test Plan Add a test case `tests/config/test_deepseek_v4_dspark_config.py` (TODO) ## Test Result Coming soon. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52165", "createdAt": "2026-08-13T13:13:35Z", "updatedAt": "2026-08-13T13:14:23Z", "timestamp": "2026-08-13T13:14:23Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "deepseek" ], "author": "xuebwang-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52166", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Zen5 image build reuse cpu", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52166", "createdAt": "2026-08-13T13:23:34Z", "updatedAt": "2026-08-13T13:24:11Z", "timestamp": "2026-08-13T13:24:11Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "ci/build", "cpu" ], "author": "Chinmay-Kulkarni-AMD", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Restore multimodal support on the plain \"vllm\" throughput backend", "text": "## Purpose `vllm bench throughput --backend vllm` crashes on any multimodal dataset (e.g. ASR/Whisper) with: ValueError: Multi-modal content is not supported on backend 'vllm'; use one of []. Regression from #50981. `get_requests()` computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handle: ```python mm_backends = (\"vllm-chat\",) if args.backend == \"vllm-chat\" else () ``` This is circular — it only lets the already-selected backend through the gate, and only if it's literally \"vllm-chat\". So --backend vllm always computes mm_backends=() and self-rejects, even though _run_vllm_requests (the function that executes --backend vllm requests) already forwards multi_modal_data straight into LLM.generate — it was always multimodal-capable, same as vllm-chat. Only hf (asserts prompts must be str) and mii (text-only) genuinely can't carry multimodal content, so the allowlist should be static rather than derived from the chosen backend: ```python mm_backends = (\"vllm\", \"vllm-chat\") ``` ## Test Plan - Added test_get_requests_allows_multimodal_on_plain_vllm_backend to tests/benchmarks/test_throughput_cli.py, mirroring the existing test_get_requests_resolves_mmvu pattern: stubs ASRDataset (network-free), parses --backend vllm --dataset-name hf --dataset-path openslr/librispeech_asr, asserts get_requests() resolves without raising. - Ran the exact vllm bench throughput command from the bug report end-to-end on real hardware (Intel XPU B70) before and after the fix. - ruff check / ruff format on both changed files. ## Test Result - New test: fails with the reported ValueError against pre-fix code, passes against post-fix code. - python -m pytest tests/benchmarks/test_throughput_cli.py -k \"allows_multimodal_on_plain_vllm_backend or resolves_mmvu\" → 2 passed. - End-to-end on B70: pre-fix reproduces the ValueError verbatim; post-fix completes cleanly (Throughput: 20.96 requests/s, 5386.74 total tokens/s). - ruff check: all checks passed. ruff format --diff: no changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52168", "createdAt": "2026-08-13T13:27:39Z", "updatedAt": "2026-08-13T17:34:35Z", "timestamp": "2026-08-13T17:34:35Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "performance" ], "author": "mganczarenko", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52169", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Note pooled aggregation semantics for mean_acceptance_length", "text": "## Purpose `SpecDecodingLogging._log` and the PromQL recipe in `SpecDecodingProm.__doc__` compute the `mean_acceptance_length` as a **pooled mean** (weighted by draft count, equivalent to `total_generated / total_verify_calls`), not an unweighted mean of per-request means. Some external benchmarks report the latter — SpecForge's `average_acceptance_length`, for example, is `statistics.fmean(acceptance_lengths)` over per-request lengths. On variable-length workloads the two aggregations can differ by several points in either direction, depending on the sign of the per-request-length ↔ acceptance-rate correlation. This has confused at least one report already (issue #42508: a 55.08%-vs-44% pooled-vs-mean-of-means gap for EAGLE3 on GSM8K, and 66.57%-vs-72% in the opposite direction for the standalone drafter — same numerator, different aggregation). ## Changes Docstring/comment only. Adds a short aggregation note - inline where `mean_acceptance_length` is computed, and - inside `SpecDecodingProm.__doc__` below the existing PromQL recipe. Both points cross-reference #42508 for the reproduction. The metric definitions and the CLI/Prometheus output are byte-identical to before. ## Test Plan None — comment/docstring only, no runtime behavior changed. `ruff format --check` clean; pre-commit hooks pass locally. ## Test Result n/a ## Documentation Update n/a (this PR *is* the documentation update). ## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [x] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1MBOTQeE4aOFIL8UkuGRyGXe6qYtsIt1FGWyxTvvIvtM/edit?tab=t.0).", "url": "https://github.com/vllm-project/vllm/pull/52169", "createdAt": "2026-08-13T13:28:14Z", "updatedAt": "2026-08-13T15:06:02Z", "timestamp": "2026-08-13T15:06:02Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "speculative-decoding" ], "author": "arrdel", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Declare SupportsEagle3 on KimiLinearForCausalLM", "text": "### Purpose `KimiK3ForConditionalGeneration` (multimodal) declares `SupportsEagle3`; the text-only `KimiLinearForCausalLM` does not — even though both serve the same inner `KimiLinearModel`, which already inherits `EagleModelMixin` and implements the aux-hidden-state tap machinery. Serving a text-only Kimi-K3 checkpoint with EAGLE3-family speculative decoding (e.g. `dspark`) therefore dies at startup: ``` RuntimeError: Model does not support EAGLE3 interface ``` Adding the interface to the class bases is the whole fix: the protocol's default `set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` delegate to `self.model`, which satisfies their `EagleModelMixin` requirement. ### Test plan - `tests/models/kimi_k3/test_eagle3.py` gains `test_kimi_linear_advertises_eagle3_support`, mirroring the existing multimodal assertion; it fails on `main` and passes with this change. - Runtime-validated: a text-only Kimi-K3 checkpoint served with a `dspark` draft on 8× RTX 3090 using an equivalent patch (that is how we hit the error). --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52171", "createdAt": "2026-08-13T13:55:11Z", "updatedAt": "2026-08-13T16:22:51Z", "timestamp": "2026-08-13T16:22:51Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "verified", "kimi", "k3" ], "author": "nickus", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52172", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable sequence parallelism for Dots3 NOTE", "text": "## Why The DeepSeek V3.2 sequence-parallel refactor changed the inherited forward paths to use `use_sequence_parallel`. Dots3 NOTE uses custom model and decoder initializers and has not adopted the new sequence-parallel execution path. This causes serving to fail during KV cache profiling with: `AttributeError: 'Dots3NoteModel' object has no attribute 'use_sequence_parallel'` ## What changed Explicitly disable sequence parallelism for Dots3 NOTE and keep its existing non-SP execution path. ## Validation - BF16 DP8 + EP + MTP3: startup and inference passed. - BF16 TP8 + EP + MTP3: startup and inference passed. - Text and image requests completed successfully. - No duplicate issue or PR was found.", "url": "https://github.com/vllm-project/vllm/pull/52172", "createdAt": "2026-08-13T13:57:37Z", "updatedAt": "2026-08-13T14:13:01Z", "timestamp": "2026-08-13T14:13:01Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "verified" ], "author": "KurodaKanbei", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52173", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Apply logit softcapping in Transformers modelling backend", "text": "This is mainly used by Gemma models and only affects workloads where the exact logit values are important. Generation is unaffected because it does not reorder anything.", "url": "https://github.com/vllm-project/vllm/pull/52173", "createdAt": "2026-08-13T14:01:25Z", "updatedAt": "2026-08-13T16:21:56Z", "timestamp": "2026-08-13T16:21:56Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "hmellor", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52174", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add forward_xpu to XDRotaryEmbedding for HunyuanOCR on XPU", "text": "## Purpose `XDRotaryEmbedding` (rope_type `xdrope`, used by `tencent/HunyuanOCR`, arch `HunYuanVLForConditionalGeneration`) overrides `forward_native` and `forward_cuda` but not `forward_xpu`. `CustomOp.dispatch_forward()` binds the method once at init based on `compilation_config.custom_ops`: - Default (Inductor on): `custom_ops=[]` -> `'none'` -> custom op disabled -> `forward_native`. Handles the 2-D `[4, num_tokens]` xdrope positions correctly -> PASS. - `--enforce-eager`: Inductor off -> `custom_ops=['all']` -> custom op enabled -> `forward_xpu`. `XDRotaryEmbedding` has none, so dispatch falls back to base `RotaryEmbedding.forward_xpu`, which calls the generic C++ kernel `torch.ops._C.rotary_embedding`. That kernel expects 1-D positions matching q/k `num_tokens`; given the 2-D `[4, num_tokens]` (P/W/H/T sections) positions it raises: ``` RuntimeError: query, key and positions must have the same batch_size and seq_len ``` crashing `EngineCore` during `profile_run`, so the model cannot serve at all under `--enforce-eager` on XPU. Fix: add `forward_xpu` to `XDRotaryEmbedding` delegating to `forward_native` (no fused XPU xdrope kernel exists), mirroring how mrope handles XPU. ## Test Plan ```bash vllm serve tencent/HunyuanOCR --dtype bfloat16 --tensor-parallel-size 1 \\ --max-model-len 8192 --gpu-memory-utilization 0.85 \\ -cc '{\"inductor_compile_config\":{\"benchmark_combo_kernel\":false}}' \\ --port 8005 --trust-remote-code --enforce-eager \\ --limit-mm-per-prompt '{\"image\": 1}' ``` ## Test Result Before fix - `EngineCore` crashes during `profile_run`: ``` File \".../vllm/model_executor/models/hunyuan_v1.py\", line 237, in forward q, k = self.rotary_emb(positions, q, k) File \".../vllm/model_executor/custom_op.py\", line 136, in forward return self._forward_method(*args, **kwargs) File \".../vllm/model_executor/layers/rotary_embedding/base.py\", line 288, in forward_xpu ops.rotary_embedding(...) File \".../vllm/_custom_ops.py\", line 212, in rotary_embedding torch.ops._C.rotary_embedding(...) RuntimeError: query, key and positions must have the same batch_size and seq_len ``` After fix: - `Application startup complete` - `/health` -> 200; `/v1/models` serves `tencent/HunyuanOCR` Default (torch.compile) path already routed to `forward_native` and is unaffected.", "url": "https://github.com/vllm-project/vllm/pull/52174", "createdAt": "2026-08-13T14:12:13Z", "updatedAt": "2026-08-13T14:12:57Z", "timestamp": "2026-08-13T14:12:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "jbyczkow", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52175", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Cohere ChatV2 citation and tool handling issues", "text": "## Purpose Fix issues ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52175", "createdAt": "2026-08-13T14:21:42Z", "updatedAt": "2026-08-13T17:07:51Z", "timestamp": "2026-08-13T17:07:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "frontend", "needs-rebase", "tool-calling", "cohere" ], "author": "andrewbcohere", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52176", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][DBO] Run the profile run without DBO when DBO is enabled.", "text": "## Purpose Running the profile run with DBO effectively sets the max batch size to `max_num_tokens // 2`. If vllm bails out of DBO for any reason, it can end up in a situation where it's running with a batch size that is larger than `max_num_tokens // 2`. Deep EP v2 crashes in this case because an internal buffer wasn't sized correctly, but it's plausible that there is other undesirable behavior in the system when we encounter this scenario. The fix is to run the profile run with DBO explicitly disabled. ## Test Result ``` vllm serve deepseek-ai/DeepSeek-V2-Lite \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --all2all-backend deepep_highthroughput \\ --enable-dbo \\ --dbo-decode-token-threshold 32 \\ --dbo-prefill-token-threshold 32 ``` lm_eval result ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3798|± |0.0134| | | |strict-match | 5|exact_match|↑ |0.3783|± |0.0134| ```", "url": "https://github.com/vllm-project/vllm/pull/52176", "createdAt": "2026-08-13T14:24:06Z", "updatedAt": "2026-08-13T14:24:44Z", "timestamp": "2026-08-13T14:24:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug" ], "author": "SageMoore", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52177", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DBO] Add DBO support to the DeepEP V2 backend", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52177", "createdAt": "2026-08-13T14:25:20Z", "updatedAt": "2026-08-13T14:26:33Z", "timestamp": "2026-08-13T14:26:33Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "SageMoore", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52178", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Propagate fatal EngineCore shutdown", "text": "## Purpose Fixes #48966. When `EngineCore` dies unexpectedly, the watchdog asks Uvicorn to stop, but `serve_http()` currently returns the same no-op shutdown coroutine used for a normal server stop. The caller therefore completes cleanly and the serving process exits with status 0, so supervisors using `Restart=on-failure` do not restart it. This change keeps a reference to the `EngineClient` across HTTP teardown. After the server task finishes, it returns a coroutine that re-raises the recorded `dead_error` only when the engine is dead and shutdown was not initiated by a user signal. SIGINT/SIGTERM and a healthy server stop retain successful exit semantics. This PR is opened as a draft to confirm the process exit contract and the placement of the check in the HTTP launcher before marking it ready. The exception is not consumed by an intermediate cleanup handler on the single-server path: `serve_http()` returns the failure coroutine, `run_server_worker()` awaits it after leaving the engine-client context and closes the socket in a `finally`, `run_server()` lets it propagate, and the CLI calls `uvloop.run(run_server(args))` without a broad catch. The process-level result below verifies the end of that chain rather than only the coroutine. ## Test Plan 1. Unit-test the four relevant terminal states through `serve_http()`: unexpected engine death, intentional signal shutdown, a signal observed alongside engine death, and a healthy server stop. If both a user signal and engine death are observed before HTTP teardown completes, the deliberate signal takes precedence and preserves exit 0. The signal callback records this flag synchronously rather than scheduling a coroutine to set it later. 2. Start the OpenAI API server on one RTX 4090, wait for `/health`, send one successful completion request, then kill only `EngineCore` and record the top-level exit status. 3. Repeat the same startup and request path, but send SIGTERM to the API server as the control case. 4. Repeat the process-level validation with TP=2 on two RTX 4090 GPUs, including killing only one TP worker to exercise worker death propagation through EngineCore. ## Test Result ```text $ python -m pytest tests/entrypoints/unit_tests/test_launcher.py -q .... [100%] 4 passed in 1.39s $ python -m ruff check \\ vllm/entrypoints/launcher.py \\ vllm/entrypoints/openai/api_server.py \\ tests/entrypoints/unit_tests/test_launcher.py All checks passed! $ python -m ruff format --check \\ vllm/entrypoints/launcher.py \\ vllm/entrypoints/openai/api_server.py \\ tests/entrypoints/unit_tests/test_launcher.py 3 files already formatted ``` Process-level validation on NVIDIA GeForce RTX 4090, PyTorch 2.13.0+cu130: ```text EngineCore SIGKILL: ready=1 request_status=200 wrapper_rc=1 server_rc=1 Intentional SIGTERM: ready=1 request_status=200 wrapper_rc=0 server_rc=0 ``` TP=2 process-level validation on two NVIDIA GeForce RTX 4090 GPUs: | Injection after `/health` and HTTP 200 | wrapper_rc | server_rc | Timed out | | --- | ---: | ---: | ---: | | SIGKILL only `VllmWorker-0` | 1 | 1 | 0 | | SIGKILL only EngineCore | 1 | 1 | 0 | | SIGTERM API server | 0 | 0 | 0 | The worker trial exercised the full propagation chain: the EngineCore log reported `VllmWorker-0 died unexpectedly`, followed by an EngineCore fatal error and `EngineDeadError` in the API server. All three trials exited without leaving an API server, EngineCore, or TP worker process behind. The multi-port DP supervisor remains a separate parent-process lifecycle path and is not claimed as covered by this focused change. Follow-up: audit and, if necessary, separately fix nonzero child-exit propagation through the multi-port `DPSupervisor` parent process. That path has different lifecycle semantics and should have its own focused regression. ## Duplicate-work check Before preparing this draft, issue #48966 had no linked development and the open-PR searches for `48966` and `EngineCore exit status` returned no matching fix. ## AI assistance disclosure AI assistance was used for code review, test design, and drafting this PR description. The submitter reviewed the changed lines and ran the tests and process-level validation above.", "url": "https://github.com/vllm-project/vllm/pull/52178", "createdAt": "2026-08-13T14:30:50Z", "updatedAt": "2026-08-13T16:00:36Z", "timestamp": "2026-08-13T16:00:36Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "frontend" ], "author": "jackLei0901", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52179", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix PP speculative decode cadence with sync scheduling", "text": "## Purpose Fixes #52071. When using the V2 model runner with pipeline parallelism and speculative decoding, the base `Scheduler` can schedule a request again before the sampled token from the previous PP step has been relayed back. The scheduler already checks `request.next_decode_eligible_step`, but this value was only being updated in `AsyncScheduler`. With async scheduling disabled, speculative draft tokens can increase `num_tokens_with_spec` and make the request runnable again while the sampled-token state is still stale. This PR updates `next_decode_eligible_step` in the base scheduler when: - the V2 model runner is enabled; - pipeline parallelism is enabled; - speculative decoding is enabled; - the request is no longer in a prefill chunk. The next eligible step is set to: ```text current_step + pipeline_parallel_size ``` This keeps non-final chunked prefill scheduling unchanged, while decode waits for the PP sampled-token relay before the request can be scheduled again. The existing `AsyncScheduler` handling is unchanged. Thanks @nickus for the detailed investigation in #52071. The reproduction, measurements, and analysis around the PP sampled-token relay and stale sampled tokens were especially helpful in narrowing this down to the scheduler cadence. ## Regression test Added: ```text tests/v1/core/test_scheduler.py::test_v2_pp_spec_decode_waits_for_sampled_token_relay ``` The test uses the base scheduler with: - V2 model runner - PP=2 - speculative decoding enabled - async scheduling disabled It simulates the final prefill step, injects speculative draft tokens, and checks that the request is not scheduled on the next step while the sampled token is still being relayed. After the sampled token is settled, the request becomes eligible at the expected PP step and schedules the sampled-token anchor together with the draft tokens. ## Test results Targeted regression: ```text tests/v1/core/test_scheduler.py::test_v2_pp_spec_decode_waits_for_sampled_token_relay 1 passed ``` Related scheduler tests: ```text tests/v1/core/test_async_scheduler.py 16 passed tests/v1/core/test_scheduler.py -k \"spec\" 17 passed ``` `git diff --check` also passes. ## GPU E2E I also ran a before/after GPU repro on 2x RTX 4090. To keep the repro small while still exercising the real PP + MTP path, I used the GLM-4.7-Flash architecture with `load_format=\"dummy\"` and reduced the model through `hf_overrides`. The test configuration was: ```text V2 model runner async_scheduling=False MTP speculative decoding num_speculative_tokens=1 4 transformer layers 2 routed experts greedy decoding 16 output tokens ``` The same setup was tested with PP=1 and PP=2. ### Before the fix | Configuration | Result | | --- | --- | | PP=1 + V2 + MTP + async=False | PASS | | PP=2 + V2 + MTP + async=False | FAIL — CUDA illegal memory access | The PP=1 run completed normally and produced 15 speculative draft steps. The PP=2 run did not produce an output and failed in the worker with: ```text torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` ### After the fix | Configuration | Result | | --- | --- | | PP=1 + V2 + MTP + async=False | PASS | | PP=2 + V2 + MTP + async=False | PASS | Both runs generated all 16 requested tokens and speculative decoding remained active: ```text PP=1: output_length = 16 num_drafts = 15 PP=2: output_length = 16 num_drafts = 15 ``` The PP=1 and PP=2 runs also produced identical token IDs: ```text [55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495] ``` So with the same model setup and scheduler configuration: | | Before fix | After fix | | --- | --- | --- | | PP=1 | PASS | PASS | | PP=2 | CUDA illegal memory access | PASS |", "url": "https://github.com/vllm-project/vllm/pull/52179", "createdAt": "2026-08-13T14:33:36Z", "updatedAt": "2026-08-13T14:47:42Z", "timestamp": "2026-08-13T14:47:42Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "icarus0adios-netizen", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52182", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Remove VLLM_TEST_FORCE_FP8_MARLIN to replace with linear_backend/moe_backend", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52182", "createdAt": "2026-08-13T14:53:00Z", "updatedAt": "2026-08-13T15:05:14Z", "timestamp": "2026-08-13T15:05:14Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "mgoin", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52183", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel)", "text": "## Purpose Update the PyTorch ecosystem to the **2.14.0** release candidate, resolving wheels from the PyTorch **test channel** (`download.pytorch.org/whl/test/...`): - `torch`: → **2.14.0** - `torchvision`: → **0.29.0** - `triton`: → **3.8.0** 2.14.0 is published on `download.pytorch.org/whl/test/` but not yet on the release index / PyPI, so the index URLs point at the test channel. This mirrors the 2.13.0 test PR #45731. ### Verified availability (test channel) before pinning - `torch-2.14.0+cu130`: present - `torchvision-0.29.0`: present - `triton==3.8.0`: present, and is exactly what `pytorch` `release/2.14` pins in `.ci/docker/triton_version.txt` - `torchaudio`: newest published is still **2.11.0** on every channel, so that pin is unchanged ### ROCm `requirements/build/rocm.txt` moves from the `rocm7.1` index to `test/rocm7.2`. `rocm7.1` has no torch 2.14.0 wheels — 2.14 is built for `rocm7.2` and `rocm7.14` only. That file was also still on torch 2.11.0 / triton 3.6.0, so this brings it in line with `CMakeLists.txt`'s `TORCH_SUPPORTED_VERSION_ROCM`. ### Ray dependency-compatibility check `.buildkite/scripts/check-ray-compatibility.sh` now offers `uv` both the stable and test channels (it already runs with `--index-strategy unsafe-best-match`), so an RC `torch` pin coming from a transitive dep resolves during the RC window instead of failing against a stable-only index. Same fix as #45731. ### Deliberately *not* carried over from #45731 That PR also carried fixes reacting to breakages found during 2.12/2.13 CI. Those are not copied here, since 2.14 should get whatever it actually needs once CI reports: - xfails for pytorch#184431, pytorch#187735, pytorch#187727 - the ROCm source-build override + `install_flash_attn_rocm.sh` (needed because ROCm/pytorch had no `release/2.13` branch) - `verify_torch_wheel.py` / `UV_NO_VERIFY_HASHES` RC-respin handling `requirements/build/tpu.txt` is also left alone — TPU torch is bumped separately (#50412). ## Test Plan Full CI (apply the `ready` label to trigger the Buildkite run). ## BC-breaking? No — test-channel version bump only. *Authored with AI assistance; modeled on #45731.*", "url": "https://github.com/vllm-project/vllm/pull/52183", "createdAt": "2026-08-13T14:56:49Z", "updatedAt": "2026-08-13T17:57:44Z", "timestamp": "2026-08-13T17:57:44Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "cpu", "nvidia" ], "author": "atalman", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52184", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "feat(inkling): add Eagle3 hidden state extraction support", "text": "## Summary Enable speculative decoding training for Inkling models by adding Eagle3 hidden state extraction support. This PR addresses three interconnected issues that prevented hidden state collection from working end-to-end: - **`vllm/models/inkling/nvidia/model.py`** -- Added `EagleModelMixin` to `InklingModel` and `SupportsEagle3` to `_TmlForCausalLMBase`. Modified `InklingModel.forward()` to collect auxiliary hidden states via `_maybe_add_hidden_state`. Without this, vLLM raises \"Model does not support EAGLE3 interface\" when trying to extract hidden states for speculative decoding training. - **`vllm/v1/core/kv_cache_utils.py`** -- Fixed KV cache page size handling for hidden state cache specs. The hidden state cache (e.g. 7 layers x 4096 dim x 2 bytes = 57344 bytes/token) can exceed the regular KV cache page size. `get_uniform_page_size` now returns `max(page_sizes)` instead of asserting uniformity, and hidden state alignment uses `max(common_page, actual_page)` for `page_size_padded`. - **`vllm/v1/worker/gpu_model_runner.py`** -- Fixed KV cache tensor reshape for non-uniform page sizes. In `_reshape_kv_cache_tensors`, raw tensors allocated with `max(page_sizes)` across groups couldn't always be evenly divided by each spec's native page size. Fixed by falling back to `kv_cache_config.num_blocks` and slicing oversized tensors before reshape. ## Test plan - [ ] Verify Inkling model loads with Eagle3 hidden state extraction enabled (no \"Model does not support EAGLE3 interface\" error) - [ ] Verify KV cache allocation succeeds with mixed page sizes (hidden state cache + regular KV cache) - [ ] Verify KV cache tensor reshape handles non-uniform page sizes without assertion errors - [ ] Run speculative decoding training data generation with Inkling model end-to-end 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52184", "createdAt": "2026-08-13T15:06:49Z", "updatedAt": "2026-08-13T15:48:25Z", "timestamp": "2026-08-13T15:48:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation", "new-model", "ci/build", "qwen", "cpu", "nvidia", "quantization", "kimi", "k3" ], "author": "orestis-z", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52185", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Pixtral: use packed multimodal encoder attention", "text": "## Purpose Fixes #52180 Pixtral currently concatenates all image patch sequences in an encoder batch. When xFormers is unavailable, it constructs a dense block-diagonal mask and applies SDPA to the combined sequence, causing latency and memory use to scale poorly with the number of images. This PR routes both Pixtral vision implementations through vLLM's `MMEncoderAttention` with per-image cumulative sequence lengths. It: - preserves image isolation without constructing a dense combined-sequence mask; - honors the existing multimodal encoder backend selection for FlashAttention, FlashInfer, Torch SDPA, Triton attention, and ROCm AITER; - delegates backend-specific sequence metadata handling to `MMEncoderAttention`; and - removes the remaining Pixtral-specific direct xFormers path. There are no checkpoint, preprocessing, output-shape, token-count, or user-facing API changes. ## Test Plan ```bash pre-commit run --files \\ vllm/model_executor/models/pixtral.py \\ tests/models/multimodal/generation/test_pixtral.py pytest -q tests/models/multimodal/generation/test_pixtral.py \\ -k packed_sequence_metadata ``` GPU validation additionally covered: - Torch SDPA, FlashAttention, and FlashInfer metadata and forward paths; - mixed-size image isolation; - 24-layer encoder parity against Torch SDPA; - controlled before/after encoder benchmarks; and - full Mistral3 vLLM inference with two differently sized images. ## Test Result - Pre-commit hooks, Ruff, mypy, SPDX, and repository checks: passed. - Packed metadata tests: `3 passed`. - Full Mistral3 smoke test: passed; the requested FlashAttention vision backend was selected and generation completed. ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [X] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52185", "createdAt": "2026-08-13T15:15:43Z", "updatedAt": "2026-08-13T15:20:02Z", "timestamp": "2026-08-13T15:20:02Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "multi-modality", "mistral" ], "author": "oliverholworthy", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52186", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][Bugfix] Fix XPU crash with speculative decoding + prefix caching in Mamba models", "text": "## Purpose Model tested: cyankiwi/Qwen3.6-27B-AWQ-INT4 HW Backend: XPU Hardware: Intel ARC Pro B70 vllm version: 0.27.1 If I only enable prefix caching, I see no error and vllm can process requests. If I only use spec decoding(MTP) that works fine too. But if I use both together I get following error: ``` ValueError: Overflow when unpacking long long File \"vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context self.state_base_addrs[idx] = state.data_ptr() ``` Root cause: On XPU, `tensor.data_ptr()` can return memory addresses >= 2^63 — exceeding the signed `int64` range. The `MambaSpecDecodeGPUContext` class stores these addresses via element-wise assignment (`tensor[idx] = value`), which fails because PyTorch's `__setitem__` converts the scalar through C-level long() before any dtype check, regardless of the tensor's declared dtype. Fix applied: Two changes, both in `vllm/v1/worker/mamba_utils.py`: 1. Change dtype from int64 to uint64 for state_base_addrs and block_table_ptrs — matching the existing pattern in block_table.py (# NOTE(woosuk): Use uint64 instead of int64 to cover all possible addresses). 2. Collect pointers into a list and bulk-create the tensor instead of element-wise assignment. torch.tensor(list, dtype=torch.uint64) correctly handles large pointer values; tensor[idx] = large_int does not (even with uint64 dtype). ## Test Plan Re-build and run the same model with both `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'` and `--enable-prefix-caching`. Send a request and it should not throw following error. Logs before: ``` vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.1.dev19912+g373592ef5.d20260813) with config: model='cyankiwi/Qwen3.6-27B-AWQ-INT4', speculative_config=SpeculativeConfig(method='mtp', model='cyankiwi/Qwen3.6-27B-AWQ-INT4', num_spec_tokens=3), tokenizer='cyankiwi/Qwen3.6-27B-AWQ-INT4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=True, dtype=torch.float16, max_seq_len=138000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=cyankiwi/Qwen3.6-27B-AWQ-INT4, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '/root/.cache/vllm/torch_compile_cache/bbdfd623a8', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': '/root/.cache/vllm/torch_compile_cache/bbdfd623a8/rank_0_0/eagle_head', 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto'), vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-bd930efe188b6bb9-b4ce8602,prompt_token_ids_len=14926,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=16384, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=False, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([0, 0, 0, 0, 1, 2, 3, 4], [0, 0, 0, 0, 5, 6, 7, 8], [0, 0, 0, 0, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={chatcmpl-bd930efe188b6bb9-b4ce8602: 8000}, total_num_scheduled_tokens=8000, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 6], finished_req_ids=[], free_encoder_mm_hashes=[], scheduled_encoder_input_stats=null, preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null, ec_manager_metadata=null, new_block_ids_to_zero=[13, 14, 15, 16, 17, 18], kv_cache_block_copies=null, partial_tail_offloads=null, num_spec_tokens_to_schedule=3) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.17307692307692313, iteration_details=None, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=14926, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] EngineCore encountered a fatal error. vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] Traceback (most recent call last): vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1337, in run_engine_core vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] engine_core.run_busy_loop() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/fault_tolerance/engine_core_sentinel.py\", line 179, in run_with_fault_tolerance vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] busy_loop_func(self) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1381, in run_busy_loop vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] self._process_engine_step() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1434, in _process_engine_step vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] outputs, model_executed = self.step_fn() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 652, in step_with_batch_queue vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] exec_future = self.model_executor.execute_model( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 131, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] output.result() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return self.__get_result() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] raise self._exception vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 109, in collective_rpc vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] result = run_method(self.driver_worker, method, args, kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py\", line 510, in run_method vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py\", line 351, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return self.worker.execute_model(scheduler_output) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 1086, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] output = self.model_runner.execute_model( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4438, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] mamba_utils.preprocess_mamba( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 1190, in preprocess_mamba vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] fused.ctx.initialize_from_forward_context( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] self.state_base_addrs[idx] = state.data_ptr() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ~~~~~~~~~~~~~~~~~~~~~^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ValueError: Overflow when unpacking long long vllm-server | (EngineCore pid=284) INFO 08-13 12:20:39 [xpu_worker.py:167] XPUWorker shutdown: cleaning up (rank=0, local_rank=0) vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] AsyncLLM output_handler failed. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] Traceback (most recent call last): vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 690, in output_handler vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] outputs = await engine_core.get_output_async() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py\", line 1104, in get_output_async vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] raise self._format_exception(outputs) from None vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] Error in chat completion stream generator. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] Traceback (most recent call last): vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py\", line 488, in chat_completion_stream_generator vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] async for res in result_generator: vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 607, in generate vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] out = q.get_nowait() or await q.get() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] ^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/output_processor.py\", line 85, in get vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] raise output vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 690, in output_handler vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] outputs = await engine_core.get_output_async() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py\", line 1104, in get_output_async vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] raise self._format_exception(outputs) from None vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. vllm-server | (EngineCore pid=284) INFO 08-13 12:20:40 [xpu_worker.py:177] XPUWorker shutdown: done (rank=0, local_rank=0) vllm-server | (EngineCore pid=284) Process EngineCore: vllm-server | (EngineCore pid=284) Traceback (most recent call last): vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/multiprocessing/process.py\", line 314, in _bootstrap vllm-server | (EngineCore pid=284) self.run() vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/multiprocessing/process.py\", line 108, in run vllm-server | (EngineCore pid=284) self._target(*self._args, **self._kwargs) vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1348, in run_engine_core vllm-server | (EngineCore pid=284) raise e vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1337, in run_engine_core vllm-server | (EngineCore pid=284) engine_core.run_busy_loop() vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/fault_tolerance/engine_core_sentinel.py\", line 179, in run_with_fault_tolerance vllm-server | (EngineCore pid=284) busy_loop_func(self) vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1381, in run_busy_loop vllm-server | (EngineCore pid=284) self._process_engine_step() vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1434, in _process_engine_step vllm-server | (EngineCore pid=284) outputs, model_executed = self.step_fn() vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 652, in step_with_batch_queue vllm-server | (EngineCore pid=284) exec_future = self.model_executor.execute_model( vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 131, in execute_model vllm-server | (EngineCore pid=284) output.result() vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result vllm-server | (EngineCore pid=284) return self.__get_result() vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result vllm-server | (EngineCore pid=284) raise self._exception vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 109, in collective_rpc vllm-server | (EngineCore pid=284) result = run_method(self.driver_worker, method, args, kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py\", line 510, in run_method vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py\", line 351, in execute_model vllm-server | (EngineCore pid=284) return self.worker.execute_model(scheduler_output) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 1086, in execute_model vllm-server | (EngineCore pid=284) output = self.model_runner.execute_model( vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4438, in execute_model vllm-server | (EngineCore pid=284) mamba_utils.preprocess_mamba( vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 1190, in preprocess_mamba vllm-server | (EngineCore pid=284) fused.ctx.initialize_from_forward_context( vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context vllm-server | (EngineCore pid=284) self.state_base_addrs[idx] = state.data_ptr() vllm-server | (EngineCore pid=284) ~~~~~~~~~~~~~~~~~~~~~^^^^^ vllm-server | (EngineCore pid=284) ValueError: Overflow when unpacking long long vllm-server | (APIServer pid=1) INFO: Shutting down vllm-server | (APIServer pid=1) INFO: Waiting for application shutdown. vllm-server | (APIServer pid=1) INFO: Application shutdown complete. vllm-server | (APIServer pid=1) INFO: Finished server process [1] ``` ## Test Result Mechanism: Locally built docker image with the applied change. vllm command: ``` vllm serve cyankiwi/Qwen3.6-27B-AWQ-INT4 --port 8000 --host 0.0.0.0 --trust-remote-code --tensor-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 138000 --kv-cache-dtype fp8 --max-num-seqs 1 --trust-remote-code --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --limit-mm-per-prompt '{\"image\": 0, \"video\": 0}' --generation-config vllm --override-generation-config '{\"temperature\": 0.6, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"repetition_penalty\": 1.0}' --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' --max_num_batched_tokens 8192 --enable-prefix-caching ``` Result: No errors Sample logs: ``` vllm-server | (APIServer pid=1) INFO: 127.0.0.1:45370 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK vllm-server | (APIServer pid=1) INFO 08-13 14:45:21 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% vllm-server | (APIServer pid=1) INFO 08-13 14:45:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.87, Accepted throughput: 19.20 tokens/s, Drafted throughput: 20.10 tokens/s, Accepted: 192 tokens, Drafted: 201 tokens, Per-position acceptance rate: 1.000, 0.955, 0.910, Avg Draft acceptance rate: 95.5% vllm-server | (APIServer pid=1) INFO 08-13 14:45:31 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.3%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:45:51 [loggers.py:310] Engine 000: Avg prompt throughput: 3560.5 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:45:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.55, Accepted throughput: 2.80 tokens/s, Drafted throughput: 3.30 tokens/s, Accepted: 84 tokens, Drafted: 99 tokens, Per-position acceptance rate: 0.879, 0.848, 0.818, Avg Draft acceptance rate: 84.8% vllm-server | (APIServer pid=1) INFO 08-13 14:46:01 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 59.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:46:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.97, Accepted throughput: 44.79 tokens/s, Drafted throughput: 45.29 tokens/s, Accepted: 448 tokens, Drafted: 453 tokens, Per-position acceptance rate: 1.000, 0.993, 0.974, Avg Draft acceptance rate: 98.9% ``` AI Disclosure: Have taken assistance of same model i.e. cyankiwi/Qwen3.6-27B-AWQ-INT4 via Claude code. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52186", "createdAt": "2026-08-13T15:42:58Z", "updatedAt": "2026-08-13T15:44:32Z", "timestamp": "2026-08-13T15:44:32Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "swakhandekar", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52187", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Perf] Zero new KV blocks by cache group", "text": "## Purpose #51749 correctly generalized recycled KV-page zeroing to every allocating `AttentionSpec`, preventing stale FP8 sliding-window pages. However, the scheduler currently flattens the newly allocated block IDs from every cache manager, and `KVBlockZeroer` applies that union to every attention-cache segment. On a hybrid or otherwise multi-group layout this creates unnecessary `all new block IDs x all cache groups` work. This PR preserves the cache-group association through `SchedulerOutput`, builds zeroing metadata per group, and zeros each newly allocated block only in its owning group's physical cache pages. It retains #51749's correctness coverage and #52058's bounded 3-D launch geometry. This is generic V1 KV-cache infrastructure, not an SM120-specific path. The DeepSeek-V4 SM120 configuration below is the reproducer because it has five active MLA/sliding-window managers. ### Relationship to adjacent work - #51749 supplies the required all-`AttentionSpec` correctness coverage. - #52058 fixed the major launch-geometry scaling problem and restored most of the prefill regression by using wide masked chunks on a 3-D grid. - #50485 is alternative 3-D launch-geometry work. - #52062 proposes reverting #51749; this PR instead preserves its stale-page fix while reducing the remaining redundant work. Open PR and issue searches for `KVBlockZeroer group zeroing`, `new_block_ids_to_zero`, `zeroing \"cache group\"`, and `\"owning group\" KV cache` found no equivalent group-aware change. ## Test Plan ```bash python -m pytest -q tests/v1/worker/test_kv_block_zeroer.py python -m pytest -q \\ tests/v1/kv_connector/unit/test_nixl_connector_hma.py::test_scheduler_filters_connector_loaded_blocks_from_zeroing \\ tests/v1/kv_connector/unit/test_nixl_connector_hma.py::test_failed_load_rezeroes_unwritten_skipped_blocks \\ tests/v1/worker/test_kv_block_zeroer.py::test_large_dsv4_launch_geometry \\ tests/v1/core/test_single_type_kv_cache_manager.py::test_sliding_window_records_new_blocks_for_zeroing \\ tests/v1/core/test_single_type_kv_cache_manager.py::test_chunked_local_attention_records_new_blocks_for_zeroing ruff format --check ruff check ``` Live A/B: DeepSeek-V4-Flash-0731, 2x RTX PRO 6000 Blackwell, TP2 + EP, FP8 KV, five cache managers, max model length 1M, FlashInfer 0.6.17. Both arms use the same rebuilt tree and dependency set; the control is the parent commit and the test arm adds only this commit. Each size has five sequential C=1 runs. ## Test Result - KV block zeroer GPU suite: `9 passed` - Focused scheduler/manager/geometry suite: `5 passed` - MLA gather compatibility slice: `34 passed, 2437 deselected` - Ruff format/check: passed - Cold-load non-thinking HTTP smoke: `200`, exact `OK.` Matched median prefill results after #52058: | Prompt | Control TTFT | This PR TTFT | Control tok/s | This PR tok/s | | ---: | ---: | ---: | ---: | ---: | | 128 | 166 ms | 156 ms (-6.0%) | 919 | 966 (+5.1%) | | 1K | 158 ms | 155 ms (-1.9%) | 6,641 | 6,753 (+1.7%) | | 4K | 498 ms | 503 ms (+1.0%) | 8,268 | 8,189 (-1.0%) | | 16K | 2.221 s | 2.224 s (+0.1%) | 7,382 | 7,376 (-0.1%) | The benefit is concentrated in short-prefill latency; 4K and 16K are flat within about 1%. Three live decode runs averaged 304.7 tok/s, unchanged from the prior 304.0 tok/s five-run result. AI assistance was used to bisect the original regression, implement the change, and draft this description. I reviewed the diff and ran the validation above.", "url": "https://github.com/vllm-project/vllm/pull/52187", "createdAt": "2026-08-13T16:06:09Z", "updatedAt": "2026-08-13T16:06:45Z", "timestamp": "2026-08-13T16:06:45Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kv-connector", "mrv2" ], "author": "alexbi29", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52188", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec decode] Support Kimi-K3 DCP with DSpark", "text": "## Purpose This PR adds support for running Kimi-K3 decode context parallel with DSpark with FlashinferMLA and Tokenspeed as target causal attention backend and Tokenspeed as the draft non-causal backend. ## Test Plan Kimi K3 GSM8k with the different backend combination. ## Test Result Default (no backend specified): ``` vllm serve moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 \\ -dcp 8 \\ --load-format fastsafetensors \\ --no-enable-flashinfer-autotune \\ --trust-remote-code \\ --language-model-only \\ --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\ --kv-cache-dtype fp8 \\ --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"rejection_sample_method\":\"standard\"}' |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9606|± |0.0054| | | |strict-match | 5|exact_match|↑ |0.9606|± |0.0054| ``` FlashinferMLA target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9613|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9613|± |0.0053| ``` Tokenspeed target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9621|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9621|± |0.0053| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52188", "createdAt": "2026-08-13T16:10:03Z", "updatedAt": "2026-08-13T16:39:35Z", "timestamp": "2026-08-13T16:39:35Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia", "mrv2", "kimi", "k3" ], "author": "wzhao18", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52189", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Fail fast when MLA head dimensions are not supported by the CPU decode kernel", "text": "## Purpose Partial implementation of **option B** from #52008: fail fast with a clear error when an MLA model's head dimensions are not supported by the CPU MLA decode kernel, instead of crashing deep inside kernel dispatch after a confusing chain of contradictory mamba / prefix-caching validation errors. The CPU MLA decode kernel (`csrc/cpu/mla_decode.cpp`) only compiles for the DeepSeek-V2/V3 cache layout (`head_dim=576, v_head_dim=512, block_size=16`). Models with different MLA head dimensions — e.g. the `inference-optimization/Kimi-K3-0.40B` tiny variant (`head_dim=160, v_head_dim=64`) — cannot run on the CPU backend at all, but previously only failed deep in kernel dispatch with an unrelated-looking error. Now `CpuPlatform.check_and_update_config` validates the dimensions at startup and raises: ``` ValueError: The CPU MLA decode kernel only supports head_dim=576, v_head_dim=512 (block_size=16); got head_dim=160, v_head_dim=64. Model cannot run on the CPU backend. ``` DeepSeek-V2/V3 standard dimensions still pass the guard and keep running; the existing MLA handling (forcing chunked prefill and prefix caching off) is unchanged. This PR does **not** implement option A (CPU MLA prefix caching / chunked prefill) or option C (honoring `--hf-overrides {\"use_mla\": false}`). This is deliberately more precise than the architecture-level fail-fast suggested in #52008: only models whose MLA head dimensions the CPU decode kernel cannot run are rejected, so DeepSeek-V2/V3 with standard dimensions remain supported on CPU. Complementary to #52045 (which works around the contradictory mamba validation so supported configs can start); #52045 explicitly lists option B as out of scope, so there is no overlap. ## Test Plan ``` .venv/bin/python -m pytest tests/config/test_cpu_mla_fail_fast.py .venv/bin/python -m pytest tests/config/test_multimodal_config.py .venv/bin/python -m ruff check vllm/platforms/cpu.py tests/config/test_cpu_mla_fail_fast.py .venv/bin/python -m ruff format --check vllm/platforms/cpu.py tests/config/test_cpu_mla_fail_fast.py ``` Local synthetic config only (EngineArgs + stubbed `hf_text_config`). No weight download, no GPU, no serve/generate. ## Test Result ``` 3 passed tests/config/test_cpu_mla_fail_fast.py 42 passed tests/config/test_multimodal_config.py (regression) All checks passed! (ruff check) 2 files already formatted (ruff format) ``` Manual verification with the real Kimi-K3-0.40B config (cached locally, `use_mla=True, kv_lora_rank=128, qk_rope_head_dim=32, v_head_dim=64`): `CpuPlatform.check_and_update_config` raises the fail-fast error above at startup; the DeepSeek-V3 layout (512/64/512) passes the guard and keeps the existing behavior (chunked prefill and prefix caching forced off). ## AI Assistance This PR was prepared with AI assistance (deepseek). The human submitter reviewed every changed line and realize the detail. **Why this is not a duplicate:** searched open PRs referencing #52008 before opening. #52045 exists but implements a different fix (mamba cache-mode validation workaround, so supported configs can start) and explicitly lists option B — a hard \"architecture not supported on CPU\" fail-fast — as out of scope. This PR is the complementary option B. **Model evaluation:** N/A. This change is a startup-time configuration guard; it does not affect inference output, accuracy, or serving behavior. ## Docs None. This does not make `Kimi-K3-0.40B` a supported CPU serving path — it turns a deep confusing crash into a clear startup error.", "url": "https://github.com/vllm-project/vllm/pull/52189", "createdAt": "2026-08-13T16:14:27Z", "updatedAt": "2026-08-13T16:15:10Z", "timestamp": "2026-08-13T16:15:10Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "cpu" ], "author": "kunkunblueberry", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52190", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage)", "text": "# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage) ## Purpose Kimi-K3's model classes carry no `@support_torch_compile` and vLLM logs: ``` `torch.compile` is turned on, but the model /path/to/Kimi-K3 does not support it. Please open an issue on GitHub if you want it to be supported. ``` Because there is no inductor post-grad graph, **none of the fusion passes ever run for K3** — even though they are enabled in the config and reported as active at startup: ``` INFO [config/compilation.py] Enabled custom fusions: norm_quant, act_quant, allreduce_rms, mla_dual_rms_norm ... 'pass_config': {..., 'fuse_mla_dual_rms_norm': True, 'fuse_allreduce_rms': True, ...} ``` In particular `MLADualRMSNormFusionPass` already knows how to fuse the paired MLA `q_a_layernorm` + `kv_a_layernorm` into AITER's `fused_qk_rmsnorm` kernel, but it never sees a graph. Confirmed by instrumenting `VllmFusionPatternMatcherPass.__call__`: zero pass invocations and zero pattern dumps on an otherwise fully-configured run. This PR enables compilation so the existing passes can do their job. Two of the three changes are **correctness fixes** that compilation exposes rather than optimizations. ## Changes 1. **`vllm/models/kimi_k3/amd/linear.py`** — add `@support_torch_compile` to `KimiLinearModel`. Its `__init__(self, *, vllm_config, prefix)` and `forward` signatures already match the shape the decorator expects (same as `DeepseekV2Model`). 2. **`vllm/models/kimi_k3/amd/kda.py` + `vllm/config/compilation.py`** — wrap the KDA conv1d + recurrent/chunk core in a `kimi_kda_attention_core` custom op declaring `mutates_args=[\"core_attn_out\"]`, and add `vllm::kimi_kda_attention_core` to the default `splitting_ops`. **This is required for correctness, not perf.** `KimiGatedDeltaNetAttention._forward` writes its result in place into a caller-allocated `core_attn_out` and returns `None`, and was not a registered custom op. Under fullgraph capture the call therefore has no compiler-visible effect and the entire linear-attention path is eliminated: with only the decorator applied, **all KDA kernels for all 69 KDA layers stop executing** (K3 is 24 MLA + 69 KDA), while the surrounding `o_proj` and MoE keep running on the uninitialized `torch.empty` buffer. It fails silently — no exception, no graph-break warning, just degenerate output. Follows the existing `vllm::qwen_gdn_attention_core` pattern. 3. **`vllm/models/kimi_k3/amd/ops/attn_res.py`** — register the attention-residual Triton launcher as `vllm::kimi_attn_res` with `mutates_args=[\"blocks\"]`. Two problems, one op. First, the launcher picks its tile shape with a host-side branch on the token count: ```python if num_tokens >= 256 or num_blocks <= 1: block_l, num_warps = 1, 4 else: block_l, num_warps = 4, 8 ``` Dynamo traced this, evaluated it on the large profile-run batch and emitted the dynamic-shape guard `s72 >= 256`. vLLM's wrapper drops guards, so the artifact specialized for the `>= 256` branch was silently reused for small decode batches. Second, with `block_write_idx >= 0` the kernel stores its prefix sum into `blocks` in place (`WRITE_BLOCK`), which is invisible to the compiler unless declared. Both are fixed by making this a custom op with the mutation declared; found via `TORCH_TRACE` + `tlparse`, reading the `user_stack` field of the raw `guard_added_fast` records. 4. **`vllm/models/kimi_k3/amd/latent_moe_runner.py`** — fullgraph capture cannot trace `logging.Logger` methods ([gb0291](https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0291.html)): ``` torch._dynamo.exc.Unsupported: logging.Logger method not supported for non-export cases Developer debug context: method: .info_once from user code: latent_moe_runner.py:67 in _shard_up_proj_tail ``` The one-shot diagnostic in `_shard_up_proj_tail` is hoisted into `__init__` under the same `self._tail_shardable` condition that gates its only call site, so the message is preserved and both the logger call and the `_logged_sharded_tail` state mutation leave the traced hot path. **No changes to any fusion pass or pattern.** The existing DeepSeek-shaped MLA dual-RMSNorm pattern matches K3 unmodified. ## Test Plan Kimi-K3, MI355X, TP8, MXFP4 weights, AITER enabled. Two stacks: - **A — upstream nightly, no spec decoding**, `--kv-cache-dtype fp8_e4m3`. Decode-only torch-profiler capture, three arms on identical settings (ISL 2048 / OSL 96 / concurrency 16, `wait_iterations=40`, `active_iterations=2`). All windows verified pure decode — `execute_context_0(0)_generation_16(16)`, zero prefill-attention kernels. - **B — AMD downstream K3 stack with DSpark MTP + CUDA graphs** (`FULL_AND_PIECEWISE`, `--kv-cache-dtype fp8`, `num_speculative_tokens=2`), i.e. the configuration this model is actually served in. Added because compilation interacts with cudagraph capture and speculative decoding, and stack A exercises neither. Using agentX coni - **Accuracy** — gsm8k via lm-eval against `/v1/completions`. ## Test Result ### Fusion passes now run (stack A) Cumulative match table for one full model forward, per rank: ``` mla_dual_rms_norm_fusion_pass : 24 <- 1 per MLA layer (K3 has 24) rocm_aiter_allreduce_fusion_pass : 92 <- ~1 per layer (K3 has 93) activation_quant_fusion_pass : 0 rocm_aiter_silu_mul_fp8_group_quant_fusion_pass: 0 RocmAiterRMSNormQuantFusionPass : 0 ``` `24` is exactly the MLA layer count, i.e. single coverage with no double counting. Cross-check against the per-graph-piece `Replaced N patterns` lines: 16x1 + 8x2 + 176x4 = 736 across 8 ranks = 92/rank. Identical with and without the KDA custom op, i.e. the new splitting op does not disturb the fusions. ### Kernel counts, pure-decode window (stack A, 2 steps, rank 0) | kernel group | eager | decorator only | **this PR** | |---|---|---|---| | MLA attention (24 x 2) | 48 | 48 | **48** | | MoE `mfma_moe1` (92 x 2) | 184 | 184 | **184** | | KDA `causal_conv1d` + `fused_recurrent_kda` | 276 | **0** ❌ | **276** ✅ | | `aiter::fused_qk_rmsnorm_kernel` (24 x 2) | 0 | 48 | **48** | | `add_rmsnorm_quant` (all) | 282 | 2 | **2** | | `aiter::allreduce_fusion_kernel_1stage` | 0 | 184 | **184** | The middle column is the decorator applied *without* change (2) — included to show the silent KDA loss that change (2) fixes. Corroborated independently of the profiler by the server log: stock and this PR both log `Triton kernel JIT compilation during inference:` for `chunk_kda_fwd_kernel_intra_sub_chunk`, `fused_recurrent_kda_fwd_kernel`, `_causal_conv1d_fwd_kernel` and seven more (223 / 224 lines); the decorator-only build logs **none** of them. ### Norm kernels in detail (stack A) | Norm kernel | eager | this PR | |---|---|---| | `add_rmsnorm_quant` — `q_a_layernorm` | 48 x 230.392 us | — | | `add_rmsnorm_quant` — `kv_a_layernorm` | 48 x 218.353 us | — | | **`aiter::fused_qk_rmsnorm_kernel`** | — | **48 x 231.194 us** | | `add_rmsnorm_quant` — per-layer input/post-attn | 184 x 912.578 us | — (folded into allreduce fusion) | | `add_rmsnorm_quant` — final norm | 2 x 10.718 us | 2 x 11.078 us | 96 launches / 448.745 us collapse into 48 launches / 231.194 us of `aiter::fused_qk_rmsnorm_kernel`, and the 184 standalone per-layer norms are absorbed by the 92 allreduce+RMSNorm fusions. Decode is where this matters: AITER's `_fused_qk_rmsnorm` wrapper deliberately falls back to two separate `rmsnorm` calls at `m >= 16384` tokens, so the fused kernel is the decode / small-batch path. ### Works under CUDA graphs + speculative decoding (stack B) Pure-decode capture, DSpark MTP, concurrency 2 (verify batch `2*CONC*(1+num_spec)` = 12), `FULL_AND_PIECEWISE`, 6 engine steps, rank 0. Window verified pure decode: `execute_context_0(0)_generation_2(6)` x12, no prefill kernels. | kernel group | launches | per step | expected | |---|---|---|---| | `aiter::fused_qk_rmsnorm` | 144 | 24 | 24 MLA layers ✅ | | `allreduce_fusion_kernel_1stage` | 558 | 93 | 93 layers ✅ | | `fused_recurrent_kda` + `causal_conv1d_update` | 828 | 69 + 69 | 69 KDA layers ✅ | | MLA `mla_a8w8_qh16_qseqlen4_gqaratio16_v3_ps` | 144 | 24 | 24 ✅ | The KDA path stays fully alive at 69 layers/step with compilation, cudagraph capture and MTP all active simultaneously, and the QK fusion covers exactly all 24 MLA layers. Trace showing Cuda graph + MTP + compiler pass fusions active \"image\" ### Accuracy — no regression | stack | harness | baseline | this PR | |---|---|---|---| | A (no spec decode) | gsm8k, full 1319, 5-shot, greedy | 0.9098 ± 0.0079 | **0.9257 ± 0.0072** | | B (DSpark MTP + cudagraph) | gsm8k, 250-sample, 5-shot | 0.988 | **0.984** | Stack A's delta is within ~1.5 combined standard errors. Stack B was run with `rejection_sample_method=block`; the `synthetic` variant is a throughput approximation and is not a valid accuracy configuration. ## Known limitation: CUDA graph capture ladder (stack B) On the AMD downstream MTP stack, enabling compilation narrows the range of CUDA graph capture sizes that can be captured. Outside that range the **per-size warmup forward** — before any capture begins, descending from the largest size — dies with `Memory access fault by GPU node-N`. **Root cause is not yet understood.** It is reported here rather than left for a reviewer to hit. Note this is observed only on the downstream stack — the DSpark MTP method and that cudagraph path are not upstream — so it may not be reachable from an upstream configuration. It does not affect stack A, which captures the default ladder without incident.", "url": "https://github.com/vllm-project/vllm/pull/52190", "createdAt": "2026-08-13T16:57:07Z", "updatedAt": "2026-08-13T16:57:46Z", "timestamp": "2026-08-13T16:57:46Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "rbrugaro-amd", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52191", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Support FP16/BF16 persisted GDN state on AMX", "text": "## Purpose BF16/FP16 GDN state reduces the aligned block size used by hybrid prefix caching and therefore reduce the prefix-caching glassjaw on CPU AMX backends. ## Test Plan Server ```bash vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \\ --kv-cache-dtype {auto|fp8} \\ --mamba-ssm-cache-dtype {auto|bfloat16} \\ --language-model-only \\ --enable-prefix-caching \\ ``` Client ```bash KV_CACHE_DTYPE=auto MAMBA_SSM_CACHE_DTYPE=auto for PREFIX_LEN in 640 1152 2176 3200 3456 3500 4000 4352; do curl -fsS -X POST http://127.0.0.1:8000/reset_prefix_cache vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B-FP8 \\ --num-prompts 512 \\ --max-concurrency 128 \\ --dataset-name prefix_repetition \\ --prefix-repetition-prefix-len \"$PREFIX_LEN\" \\ --prefix-repetition-suffix-len 200 \\ --prefix-repetition-output-len 128 \\ --prefix-repetition-num-prefixes 1 done ``` ## Test Result BF16 GDN state throughput improvement over auto/FP32 GDN state, measured as the geometric mean across all eight prefix lengths: | KV cache dtype | Output-token throughput | | --- | ---: | | `auto` | +35.18% | | `fp8` | +37.51% | Align mode uses a common block size for attention KV and recurrent state. In this run, the effective blocks were: | KV cache dtype | Auto GDN block | BF16 GDN block | | --- | ---: | ---: | | `auto` | 1152 tokens | 640 tokens | | `fp8` | 2176 tokens | 1152 tokens | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52191", "createdAt": "2026-08-13T16:57:49Z", "updatedAt": "2026-08-13T16:58:27Z", "timestamp": "2026-08-13T16:58:27Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "cpu" ], "author": "tianmu-li", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52192", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add tuned fused_moe config for NVIDIA GB10 (Nemotron-3-Super shape, E=512 N=2688)", "text": "**Problem.** No `E=512,N=2688,device_name=NVIDIA_GB10` config exists: every Nemotron-3-Super serve on a DGX Spark falls back to the default heuristic and logs \"Using default MoE config. Performance might be sub-optimal!\" Second config for this device, first for this shape (#45949 covers E=512,N=512 fp8). **What's in the file.** Decode keys M=1,2,4,8 tuned on hardware; rows at M=16,32,64,256,1024,2048,4096 pinned to the default heuristic's own per-M choices. The pins are required: `try_get_optimal_moe_config` snaps to the nearest key with no fallback, and a tuned-keys-only ladder measured −2.35% on pp2048 (prefill-scale M landing on a small-M config); the pinned rows make large-M lookups exactly-stock. The ladder stops at M=8 because the stock tuner cannot complete this shape on unified memory (companion issue to follow — [#MEMORY-ISSUE]); extension is mechanical. **Evidence.** - Kernel-level A/B (CUDA-event, same boot): +2.3% to +5.2% per decode key, **+3.5% geomean** over the default heuristic. - Serve-level paired A/B (12 boots): pp2048 −0.59% [−1.74, +0.55] (n=32/arm); serve decode a wash (+0.09% [−2.56, +2.75]). We claim the kernel-level win only, not an end-to-end speedup. - The startup warning clears with the config present (verified in the leg-B serve log). - Environment: vLLM 83ad767ee, driver 610.57.04, GB10 121 GB unified.", "url": "https://github.com/vllm-project/vllm/pull/52192", "createdAt": "2026-08-13T17:02:39Z", "updatedAt": "2026-08-13T17:03:18Z", "timestamp": "2026-08-13T17:03:18Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "nvidia" ], "author": "maxspevack", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52193", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "diagnostic patch created covering all workspace creation", "text": "## Purpose Checks #52023 ## Test Plan This is still in WIP, it is for logs from reporter. ## Test Result WIP ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52193", "createdAt": "2026-08-13T17:14:42Z", "updatedAt": "2026-08-13T17:15:17Z", "timestamp": "2026-08-13T17:15:17Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia" ], "author": "khushali9", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52194", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[LoRA][MoE] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts", "text": "# [LoRA][MoE] Fix PEFT 0.18+ `target_parameters` LoRA loading for 3D MoE experts ## Summary PEFT 0.18+ can train a LoRA against a **fused 3D MoE expert parameter** via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`) rather than against an `nn.Module`. When it does, it reads the parameter `[E, dim1, dim2]` positionally as `[E, in=dim1, out=dim2]`. vLLM's `add_lora_w13` / `add_lora_w2` punica kernels use the opposite `nn.Linear` `[out, in]` convention, so per expert the trained delta is the transpose of what vLLM expects: `ΔW_vllm = ΔW_peft^T`. On stock vLLM this is **not** a silent no-op — `_stack_moe_lora_weights` reshapes the adapter with the wrong dimension semantics and `set_lora` raises a shape mismatch that kills the engine. Serving a real Qwen3.5-35B-A3B (`Qwen3_5MoeForConditionalGeneration`) expert LoRA fails in `vllm/lora/layers/fused_moe.py`: ``` RuntimeError: The size of tensor a (512) must match the size of tensor b (2048) at non-singleton dimension 2 ``` (512 = `moe_intermediate_size`, 2048 = `hidden_size`.) ## Fix 1. **`peft_helper.py`** — parse the `target_parameters` list from `adapter_config.json`. 2. **`lora_model.py`** — propagate `target_parameters` onto `LoRAModel` (constructor, `clone()`, and `from_lora_tensors`). 3. **`model_manager.py`** — in `_stack_moe_lora_weights`, detect the PEFT 3D layout and, when present, swap `lora_A`/`lora_B` and transpose the trailing two dims **after** the existing EP-aware expert slicing (so the fix composes with expert parallelism). ### Why the layout must be *detected*, not assumed `create_dummy_lora_weights` builds dummy MoE LoRA buffers in the **native** layout that shares the same flat shape as the PEFT layout, so an unconditional transpose would corrupt native/dummy adapters. Detection uses: - **Primary signal:** `target_parameters` names a fused expert weight (`experts.gate_up_proj` / `experts.down_proj` / `experts.w13_weight` / `experts.w2_weight`). - **Fallback** (older adapters that don't record `target_parameters`): native `lora_a` trailing dim equals `hidden_size`; the PEFT layout does not. ## Testing All results below were produced against this branch's **source** (editable install, `VLLM_USE_PRECOMPILED=1 uv pip install -e .`) on an H200 — not a monkeypatch over a released wheel. ### Unit tests (added in this change) ``` python -m pytest -v \\ tests/lora/test_peft_helper.py::test_peft_helper_target_parameters_direct \\ tests/lora/test_lora_manager.py::test_target_parameters_indicates_3d_lora \\ tests/lora/test_lora_manager.py::test_from_lora_tensors_propagates_target_parameters \\ tests/lora/test_peft_helper.py::test_peft_helper_pass \\ tests/lora/test_lora_manager.py::test_from_lora_tensors ``` → **10 passed** (the discriminator test is parametrized over 6 cases). The last two are pre-existing tests on adjacent code, included as a regression check. ### End-to-end numerical proof (Qwen3.5-35B-A3B expert LoRA) Bug reproduced: serving the raw PEFT adapter on the **unpatched** loader crashes in `set_lora` with the `512 vs 2048` error above. With the fix, three arms are compared by **chosen-token logprobs** (see caveat): base (no adapter), an independent offline-converted oracle (native layout, self-checks `max|ΔW_vllm − ΔW_peft^T| = 0.000e+00` over 320 experts), and the raw PEFT adapter loaded directly. | prompt | [effect] oracle-vs-base max\\|Δlogprob\\| | [equiv] raw-vs-oracle max\\|Δlogprob\\| | |---|---|---| | chest pain dx | 0.2808 | 0.0000 | | night blindness | 0.1846 | 0.0000 | | neonatal meningitis | 0.1028 | 0.0000 | | diabetic antihypertensive | 0.1601 | 0.0000 | | PKU enzyme | 0.0758 | 0.0000 | - **[effect]** the adapter demonstrably changes output (one prompt even flips the greedy answer ordering). - **[equiv]** the directly-loaded raw adapter is **bit-identical** to the offline-converted oracle across all 32 tokens × 5 prompts, no greedy-path divergence → the in-loader transpose matches the independently-verified transform. **VERDICT: PASS.** **Testing caveat (why logprobs, not decoded text):** for an experts-only (8-of-256, rank-32) adapter, greedy decoded text is a false-negative trap — the adapter shifts logits substantially yet often does not flip the greedy argmax on short prompts, so text can look identical to base even when the adapter is correctly applied. The verdict is therefore numerical. ## Model evaluation This change is the difference between a hard crash and a correctly-applied adapter for the affected adapter class; the numerical equivalence-vs-oracle result above is the output-affecting evidence. Broader `tests/evals/` accuracy runs can be added if reviewers want them. ## AI assistance disclosure AI assistance (Claude) was used to author this change. A human submitter has reviewed every changed line and run the tests above.", "url": "https://github.com/vllm-project/vllm/pull/52194", "createdAt": "2026-08-13T17:29:22Z", "updatedAt": "2026-08-13T17:32:29Z", "timestamp": "2026-08-13T17:32:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "feixiangpeng", "state": "closed", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52195", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Add Stage 1 (CPU-only, reference) BitNet ternary quantization backend", "text": "## Purpose Adds an initial CPU-only, reference implementation of a BitNet b1.58-style ternary ({-1, 0, 1}) weight quantization backend for linear layers (`vllm/model_executor/layers/quantization/bitnet.py`), registered through the existing `register_quantization_config` out-of-tree plugin mechanism (no core files touched). There is real, currently-unclaimed demand for this: #33142 (open feature request), and #18213 / #17279 (closed as \"not planned\" due to inactivity, not rejected on merit). Weight quantization: absmean ternary rule per BitNet b1.58 (Eq. 7), packed 2 bits/value (4 values/byte). ## Scope / what this is NOT This is intentionally a correctness-first Stage 1: online (load-time) quantization of a bf16/fp16 checkpoint into the ternary representation, unpack-then-matmul execution path. It does **not** include a fused Triton/CUDA kernel, reading pre-packed ternary checkpoints, or end-to-end validation against a real BitNet checkpoint (e.g. microsoft/bitnet-b1.58-2B-4T) - all flagged here as follow-up work for a later PR, not included in this change. ## Test Plan Added `tests/quantization/test_bitnet.py`: CPU-only pytest tests covering the pack/unpack round trip, rejection of non-2D input, equivalence of `BitNetLinearMethod.apply` against a manual dequantize+matmul reference, and the no-bias path. ## Test Result I have visually verified the pasted source is syntactically well-formed, but I have not run this test suite myself in a real Python/PyTorch environment, nor exercised it against a real checkpoint or on GPU. Please run `pytest tests/quantization/test_bitnet.py` and a smoke import test before relying on this - I'd appreciate maintainer/CI verification given how this was authored. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described above (prototype BitNet ternary quantization backend, addressing #33142). - [x] A test plan is included above. - [ ] Test results are NOT included (not yet run by the author - see Test Result section above for why). - [x] This is a breaking-change-free, additive-only change (new file + new OOT-registered quant method; no existing files modified).
", "url": "https://github.com/vllm-project/vllm/pull/52195", "createdAt": "2026-08-13T17:33:34Z", "updatedAt": "2026-08-13T17:34:13Z", "timestamp": "2026-08-13T17:34:13Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "Laalinibh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52196", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Mamba][Quantization] Add Stage 1 (CPU-only, reference) int8 quantization utilities for SSM state", "text": "## Purpose Adds a self-contained, pure-PyTorch reference implementation of per-channel dynamic int8 quantize/dequantize helpers for the Mamba/SSM recurrent state (`vllm/model_executor/layers/mamba/state_quant.py`), plus a `QuantizedSSMState` wrapper modeling the read/compute/write pattern a real integration would use each decode step. **Motivation:** vLLM already ships FP8/TurboQuant/NVFP4 quantization for the attention KV cache (see `STR_DTYPE_TO_TORCH_DTYPE` in `vllm/utils/torch_utils.py`) because that cache is read and rewritten every decode step and is memory-bandwidth bound. The Mamba/SSM state (conv + temporal state, used by Mamba1/2, GatedDeltaNet, KDA, ShortConv and every hybrid model built on them - Jamba, NemotronH, Zamba2, Qwen3.5, FalconH1, etc.) has the exact same profile, but `MambaDType` (`vllm/config/cache.py`) only allows `float32`/`float16`/`bfloat16` today, and `MambaStateDtypeCalculator` (`vllm/model_executor/layers/mamba/mamba_utils.py`) has no quantize/dequantize step anywhere in it. Interestingly, `STR_DTYPE_TO_TORCH_DTYPE` already contains `\"int8\"`/`\"fp8*\"` entries (reused from KV-cache dtype strings), and `MambaBase.bind_kv_cache` (`vllm/model_executor/layers/mamba/abstract.py`) already slices the raw per-block byte page and reinterprets it per declared dtype/shape - so a smaller-dtype state would already be sized correctly by the existing allocator. What's missing is the actual quantize-before-write / dequantize-before-compute step, since the selective-scan recurrence itself must run in floating point. ## Scope / what this is NOT This change is intentionally self-contained and is **NOT** wired into `MambaDType`, `CacheConfig`, or any real mixer/kernel (`mamba_mixer.py`, `mamba_mixer2.py`, the `gdn`/`kda`/`short_conv` variants). Doing that safely means touching the compiled CPU/Triton/CUDA selective-scan kernels and every per-architecture mixer forward pass, which I don't think can be responsibly done through source-browsing/PR authorship alone without a real local test run on the actual kernels. See the module docstring in `state_quant.py` for the full list of deferred follow-up work (Literal/config plumbing, per-mixer wiring, fused kernel). I'm opening this as a numerically-verified building block / RFC-style prototype rather than a claim that quantized Mamba state is fully usable today. ## Test Plan Added `tests/kernels/mamba/test_state_quant.py`: CPU-only pytest tests covering quantize/dequantize round-trip error bounds, int8 value range, the all-zero-state edge case, rejecting non-int8 input to dequantize, the `QuantizedSSMState` wrapper's round-trip and in-place update, and numerical drift of a simulated diagonal recurrence when the state is quantized after every step. ## Test Result I have visually verified the pasted source is syntactically well-formed, but I have not run this test suite myself in a real Python/PyTorch environment. Please run `pytest tests/kernels/mamba/test_state_quant.py` before relying on this. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described above (int8 quantization utilities for the Mamba/SSM state, as a Stage 1 prototype/RFC). - [x] A test plan is included above. - [ ] Test results are NOT included (not yet run by the author - see Test Result section above for why). - [x] This is a breaking-change-free, additive-only change (two new files; no existing files modified).
", "url": "https://github.com/vllm-project/vllm/pull/52196", "createdAt": "2026-08-13T17:34:41Z", "updatedAt": "2026-08-13T17:35:17Z", "timestamp": "2026-08-13T17:35:17Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "Laalinibh", "state": "open", "assignees": [] }, { "id": "github:vllm-project/vllm:pull_request:52197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Support DSpark configs with `architectures=DSparkDraftModel` + `model_type=qwen3`", "text": "## Purpose Generic Qwen `DSparkDraftModel` is now normalized to `Qwen3DSparkModel` so models like https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark can now run on vLLM ## Test Plan ## Test Result Tested gsm8k with ``` vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned75 -tp=4 --spec-model RadixArk/Qwen3.8-2.4T-A95B-DSpark --spec-method dspark --spec-tokens 7 ... (APIServer pid=2078258) INFO 08-13 17:48:18 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.90, Accepted throughput: 1537.03 tokens/s, Drafted throughput: 2757.00 tokens/s, Accepted: 15372 tokens, Drafted: 27573 tokens, Per-position acceptance rate: 0.901, 0.795, 0.667, 0.550, 0.436, 0.327, 0.225, Avg Draft acceptance rate: 55.8% ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52197", "createdAt": "2026-08-13T17:37:50Z", "updatedAt": "2026-08-13T17:53:53Z", "timestamp": "2026-08-13T17:53:53Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "speculative-decoding", "ready", "qwen" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" }, { "id": "github:vllm-project/vllm:pull_request:52198", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts", "text": "# [Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts ## Problem PEFT 0.18+ can train a LoRA against a fused 3D MoE expert parameter via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`), reading it `[E, dim1, dim2]` as `[E, in, out]`. vLLM's `add_lora_w13` / `add_lora_w2` punica kernels use the opposite `nn.Linear` `[out, in]` convention, so per expert `ΔW_vllm = ΔW_peft^T`. On stock vLLM, `_stack_moe_lora_weights` reshapes with the wrong semantics and `set_lora` raises a shape mismatch that drops the adapter (e.g. Qwen3.5-35B-A3B `down_proj`: `512 vs 2048`). ## Fix - `peft_helper.py`: parse `target_parameters` from `adapter_config.json`. - `lora_model.py`: propagate it onto `LoRAModel` (ctor, `clone()`, `from_lora_tensors`). - `model_manager.py`: in `_stack_moe_lora_weights`, detect the PEFT 3D layout and swap `lora_A`/`lora_B` + transpose the trailing two dims **after** the EP-aware expert slicing. Detection is required, not assumed: `create_dummy_lora_weights` builds dummies in the native layout with the same flat shape, so an unconditional transpose would corrupt them. Primary signal is `target_parameters` naming a fused expert weight; fallback is `lora_a` trailing dim != `hidden_size`. ## Testing ``` python -m pytest -v \\ tests/lora/test_peft_helper.py::test_peft_helper_target_parameters_direct \\ tests/lora/test_lora_manager.py::test_target_parameters_indicates_3d_lora \\ tests/lora/test_lora_manager.py::test_from_lora_tensors_propagates_target_parameters \\ tests/lora/test_peft_helper.py::test_peft_helper_pass \\ tests/lora/test_lora_manager.py::test_from_lora_tensors ``` → **10 passed** (the discriminator test is parametrized over 6 cases). **End-to-end (Qwen3.5-35B-A3B expert LoRA).** The raw PEFT adapter crashes the unpatched loader with the `512 vs 2048` error. With the fix, the directly-loaded adapter is **bit-identical** (chosen-token logprobs, 32 tokens × 5 prompts) to an independent offline-converted oracle that self-checks `max|ΔW_vllm − ΔW_peft^T| = 0` over 320 experts, while both shift logits measurably vs base — so the adapter takes effect and the in-loader transpose matches the verified transform. Logprobs, not decoded text: for an experts-only (8-of-256, rank-32) adapter, greedy text is a false-negative trap since the adapter shifts logits without always flipping the argmax on short prompts. ## AI assistance disclosure AI assistance was used to author this change. A human submitter has reviewed every changed line and run the tests above.", "url": "https://github.com/vllm-project/vllm/pull/52198", "createdAt": "2026-08-13T17:48:23Z", "updatedAt": "2026-08-13T17:55:50Z", "timestamp": "2026-08-13T17:55:50Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "darrenpeng-amazon", "state": "open", "assignees": [], "change": "new" }, { "id": "github:vllm-project/vllm:pull_request:52199", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add local/external prefix-cache hit breakdown to prompt_tokens_details (additive)", "text": "## Summary A **strictly additive** change that exposes the **local vs external** prefix-cache hit breakdown per request in `prompt_tokens_details` (closes #52137). No existing field or behavior changes: `cached_tokens` keeps its exact meaning (total = `local + external`), and the two new fields are `None` by default / omitted from the response unless the flag is on. ## Background The scheduler already tracks `num_local_cached_tokens` / `num_external_cached_tokens` separately in `PrefillStats`, but `OutputProcessor` collapsed them to their sum (`num_cached_tokens`) before forwarding, so users of distributed prefix caching (Mooncake / lmcache / PD disaggregation) could not tell, per request, where cache hits came from (relevant for billing and diagnosing hit-rate changes). ## Changes (all additive) - **Data pipeline**: `RequestState` and `RequestOutput` gain `num_local_cached_tokens` / `num_external_cached_tokens`; `OutputProcessor` threads them from `PrefillStats` (same assignment site, so `cached_tokens == local + external` holds by construction). - **Protocol**: `PromptTokenUsageInfo` gains `local_cached_tokens` / `external_cached_tokens` (both default `None`). - **Endpoints**: chat and completion, streaming and non-streaming, all forward the breakdown. Other endpoints (responses / cohere / scale_out) are intentionally out of scope. - **Compatibility**: existing fields are untouched; new fields are `None` by default and omitted from the response when `--enable-prompt-tokens-details` is off — existing clients and dashboards keep working unchanged. ## Tests - `tests/entrypoints/openai/chat_completion/test_serving_chat.py`: `_make_prompt_tokens_details` with the new signature — defaults, gated-off behavior, multimodal coexistence, and the `cached_tokens == local + external` invariant. - `tests/v1/engine/test_output_processor.py`: `PrefillStats` → `RequestState` threading with non-zero local/external values.", "url": "https://github.com/vllm-project/vllm/pull/52199", "createdAt": "2026-08-13T17:48:30Z", "updatedAt": "2026-08-13T17:49:11Z", "timestamp": "2026-08-13T17:49:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "frontend" ], "author": "SayHelloToWorld", "state": "open", "assignees": [], "change": "new" } ], "events": [ { "id": "event:581f0333c5ebd68e0d53", "signalId": "github:vllm-project/vllm:pull_request:49009", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Support sequence parallelism for block fp8 on XPU", "text": "## Purpose Enable sequence parallelism (SP) support on XPU for the block-quant W8A8 FP8 model SP is enabled with: ```json {\"pass_config\":{\"enable_sp\":true},\"use_inductor_graph_partition\":true} ``` ## Test Plan Model: `RedHatAI/Qwen3-32B-FP8-block` (dense, block-quant W8A8 FP8, `weight_block_size=[128,128]`), tp=4, `--linear-backend xpu`. ### Server (one of the three, per case) ```bash # sp_on VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 \\ --compilation-config '{\"pass_config\":{\"enable_sp\":true},\"use_inductor_graph_partition\":true}' # sp_off VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 # eager VLLM_XPU_FUSED_MOE_USE_REF=1 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \\ python3 -m vllm.entrypoints.openai.api_server \\ --model RedHatAI/Qwen3-32B-FP8-block --port 8072 --host 0.0.0.0 \\ --trust-remote-code --gpu-memory-util=0.9 --max-num-batched-tokens=8192 \\ --max-model-len=6000 --no-enable-prefix-caching --block-size 64 \\ --linear-backend xpu -tp=4 --enforce-eager ``` ### Client (bench serve, run twice per case; better of the two reported) ```bash timeout -s KILL 6000000s python3 -m vllm.entrypoints.cli.main bench serve \\ --model RedHatAI/Qwen3-32B-FP8-block \\ --ready-check-timeout-sec 1 --temperature=0 \\ --dataset-name random --random-input-len=4096 --random-output-len=1024 \\ --ignore-eos --port=8072 --host 0.0.0.0 \\ --num-prompt 15 --request-rate inf --backend vllm --trust-remote-code \\ --max-concurrency 3 ``` ## Test Result Model: `RedHatAI/Qwen3-32B-FP8-block`, tp=4, `--linear-backend xpu`, input_len=4096, output_len=1024, num_prompts=15, max_concurrency=3. ### Accuracy (GSM8K, 250 questions) Case | Accuracy | Invalid rate -- | --: | --: `eager` (baseline) | 0.584 | 0.000 `sp_off` | 0.648 | 0.000 `sp_on` | 0.668 | 0.000 ### Benchmark Throughput: higher raw value is better. TTFT/TPOT: lower raw value is better. % shown is the improvement vs. `eager` baseline (positive % = better than eager, negative % = worse than eager). Case | Output throughput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) -- | --: | --: | --: | --: | --: | --: | --: `eager` (baseline) | 21.06 | 12729.52 | 14200.37 | 20602.96 | 130.09 | 129.23 | 136.14 `sp_off` | 22.34 (+6.08%) | 12891.37 (-1.27%) | 14516.25 (-2.22%) | 20284.13 (+1.55%) | 121.78 (+6.39%) | 121.62 (+5.89%) | 129.17 (+5.12%) `sp_on` | 22.34 (+6.08%) | 12344.39 (+3.03%) | 13671.84 (+3.72%) | 20345.60 (+1.25%) | 122.35 (+5.95%) | 121.68 (+5.84%) | 128.27 (+5.78%)", "url": "https://github.com/vllm-project/vllm/pull/49009", "createdAt": "2026-07-18T04:03:57Z", "updatedAt": "2026-08-13T13:47:52Z", "timestamp": "2026-08-13T13:47:52Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu", "ci/build" ], "author": "chaojun-zhang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4eecc0fd5381cb51f9ed", "signalId": "github:vllm-project/vllm:pull_request:52114", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52114", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] [Quantization] Add Ling hybrid MXFP4 routed experts support", "text": "## Purpose Add support for Ling checkpoints that use hybrid quantization: block FP8 for dense and shared-expert projections, and MXFP4 for routed experts. This change reads Ling-specific quantization metadata and remaps routed-expert scale names to the convention expected by `Mxfp4MoEMethod` for both the main and MTP models", "url": "https://github.com/vllm-project/vllm/pull/52114", "createdAt": "2026-08-13T06:59:24Z", "updatedAt": "2026-08-13T13:46:53Z", "timestamp": "2026-08-13T13:46:53Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "quantization" ], "author": "zexplorerhj", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1bc7849d8bb0380e617a", "signalId": "github:vllm-project/vllm:pull_request:52086", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52086", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Gate cooperative top-k on CUDA 13", "text": "## Fixes - Build cooperative top-k kernels only on CUDA 13 or newer. - Let source builds skip bundled FA2/FA3 extensions while retaining Python-only FA4. - Treat an FA4-only installation as a valid FlashAttention installation. ## Verification - The patched source builds as vLLM 0.23.1rc1.dev1407+gdab991476 on CUDA 12.8. - The baked SkyRL image imports FlashAttention successfully.", "url": "https://github.com/vllm-project/vllm/pull/52086", "createdAt": "2026-08-13T03:17:05Z", "updatedAt": "2026-08-13T13:46:42Z", "timestamp": "2026-08-13T13:46:42Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "ci/build", "nvidia" ], "author": "j316chuck", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f0d1f4d85aea2c026ca0", "signalId": "github:vllm-project/vllm:issue:52170", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52170", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "OffloadingConnector AssertionError in _build_store_jobs under MultiConnector with high concurrency multi-turn", "text": "## Summary The OffloadingConnector's scheduler crashes with an `AssertionError` in `_build_store_jobs()` when running under a MultiConnector (NixlConnector + OffloadingConnector) with high-concurrency multi-turn agentic workloads. ## Environment - **vLLM**: v0.26.0 (`vllm/vllm-openai:v0.26.0`) - **Model**: RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block (Mamba-hybrid MoE, TP8) - **Hardware**: 4× H200 nodes (140GB each), 8 GPUs per node, InfiniBand RDMA - **Architecture**: Prefill/Decode disaggregation (3 prefill + 1 decode, TP8) - **KV transfer config**: MultiConnector with NixlConnector (kv_consumer) + OffloadingConnector (200GB CPU, lazy_offload=true) - **Workload**: 30 concurrent multi-turn conversations, 540 turns per conversation, 160K first-prompt tokens, 15s inter-turn delay, FP8 KV cache, prefix caching enabled ## Error ``` (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] EngineCore encountered a fatal error. (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] Traceback (most recent call last): File \".../vllm/v1/engine/core.py\", line 1323, in run_engine_core engine_core.run_busy_loop() File \".../vllm/v1/engine/core.py\", line 1364, in run_busy_loop self._process_engine_step() File \".../vllm/v1/engine/core.py\", line 1403, in _process_engine_step outputs, model_executed = self.step_fn() File \".../vllm/v1/engine/core.py\", line 645, in step_with_batch_queue scheduler_output = self.scheduler.schedule(self._should_throttle_prefills()) File \".../vllm/v1/core/sched/scheduler.py\", line 1174, in schedule meta = self._build_kv_connector_meta(self.connector, scheduler_output) File \".../vllm/v1/core/sched/scheduler.py\", line 1196, in _build_kv_connector_meta return connector.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 422, in build_connector_meta metadata=tuple( File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 423, in c.build_connector_meta(scheduler_output) for c in self._connectors File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py\", line 157, in build_connector_meta return self.connector_scheduler.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 1157, in build_connector_meta store_jobs=self._build_store_jobs(scheduler_output), File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 975, in _build_store_jobs assert len(offload_keys) == len(offload_block_ids) AssertionError ``` ## Reproduction The crash occurs on the **decode** pod after several minutes of sustained multi-turn traffic. It does not happen immediately on startup — the model loads and serves requests successfully before the offloading scheduler's internal state diverges. KV transfer config used: ```json { \"kv_connector\": \"MultiConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"connectors\": [ { \"kv_connector\": \"NixlConnector\", \"kv_role\": \"kv_consumer\", \"kv_connector_extra_config\": {\"bidirectional_kv_xfer\": true} }, { \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"cpu_bytes_to_use\": 214748364800, \"lazy_offload\": true } } ] } } ``` vLLM flags: `--kv-cache-dtype fp8 --enable-prefix-caching --enable-expert-parallel --max-model-len 262144` ## Analysis The assertion at `scheduler.py:975` expects `offload_keys` and `offload_block_ids` to have the same length when building CPU offload store jobs. Under heavy multi-turn load with 30 concurrent conversations accumulating context over hundreds of turns, the offloading scheduler's block tracking appears to get out of sync — likely a race or bookkeeping error when blocks are evicted and reassigned rapidly. This is replicating the workload from the [llm-d Nemotron agentic serving guide](https://github.com/llm-d/llm-d/blob/main/guides/agentic-serving/nemotron-3-ultra-550b-h200.md) with the same MultiConnector configuration.", "url": "https://github.com/vllm-project/vllm/issues/52170", "createdAt": "2026-08-13T13:46:21Z", "updatedAt": "2026-08-13T13:46:21Z", "timestamp": "2026-08-13T13:46:21Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "bbenshab", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:540df361f8cbf035d78a", "signalId": "github:vllm-project/vllm:pull_request:52062", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52062", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert KV block zeroing generalization due to ROCm launch overflow", "text": "## Summary - Revert #51749, which generalized KV block zeroing from `FullAttentionSpec` to every `AttentionSpec`. - On DeepSeek-V4-Pro with ROCm and 64 concurrent GSM8K requests, the broader zeroing path makes `_zero_kv_blocks_kernel` launch with a grid that exceeds the AMD Triton driver's signed integer limit. - The resulting `OverflowError: signed integer is greater than maximum` kills the engine and causes all in-flight `/v1/chat/completions` requests to return HTTP 500. ## Duplicate check I searched open PRs for `51749`, `KVBlockZeroer OverflowError ROCm`, and `Generalize KV block zeroing revert`; no existing PR addresses this failure. ## Test plan - [x] `.venv/bin/python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/worker/test_kv_block_zeroer.py -q` — 14 passed - [x] Before revert: `dsv4_rocm_bench/run_gsm8k.sh` at concurrency 64 reproduced the worker failure in `KVBlockZeroer.zero_block_ids`, followed by `EngineDeadError` and HTTP 500 responses. - [x] After revert: the full 1,319-sample GSM8K run completed at concurrency 64 in 192.2 seconds. - strict exact match: 0.96285 - flexible exact match: 0.96209 This PR intentionally makes no claim about output quality from raw `/v1/completions` requests; DeepSeek-V4-Pro expects chat-template formatting, and the post-revert GSM8K evaluation used `/v1/chat/completions`. AI assistance was used for log triage, identifying the failing code path, and drafting this PR. The human submitter reviewed the changes and ran the tests above. Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/52062", "createdAt": "2026-08-12T22:50:19Z", "updatedAt": "2026-08-13T13:45:31Z", "timestamp": "2026-08-13T13:45:31Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "needs-rebase" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9c5acc2fccf6486a1dd2", "signalId": "github:vllm-project/vllm:pull_request:33763", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:33763", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add vllm_enable_compile_cache config flag with backward compatibility", "text": "- Add vllm_enable_compile_cache boolean flag to CompilationConfig (defaults to True) - Keep VLLM_DISABLE_COMPILE_CACHE environment variable for backward compatibility - Environment variable overrides config flag when set via model_validator - Update is_compile_cache_enabled() to accept config flag parameter - Update all test files to use config flag instead of environment variable - Update documentation to mention both config flag (preferred) and env var (backward compatibility) This change provides a cleaner API via config while maintaining backward compatibility with existing code using the environment variable. For https://github.com/vllm-project/vllm/issues/29917 ---
- [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/33763", "createdAt": "2026-02-04T04:50:40Z", "updatedAt": "2026-08-13T13:45:19Z", "timestamp": "2026-08-13T13:45:19Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "documentation", "frontend", "ready", "llama", "qwen" ], "author": "elizabetht", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:dcae76bc0d225889ddc4", "signalId": "github:vllm-project/vllm:pull_request:51704", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec", "text": "## Purpose Part of the KV-cache layout standardization series (RFC #42082). **Stacked on #51612** — the diff shown includes it until that lands and this retargets `main`. Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants in MLA spec overrides. This PR makes specs plain data and moves each packed format to the backend/component that owns it: - `AttentionSpec` gains two optional packing fields `num_head_slots` (H in standard layout) and`state_content_bytes` (C in standard layout). Page sizes derive uniformly as `num_heads × storage_block_size × content`. > NOTE: i dont love the num_head_slots name but this should hopefully be temporary until we can [fully refactor the backends](https://github.com/vllm-project/vllm/issues/42449); i.e. separate allocation from connector and kv-cache manager concerns - `AttentionBackend.customize_spec(spec)` is a temporary hook until we can move to having the backend return the spec instead of the layer (see: https://github.com/vllm-project/vllm/issues/42449) Prepares the ground for the layout-standardization PR ([[6/N]](https://github.com/vllm-project/vllm/pull/51718)), where these fields describe the standardized `[B, H, N, C]` page and the AITER backends publish separate K/V head groups through the same hook. --- AI-assisted (Claude); reviewed by submitter", "url": "https://github.com/vllm-project/vllm/pull/51704", "createdAt": "2026-08-10T16:20:40Z", "updatedAt": "2026-08-13T13:45:16Z", "timestamp": "2026-08-13T13:45:16Z", "metrics": { "reactions": 2, "comments": 20 }, "labels": [ "rocm", "ready", "nvidia", "ready-run-all-tests", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f1b57284230544aaf171", "signalId": "github:vllm-project/vllm:pull_request:49357", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49357", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] bound FlashMLA sparse decode intermediate tensors size", "text": "## Purpose Fix #44545 - not in an ideal way though. When FlashMLA sparse kernel is selected with FP8 KV cache dtype, FlashMLA sparse decode kernel(https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_decode.h#L184) is called in mixed batch mode. This kernel internally allocates intermediate tensors (`o_accum`) on call, which quickly dominates the usual unallocated workspace size (~2GB per the comment in `flashmla_sparse.py`) for long prefill: For 32k input on H200, the bytesize of `o_accum` is ``` # b = 1 # num_sm_parts = 1 # s_q = 32768 # h_q = 64 # d_v = 512 (b + num_sm_parts) * s_q * h_q * d_v * 4 bytes = 8 GiB ``` For BF16 KV cache, a different kernel (FlashMLA sparse prefill kernel - https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_fwd.h#L101) is called, which does not allocate a massive VRAM on kernel call and thus does not suffer an OOM contrary to the FP8 code branch. (Note the additional `d_v` factor in the formular for `o_accum`, no other scratch tensors in FlashMLA have this factor thus negligible in terms of VRAM occupancy) Hence we chunk the mixed-batch input into manageable bytes (`VLLM_FLASHMLA_SPARSE_MAX_SCRATCH_MB`) and loop over the chunks to aggregate the output attention tensor, which is in the same spirit as the indexer budget management logic in #36178. Due to the mechanism (looping the kernel over sub-chunks instead of single big shot), it may incur a throughput penalty as a tradeoff, but I have no better idea yet. As the context window for MLA model families keep growing, FP8 KV cache is kind of inevitable, so I think this PR can work as a bandaid. ## Test Plan No OOM for the reproducer in #44545 ## Test Result pass ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49357", "createdAt": "2026-07-21T18:06:42Z", "updatedAt": "2026-08-13T13:45:01Z", "timestamp": "2026-08-13T13:45:01Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "needs-rebase", "v1" ], "author": "cjackal", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:af30729d6508705a868a", "signalId": "github:vllm-project/vllm:pull_request:51506", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51506", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Hoist tensor-descriptor build out of unified-attention TD ti…", "text": "## Summary Part of the TD adoption strategy proposed in #42545. `USE_TD`'s K/V tile loader rebuilt the tensor descriptor every tile iteration, since the descriptor's `base` baked in the per-tile physical block index. That forces a `tensormap_create` per iteration, which the pipeliner can't predicate: - On Hopper (sm_90), this only compiles at `num_stages=1`, costing roughly 50% throughput vs the raw-pointer default. - On sm_100/sm_120, it fails to compile at all: `'ttng.tensormap_create' op pipeliner doesn't know how to predicate this op`. ## Fix Move the loop-invariant `kv_head_idx` into the descriptor base and build one descriptor per K/V cache before the tile loop, addressing the per-tile physical block via a `load()` coordinate instead. This makes the descriptor loop-invariant, restoring `num_stages > 1` compilation on Hopper and fixing the sm_100/sm_120 compile failure outright. The hoisted 3D descriptor also needs a registered scratch allocator on sm_100/sm_120 (`Kernel requires a runtime memory allocation, but no allocator was set` otherwise). Wired up `set_triton_allocator()`, matching the existing pattern in `fused_moe` / GDN / LoRA's TD paths, cached per device to avoid a redundant call on every forward pass. ## Testing ### Correctness ``` pytest tests/kernels/attention/test_triton_unified_attention.py -k use_td ``` `290 passed` on H100/H200. ### Throughput `vllm bench throughput`, Qwen3-4B bf16, TRITON_ATTN, H100/H200: | Workload | in/out | raw ptr | TD hoisted | hoisted/raw | |---|---|---|---|---| | prefill_heavy | 2048/128 | 17.57 req/s | 15.85 req/s | 0.90x | | decode_heavy | 128/1024 | TBD | TBD | TBD | | balanced | 1024/256 | 28.25 req/s | 25.13 req/s | 0.89x | Repro: ``` COMMON=\"--model Qwen/Qwen3-4B --dtype bfloat16 --attention-config '{\\\"backend\\\":\\\"TRITON_ATTN\\\"}' --max-model-len 4096 --gpu-memory-utilization 0.85 --seed 42\" VLLM_TRITON_USE_TD=0 vllm bench throughput $COMMON --dataset-name random --random-input-len 2048 --random-output-len 128 --num-prompts 200 VLLM_TRITON_USE_TD=1 vllm bench throughput $COMMON --dataset-name random --random-input-len 2048 --random-output-len 128 --num-prompts 200 ``` Note: use `--random-input-len` / `--random-output-len`, not `--input-len` / `--output-len`, with `--dataset-name random`, the latter are silently overridden by the random defaults. B200 (sm_100), same model/backend/workloads: | Workload | in/out | raw ptr | TD hoisted | hoisted/raw | |---|---|---|---|---| | prefill_heavy | 2048/128 | 26.01 req/s | 22.05 req/s | 0.85x | | decode_heavy | 128/1024 | 23.25 req/s | 17.98 req/s | 0.77x | | balanced | 1024/256 | 40.22 req/s | 32.34 req/s | 0.80x | Correctness on B200: `290 passed` as well, same suite. Close to oonyshch's own B200 numbers (0.85x/0.83x/0.78x), independently confirmed on our own commit. cc: joint effort with @oonyshch @BabyDrangoner,coauthored the repro and root cause on H200/b200", "url": "https://github.com/vllm-project/vllm/pull/51506", "createdAt": "2026-08-08T16:03:02Z", "updatedAt": "2026-08-13T13:44:40Z", "timestamp": "2026-08-13T13:44:40Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [], "author": "cinnamonica02", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:81a12f0bb592107cf2fe", "signalId": "github:vllm-project/vllm:pull_request:52148", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52148", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Fix FlashInfer SM12x prefill with sinks", "text": "## Summary Use FlashInfer's sink-aware paged prefill wrapper on SM12x when attention sinks are enabled. The generic FA2 prefill path accepts a `sinks` argument but does not apply it, so #49718 can use XQA for decode while producing incorrect prefill output. The wrapper is specialized with the active dtypes, head dimensions, sliding window, and softmax scale. DCP, NVFP4, SM90/SM100, and sink-free paths are unchanged. This is not a duplicate of the automatic revert #51987: that PR removes SM12x XQA support, while this change preserves XQA and fixes the prefill path for sink-attention models on SM12x . ## Validation - ` pytest tests/v1/attention/test_attention_backends.py::test_flashinfer_native_prefill_with_sinks -q` passing - SM12x GPT-OSS model eval fixed and passing - Nemotron and Qwen models both working as expected", "url": "https://github.com/vllm-project/vllm/pull/52148", "createdAt": "2026-08-13T10:33:31Z", "updatedAt": "2026-08-13T13:44:27Z", "timestamp": "2026-08-13T13:44:27Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ready", "nvidia" ], "author": "askliar", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c1b9d80895fecf85429d", "signalId": "github:vllm-project/vllm:pull_request:52124", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52124", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Renderer] Shutdown the renderer properly.", "text": "## Purpose Currently, during online serving, the renderer is shutdown twice, while it is not shutdown during offline inference. ``` offline: python examples/basic/offline_inference/embed.py this pr: INFO 08-13 16:11:10 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1340] [shutdown] EngineCore: exiting busy loop INFO 08-13 16:11:11 [base.py:305] [shutdown] BaseRenderer main: INFO 08-13 16:19:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop online: vllm serve intfloat/e5-small this pr: (EngineCore pid=20072) INFO 08-13 16:12:24 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=20010) INFO 08-13 16:12:24 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=20010) INFO 08-13 16:12:24 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=20072) INFO 08-13 16:12:24 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20072) INFO 08-13 16:12:24 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20072) INFO 08-13 16:12:24 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=20010) INFO 08-13 16:12:24 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=20010) INFO 08-13 16:12:24 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=20010) INFO 08-13 16:12:24 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=20010) WARNING 08-13 16:12:24 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=20010) WARNING 08-13 16:12:24 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 20072 (APIServer pid=20010) INFO 08-13 16:12:24 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=20010) INFO 08-13 16:12:24 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=20010) INFO 08-13 16:12:24 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=20010) INFO 08-13 16:12:24 [launcher.py:139] [shutdown] API server: engine client stopped (APIServer pid=20010) INFO 08-13 16:12:24 [launcher.py:142] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=20010) INFO 08-13 16:12:24 [launcher.py:163] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=20010) INFO: Shutting down (APIServer pid=20010) INFO: Waiting for application shutdown. (APIServer pid=20010) INFO: Application shutdown complete. (APIServer pid=20010) INFO 08-13 16:12:24 [base.py:305] [shutdown] BaseRenderer main: (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=20533) INFO 08-13 16:21:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 20593 (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:139] [shutdown] API server: engine client stopped (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:142] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:163] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO: Shutting down (APIServer pid=20533) INFO: Waiting for application shutdown. (APIServer pid=20533) INFO: Application shutdown complete. ``` ## Best practices for shutdown? - should we use weakref.finalize having classes handle their own shutdown? - should we have a centralized orchestration for shutdown? - Or do we need a centralized orchestration for shutdown plus having classes handle their own exits? ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52124", "createdAt": "2026-08-13T08:22:32Z", "updatedAt": "2026-08-13T13:43:58Z", "timestamp": "2026-08-13T13:43:58Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend" ], "author": "noooop", "state": "open", "assignees": [ "sangstar" ], "change": "updated" } }, { "id": "event:960f60f0175b3ab8d881", "signalId": "github:vllm-project/vllm:pull_request:51338", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51338", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RFC][Bug Fix][Spec Decode] Trust checkpoint-declared method over path-name heuristics", "text": "This PR replaces speculative-method guessing from arbitrary model-path substrings with checkpoint declarations and registered drafter architectures. Explicit `method` values remain authoritative, and every implicit selection is logged at INFO. ## Problem On current `main`, a speculators-format checkpoint at a normal training path is silently treated as a generic autoregressive draft model: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15}' ``` Copying the same checkpoint to a directory containing the algorithm name changes the detected method: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/dflash_b16_speculators\", \"num_speculative_tokens\": 15}' ``` `main` checks substrings such as `eagle3`, `dflash`, and `dspark` in the complete model path before consulting structural configuration. An unrelated parent directory can therefore override the checkpoint's actual type. With this PR, the first command detects DFlash from the checkpoint architecture at any path. End-to-end Qwen3-8B serving with the same checkpoint and greedy chat traffic produced: | | detected method | acceptance length | |---|---|---| | `main` | `draft_model` | 1.02 | | this PR | `dflash` | 3.88 | The 3.88 result matches serving with an explicit `\"method\": \"dflash\"`. ## RFC: Legacy checkpoint compatibility This PR intentionally changes implicit speculative-method detection from model path/name matching to checkpoint architecture matching. Legacy checkpoints without a recognized architecture may now fall back to `draft_model`. Users of local copies, mirrors, renamed repositories, or private checkpoints may need to specify `method` explicitly. The current compatibility shim recognizes selected Hugging Face owners/IDs, but this may be incomplete and brittle. Feedback requested: - Which legacy EAGLE/EAGLE3, DFlash, and DSpark checkpoints are still supported? - Is requiring an explicit `method` for undeclared legacy checkpoints acceptable? - Should the old detection remain temporarily with a deprecation warning? - Can model owners validate their published checkpoint and a local copy? ## Design - Explicit `method` remains authoritative. - Config parsing normalizes self-describing checkpoints to canonical drafter architectures. - `SPEC_METHOD_BY_DRAFTER_ARCH`, next to the speculative model registry, maps each architecture to its method and parallel-drafting behavior. - Structural `model_type` handling remains for Medusa, MLP speculators, and MTP. - A narrowly scoped Hugging Face ID compatibility shim handles legacy EAGLE checkpoints whose configs do not declare their method and may be obscured by MTP normalization. It rejects arbitrary local paths and emits a deprecation warning. - If no specialized method is declared, the existing `draft_model` fallback is preserved. The selected method is INFO-logged. Speculators-format conversion now declares the canonical architecture and token count, so DFlash, EAGLE, EAGLE3, and PEagle share the registry-based resolution path instead of separate path checks. ## Compatibility Conventional autoregressive draft models continue to work without an explicit method. Known legacy EAGLE Hub IDs also retain automatic detection. A local copy of a legacy checkpoint that does not declare its method must pass `\"method\": \"eagle\"` or `\"method\": \"eagle3\"`, because arbitrary local path names are no longer interpreted. `hf_config_override` still combines structural normalization with loading-time canonicalization for several embedded MTP families. Separating those phases is useful follow-up work but is outside this bugfix. ## Validation - End-to-end Qwen3-8B + DFlash: acceptance length improved from 1.02 (misdetected on `main`) to 3.88, matching an explicit `\"method\": \"dflash\"`. - Config tests: 186 passed; 2 gated-model tests failed with HTTP 403. - Related speculative-config tests: 26 passed. - Pre-commit hooks passed.
Commands run ```bash .venv/bin/python -m pytest -q -p no:cacheprovider tests/test_config.py .venv/bin/python -m pytest -q -p no:cacheprovider \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/config/test_bailing_mtp_config.py \\ tests/transformers_utils/test_dspark_mla_config.py \\ tests/config/test_speculative_draft_max_position_embeddings.py .venv/bin/pre-commit run --from-ref upstream/main --to-ref HEAD ``` The two `tests/test_config.py` failures require access to the gated `meta-llama/Meta-Llama-3-8B-Instruct` repository and returned HTTP 403.
## Contribution notes - This is not a duplicate of #47490 or #36979; neither replaces path-based detection with architecture-based resolution. - AI assistance was used. I reviewed every changed line and ran the validation listed above.", "url": "https://github.com/vllm-project/vllm/pull/51338", "createdAt": "2026-08-07T00:53:10Z", "updatedAt": "2026-08-13T13:42:54Z", "timestamp": "2026-08-13T13:42:54Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "documentation", "new-model", "needs-rebase", "rust" ], "author": "WindChimeRan", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:1e0751e5dbe7970fc0ab", "signalId": "github:vllm-project/vllm:pull_request:52164", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52164", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][DSA] Take the native decode path for MTP=3 on SM90", "text": "# [Attention][DSA] Take the native decode path for MTP=3 on SM90 ## Purpose Closes #35878. The DSA indexer flattens a spec-decode batch into one single-token row per query whenever `next_n` falls outside `{1, 2}`, so with MTP=3 (`next_n = 4`) each request's KV tile is read four times instead of once. DeepGEMM's `nv_dev` branch, which vLLM already pins (`cmake/external_projects/deepgemm.cmake`), implements `next_n = 4` on SM90 through a 2-CTA multicast cluster, so Hopper no longer needs that expansion. #45322 did the equivalent for SM100; this is the SM90 case the issue asks for. Behavior matrix (\"native\" = kernel sees the real batch `(B, next_n)`; \"flatten\" = requests expanded into `B × next_n` single-token pseudo-requests): | Platform | `next_n` (= spec tokens + 1) | Before | After | Changed? | |---|---|---|---|---| | SM100 (B200/GB200) | any | native | native | — | | **SM90 (H100/H200)** | **4 (MTP=3)** | **flatten** | **native** | ✅ **this PR** | | SM90 | 1, 2 | native | native | — | | SM90 | 3, ≥ 5 | flatten | flatten | — (no kernel) | | SM12x and others | 1, 2 | native | native | — | | SM12x and others | > 2 | flatten | flatten | — (unvalidated) | Three pieces: 1. The hardcoded gate becomes `_supports_native_decode(next_n)`, which asks `native_next_n_supported()` per architecture. SM90 implements `{1, 2, 4}` — note `3` is absent, so the gate is not a simple `>=` threshold. 2. `get_paged_mqa_logits_metadata` now sizes its own scheduler slots. `fp8_fp4_paged_mqa_logits` asserts the metadata against `num_sms / num_kv_multicast`, and SM90 `next_n = 4` runs one task per 2-CTA cluster rather than per SM, so the count must be halved. The wrapper derives `next_n` from `context_lens.shape[1]` and divides internally, so `num_sms` keeps its literal meaning and callers cannot get it wrong. The metadata buffer stays sized for one slot per SM; `build()` narrows it to a prefix view using the returned tensor's own shape. 3. Legality is a property of the step, not of the configuration: the kernel is handed `max_decode_len` Q rows, not the configured `next_n`, so a batch that happens to be uniformly 3 tokens deep still flattens on SM90. ## Test Plan Unit and kernel tests: ```bash pytest tests/v1/attention/test_indexer_native_next_n.py -v pytest tests/kernels/attention/test_deepgemm_attention.py -v ``` A standalone microbenchmark (not included in this PR) timed `fp8_fp4_paged_mqa_logits` on the same inputs fed natively as `(B, next_n)` and flattened to `B × next_n` single-token rows, to size the effect the decode path is trading on. End-to-end, DeepSeek-V3.2 on 8×H200, TP8 + EP, MTP=3. The \"before\" arm is this same tree with only `vllm/v1/attention/backends/mla/indexer.py` reverted to the flattening gate, so nothing else differs between arms. Every run was checked against the startup log line to confirm which path it took: ``` before: DSA indexer decode path: use_flattening=True supports_varlen=False (next_n=4, ...) after: DSA indexer decode path: use_flattening=False supports_varlen=False (next_n=4, ...) ``` Accuracy: ```bash cd tests/evals/gsm8k pytest -s -v test_gsm8k_correctness.py \\ --config-list-file=configs/models-h200.txt -k DeepSeek-V3.2-TP ``` `DeepSeek-V3.2-TP.yaml` only reports acceptance length when `min_acceptance_length` is set, so a local copy with `min_acceptance_length: 1.0` was used to make `get_acceptance_length()` print it. Its `startup_max_wait_seconds: 1200` is also not enough on a cold JIT cache — the first start on this machine took about 24 minutes. gsm8k runs at 4096 context while `index_topk` is 2048, so its top-k is close to \"select everything\" and it is not sensitive to the indexer's numerics. MRCR was run as a long-context check where the selection is genuinely sparse: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 32768 -tp 8 \\ --enable-expert-parallel --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' python tests/evals/mrcr/mrcr_eval.py --port 8000 --num-samples 24 --max-tokens 1024 ``` Serving performance: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 40960 -tp 8 \\ --enable-expert-parallel --trust-remote-code --no-enable-prefix-caching \\ --num-gpu-blocks-override 7000 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' vllm bench serve --model deepseek-ai/DeepSeek-V3.2 --dataset-name random \\ --random-input-len 32000 --random-output-len 512 --random-range-ratio 0 \\ --num-prompts $C --max-concurrency $C --ignore-eos --seed 1234 # C in 1 4 8 12 ``` Two methodology notes, because a naive sweep gives misleading numbers here: * `--num-gpu-blocks-override` pins both arms to the same KV cache. Left to the memory profiler the two arms landed on 506k vs 483k tokens on one pair of runs (and the other way round on another), which by itself changes queueing. * `--num-prompts == --max-concurrency` keeps the run to a single wave, so it is decode-bound. At 32K a 143GB×8 node holds roughly 12 concurrent requests, so higher concurrency measures the scheduler queueing, not this kernel. Every configuration was run twice so the run-to-run spread is visible next to the effect. ## Test Result ### Unit and kernel tests ``` tests/v1/attention/test_indexer_native_next_n.py ......... 9 passed tests/kernels/attention/test_deepgemm_attention.py ..... 5 passed ``` `test_deepgemm_fp8_fp4_paged_mqa_logits[2-4]` covers `next_n = 4` against the reference implementation. ### Kernel-level effect (H200, `next_n = 4`) Native `(B, next_n)` vs flattened, speedup of the paged MQA logits call alone: \"image\" | batch | 4K ctx | 16K ctx | 32K ctx | |---|---|---|---| | 1 | 0.97x | 1.03x | 1.03x | | 4 | 1.02x | 1.10x | 1.15x | | 8 | 1.02x | 1.19x | 1.33x | | 16 | 1.07x | 1.25x | 1.33x | | 32 | 1.12x | 1.27x | 1.36x | | 64 | 1.27x | 1.45x | 1.46x | The gain grows with both batch and context, which is what sharing the KV tile predicts. At batch 1 / 4K the native path is 3% slower — flattening one request into four rows gives the kernel more parallelism than it can otherwise use. That case does not appear end-to-end (see below), so the gate is left unconditional. ### End-to-end serving (32K input, 512 output) \"image\" | concurrency | mean TPOT before | after | Δ | output tok/s before | after | Δ | |---|---|---|---|---|---|---| | 1 | 16.44 ms | 16.23 ms | −1.2% | 44.61 | 45.00 | +0.9% | | 4 | 28.02 ms | 26.00 ms | −7.2% | 88.85 | 89.81 | +1.1% | | 8 | 43.43 ms | 42.54 ms | −2.0% | 108.90 | 109.84 | +0.9% | | 12 | 60.99 ms | 58.88 ms | −3.5% | 117.19 | 118.46 | +1.1% | Values are the mean of two runs. Throughput improves by 0.9–1.1% at every concurrency, and at 1, 8 and 12 that clears each arm's own run-to-run range; the concurrency-4 cell does not (its TPOT spread is ±9%, larger than the effect). There is no regression at concurrency 1, so the batch-1 kernel result above does not surface in serving. The end-to-end gain is far smaller than the kernel speedup because the paged MQA logits kernel is a small share of a 671B MoE decode step.
Raw output, concurrency 12 Before (`use_flattening=True`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 52.52 Output token throughput (tok/s): 116.99 Total token throughput (tok/s): 7428.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 61.62 Median TPOT (ms): 60.08 P99 TPOT (ms): 93.81 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.12 Median ITL (ms): 33.42 ``` After (`use_flattening=False`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 51.88 Output token throughput (tok/s): 118.43 Total token throughput (tok/s): 7520.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 59.60 Median TPOT (ms): 60.80 P99 TPOT (ms): 92.96 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.36 Median ITL (ms): 32.64 ```
### Accuracy \"image\" gsm8k, 1319 questions, invalid rate 0.000 on every run: | run | before | after | |---|---|---| | 1 | 0.9560 | 0.9538 | | 2 | 0.9522 | 0.9553 | | 3 | — | 0.9530 | | **mean** | **0.9541** | **0.9540** | Mean acceptance length, which is the control that matters for comparing the performance numbers at all: | run | before | after | |---|---|---| | 1 | 3.074 | 3.069 | | 2 | 3.081 | 3.079 | | 3 | — | 3.078 | | **mean** | **3.077** | **3.075** | The arms overlap on both measures. At n = 1319 one standard error on accuracy is about 0.006, and the arms differ by 0.0001. MRCR at 32K context, where the indexer's top-k is genuinely selective: | | before | after | |---|---|---| | match_ratio | 0.4873 | 0.5637 | | prefix_hit_rate | 1.0000 | 0.9583 | | n=2 / n=4 / n=8 | 0.6494 / 0.4745 / 0.3380 | 0.8091 / 0.5510 / 0.3310 | One run per arm at 24 samples over three needle buckets, so this shows no degradation rather than a gain. ### Path coverage The change adds a per-step fallback for the case where `max_decode_len` is 3, which has no SM90 kernel. Instrumented runs across gsm8k, MRCR and the serving sweep recorded over 140,000 decode steps and saw only `max_decode_len` 1 (the draft model's own decode) and 4 (the verify step) — never 3 — so the fallback stays cold in steady-state MTP decoding. Both cudagraph modes were exercised: gsm8k under `--enforce-eager`, and the serving sweep with `CUDAGraphMode.FULL_AND_PIECEWISE` capturing all 49 decode sizes. --- Not duplicating existing work: no open PR references #35878, and none of the open sparse-indexer PRs touch the SM90 `next_n` gate — #47469 is SM100 varlen, #51555 and #43327 are ROCm, #47629 and #38476 are SM8x/SM12x backends. #45322 is merged and covers SM100 only. AI assistance was used for this change and its validation.", "url": "https://github.com/vllm-project/vllm/pull/52164", "createdAt": "2026-08-13T13:06:39Z", "updatedAt": "2026-08-13T13:42:03Z", "timestamp": "2026-08-13T13:42:03Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "verified" ], "author": "zobinHuang", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:cd061be896bf2943b89c", "signalId": "github:vllm-project/vllm:pull_request:45584", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45584", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: trim trailing MTP entry from text_config.layer_types", "text": "## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stack. (Still present in the current `LGAI-EXAONE/EXAONE-4.5-33B-FP8` config: `num_hidden_layers=64`, `len(layer_types)=65`.) On `transformers` v5 the inner text-config is instantiated as `Exaone4Config` — the `exaone4_5_text` model type is remapped to `exaone4` by the upstream `Exaone4_5_Config.__post_init__` — and `Exaone4Config` has no notion of the MTP layer. Its `@strict` dataclass init runs `validate_layer_type` and fails before vLLM ever reaches the model loader: ``` ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` Full traceback from `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'`: ``` File \".../transformers/models/exaone4_5/configuration_exaone4_5.py\", line 84, in __post_init__ self.text_config = CONFIG_MAPPING[self.text_config[\"model_type\"]](**self.text_config) File \".../huggingface_hub/dataclasses.py\", line 276, in init_with_validate cls.validate(self) huggingface_hub.errors.StrictDataclassClassValidationError: Class validation error for validator 'validate_layer_type': ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` ## Fix Register a vLLM-side `Exaone4_5Config` (`vllm.transformers_utils.configs.exaone4_5`) that subclasses the upstream config and trims the trailing `layer_types` entry before the inner `Exaone4Config` is built. The trim is gated on `len(layer_types) == num_hidden_layers + 1` exactly; any other shape is left untouched so unrelated config bugs are not silently masked. The MTP layer is consumed separately by the speculative-decode path via `num_nextn_predict_layers` (and the MTP weights come from the `mtp.*` checkpoint keys), so dropping the metadata entry in `layer_types` does not change MTP behavior — it only unblocks the inner `Exaone4Config` validation. 3 files, 60 lines added: - `vllm/transformers_utils/configs/exaone4_5.py` (new) - `vllm/transformers_utils/configs/__init__.py` (+2 — register in `_CLASS_TO_MODULE` and `__all__`) - `vllm/transformers_utils/config.py` (+1 — map `exaone4_5` → `Exaone4_5Config` in `_CONFIG_REGISTRY`) ## Not a duplicate No open PR or issue currently addresses the EXAONE 4.5 `layer_types` / MTP config validation failure. The recently merged #45073 fixes a separate surface (the vision encoder `sequence_lengths` kwarg) and does not touch the text-config validation path. ## Test Manually verified end-to-end on: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before the patch, `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails at engine init with the `validate_layer_type` ValueError above. After the patch the same command starts cleanly and the speculative-decode path is active: ``` INFO [model.py:611] Resolved architecture: Exaone4_5_MTP INFO [llm_base_proposer.py:1347] Detected MTP model. Sharing target model embedding weights with the draft model. INFO [llm_base_proposer.py:1403] Detected MTP model. Sharing target model lm_head weights with the draft model. ``` `vllm bench serve --dataset-name sonnet --dataset-path benchmarks/sonnet.txt --sonnet-input-len 550 --sonnet-output-len 256 --sonnet-prefix-len 200 --num-prompts 64 --max-concurrency 16 --request-rate inf` against `LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'`: ``` ============ Serving Benchmark Result ============ Successful requests: 64 Maximum request concurrency: 16 Benchmark duration (s): 24.06 Total input tokens: 34915 Total generated tokens: 16384 Request throughput (req/s): 2.66 Output token throughput (tok/s): 680.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.51 Median TPOT (ms): 20.66 P99 TPOT (ms): 24.84 ---------------Speculative Decoding--------------- Acceptance rate (%): 35.91 Acceptance length: 2.08 Per-position acceptance (%): Position 0: 59.64 Position 1: 31.64 Position 2: 16.45 ================================================== ``` The per-position monotonic decay (≈60% / 32% / 16%) is consistent with the published MTP behavior for this checkpoint family on real text. (A control run with `--dataset-name random` reports near-zero acceptance, as expected.) ## AI assistance AI assistance (Claude) was used while developing this change. I reviewed every changed line, understand the fix end-to-end, and ran the verification above myself on my own RTX PRO 6000 instance. The AI contribution is recorded with a `Co-authored-by:` trailer on the commit per `AGENTS.md`.", "url": "https://github.com/vllm-project/vllm/pull/45584", "createdAt": "2026-06-14T13:32:28Z", "updatedAt": "2026-08-13T13:40:29Z", "timestamp": "2026-08-13T13:40:29Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "bug" ], "author": "seongyun1104", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:5230a417ac4176a9748d", "signalId": "github:vllm-project/vllm:pull_request:49879", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49879", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[FEAT] Support fast engine recovery through weight cache", "text": "## Purpose Author: Siyu Liu @liusy58 Michael Qiu ; --- Engine restarts are dominated by weight loading from disk. This PR proposes a persistent per-GPU daemon holds post-quantized, TP-sharded weights in GPU memory; restarting engines map them via CUDA IPC (zero-copy) instead of reloading from disk. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49879", "createdAt": "2026-07-26T13:26:41Z", "updatedAt": "2026-08-13T13:35:11Z", "timestamp": "2026-08-13T13:35:11Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "needs-rebase", "nvidia" ], "author": "liusy58", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:6fa0814cba8813089563", "signalId": "github:vllm-project/vllm:pull_request:52161", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52161", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Detect all attention-spelling variants in ModelConfig.is_hybrid", "text": "## Purpose `ModelConfig.is_hybrid`'s carve-out for granite-4.0-micro (a hybrid-capable architecture with zero actual mamba layers) only matched `layer_types` entries spelled `\"attention\"`. Transformers >= 5.13 normalizes pure-attention entries to `\"full_attention\"` instead (the model's own `granitemoehybrid.py` already handles both spellings in `ALL_DECODER_LAYER_TYPES` for layer dispatch, but `is_hybrid` never learned the new spelling). As a result `is_hybrid` stayed `True` for granite-4.0-micro. Combined with prefix caching now defaulting `mamba_cache_mode` to `\"align\"` for any hybrid-flagged model, `_get_mamba_bufs()` runs unconditionally and hits `AssertionError: no mamba layers in the model` on the first inference request — the engine starts and loads fine, then dies on the first completion. Fix: match both spellings in the carve-out. AI assistance was used to root-cause and author this fix; I reviewed every changed line and reproduced/verified the behavior end-to-end myself before opening this PR. ## Test Plan Reproduced on vLLM HEAD `37c3bdf5a` (2026-08-13): ``` vllm serve ibm-granite/granite-4.0-micro --max-model-len 8192 # then, once the server is up: curl http://localhost:8000/v1/completions \\ -H 'Content-Type: application/json' \\ -d '{\"model\":\"ibm-granite/granite-4.0-micro\",\"prompt\":\"The capital of France is\",\"max_tokens\":10}' ``` Then applied the fix and repeated the same two steps, plus a direct property check: ```python from vllm.config import ModelConfig mc = ModelConfig(model=\"ibm-granite/granite-4.0-micro\") print(mc.is_hybrid) ``` ## Test Result **Before fix:** server starts and loads the model successfully, then the completion request fails the engine: ``` AssertionError: no mamba layers in the model File \".../vllm/v1/worker/gpu_model_runner.py\", line 4437, in execute_model mamba_bufs = self._get_mamba_bufs() File \".../vllm/v1/worker/mamba_utils.py\", line 554, in get_mamba_groups assert len(mamba_group_ids) > 0, \"no mamba layers in the model\" vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` `ModelConfig(\"ibm-granite/granite-4.0-micro\").is_hybrid` → `True` (incorrect). **After fix:** `is_hybrid` → `False` (correct — this model has no mamba layers). Server starts and the same completion request succeeds: ```json {\"id\":\"cmpl-...\",\"choices\":[{\"text\":\" Paris.\\n\\n**Output:**\\n...\"}], ...} ``` No crash, no assertion. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52161", "createdAt": "2026-08-13T11:34:51Z", "updatedAt": "2026-08-13T13:32:59Z", "timestamp": "2026-08-13T13:32:59Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "ready" ], "author": "mganczarenko", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4cfbdbd9dd668983cdff", "signalId": "github:vllm-project/vllm:pull_request:52050", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52050", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fall back to FA2 for Blackwell head-dim-256 paged attention", "text": "## Purpose I came across this issue while enabling ColPali for model runner v2 pooling migration. ColPali MRV2 exposed the issue because its decoder attention supplies `seqused_k,` while FA4’s specialized SM100 head-dim-256 kernel currently rejects `seqused_q/k`. This caused the inference to abort after selecting FA4. This PR consistently resolves FA2 when sequence-length tensors are required, rejects unsupported FP8, DiffKV, and FA4-only masking combinations during backend selection, and preserves FA4 for encoder and other callers that do not pass `seqused_q/k`. ## Reproducer ```bash CUDA_VISIBLE_DEVICES=0 \\ VLLM_USE_V2_MODEL_RUNNER=1 \\ VLLM_ATTENTION_BACKEND=FLASH_ATTN \\ .venv/bin/python -m pytest \\ tests/models/multimodal/pooling/test_colpali.py::test_colpali_multimodal_text_query_image_docs \\ -v ``` ## Output on main / on branch ```text main: AssertionError: SM100 forward with head_dim=256 does not support seqused_q/seqused_k branch: PASSED [100%] 1 passed, 15 warnings in 85.21s ``` ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/kernels/attention/test_attention_selector.py \\ -k sm100_hd256 -q ``` ```text 3 passed, 38 deselected, 20 warnings in 0.89s ``` ```bash CUDA_VISIBLE_DEVICES=0 \\ VLLM_ATTENTION_BACKEND=FLASH_ATTN \\ .venv/bin/python -m pytest \\ tests/models/multimodal/pooling/test_colpali.py::test_colpali_multimodal_text_query_image_docs \\ -v ``` ```text 1 passed, 15 warnings in 85.21s ``` ```bash .venv/bin/pre-commit run --files \\ $(git diff --name-only upstream/main...HEAD) ``` ```text All hooks passed ``` ## Model evaluation Using `vidore/colpali-v1.3-hf`: | Comparison | Result | |---|---| | V1 versus MRV2 query output | Exact match | | V1 versus MRV2 image cosine | `0.999953` red, `0.999977` blue | | V1 MaxSim | red `3.766706`, blue `3.047093` | | MRV2 MaxSim | red `3.766586`, blue `3.047139` | | HF MaxSim | red `3.743096`, blue `3.112920` | V1, MRV2, and the HF reference all produced the same document ranking. ## Performance impact On Blackwell, decoder attention with `head_size=256` and `seqused_q/k` now uses FA2. A direct FA4 performance comparison is unavailable because FA4 aborts for this input. Encoder and other paths that do not require these tensors continue using FA4. ## AI assistance disclosure OpenAI Codex (GPT-5) assisted with drafting the code. I reviewed every changed line and personally ran the tests and model evaluations listed above.", "url": "https://github.com/vllm-project/vllm/pull/52050", "createdAt": "2026-08-12T21:30:39Z", "updatedAt": "2026-08-13T13:29:20Z", "timestamp": "2026-08-13T13:29:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "multi-modality" ], "author": "taneem-ibrahim", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4dc42504f590a99e18df", "signalId": "github:vllm-project/vllm:pull_request:52169", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52169", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Note pooled aggregation semantics for mean_acceptance_length", "text": "## Purpose `SpecDecodingLogging._log` and the PromQL recipe in `SpecDecodingProm.__doc__` compute the `mean_acceptance_length` as a **pooled mean** (weighted by draft count, equivalent to `total_generated / total_verify_calls`), not an unweighted mean of per-request means. Some external benchmarks report the latter — SpecForge's `average_acceptance_length`, for example, is `statistics.fmean(acceptance_lengths)` over per-request lengths. On variable-length workloads the two aggregations can differ by several points in either direction, depending on the sign of the per-request-length ↔ acceptance-rate correlation. This has confused at least one report already (issue #42508: a 55.08%-vs-44% pooled-vs-mean-of-means gap for EAGLE3 on GSM8K, and 66.57%-vs-72% in the opposite direction for the standalone drafter — same numerator, different aggregation). ## Changes Docstring/comment only. Adds a short aggregation note - inline where `mean_acceptance_length` is computed, and - inside `SpecDecodingProm.__doc__` below the existing PromQL recipe. Both points cross-reference #42508 for the reproduction. The metric definitions and the CLI/Prometheus output are byte-identical to before. ## Test Plan None — comment/docstring only, no runtime behavior changed. `ruff format --check` clean; pre-commit hooks pass locally. ## Test Result n/a ## Documentation Update n/a (this PR *is* the documentation update). ## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [x] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1MBOTQeE4aOFIL8UkuGRyGXe6qYtsIt1FGWyxTvvIvtM/edit?tab=t.0).", "url": "https://github.com/vllm-project/vllm/pull/52169", "createdAt": "2026-08-13T13:28:14Z", "updatedAt": "2026-08-13T13:28:51Z", "timestamp": "2026-08-13T13:28:51Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "speculative-decoding" ], "author": "arrdel", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:e29a9b8bdaaa0e0ebef1", "signalId": "github:vllm-project/vllm:pull_request:52117", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52117", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Do not require SupportsPP of draft models under pipeline parallelism", "text": "Fixes #52069. `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so `_verify_args` demands `SupportsPP` from the draft model. No head in `MTPModelTypes` implements it, so every MTP speculative config dies in `create_engine_config` under PP > 1. A draft model always runs whole on a single PP rank (the V2 runner builds it on the last rank only), so this verifies it as single-stage, on a shallow copy of the config. **Test plan:** without the change, `--pipeline-parallel-size 4 --speculative-config '{\"method\":\"mtp\",...}'` fails at config time on GLM-4.5-Air; with it, the server loads and serves (8× RTX 3090, three machines, nightlies dev678–dev693). Complementary to #46994, which does not touch this file. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52117", "createdAt": "2026-08-13T07:22:01Z", "updatedAt": "2026-08-13T13:28:30Z", "timestamp": "2026-08-13T13:28:30Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b596865d680179aa3ea9", "signalId": "github:vllm-project/vllm:issue:42508", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:42508", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Inconsistent speculative decoding acceptance metrics between vLLM and SpecForge on Qwen3-32B baselines", "text": "### Proposal to improve performance _No response_ ### Report of performance regression _No response_ ### Misc discussion on performance I am benchmarking speculative decoding baselines in vLLM and observed inconsistent acceptance metrics compared with SpecForge. I would like to understand whether this difference is expected due to different metric definitions, verification logic, benchmark prompt formatting, or whether it may indicate a configuration mismatch or a potential issue in the vLLM speculative decoding benchmark path. The target model is Qwen3-32B. I tested two speculative decoding baselines: Qwen3-32B with Eagle3-Qwen3-32B-zh, and Qwen3-32B with Qwen3-0.6B as a standalone draft model. In both cases, I set num_speculative_tokens to 3, used max concurrency / batch size 16, greedy decoding with temperature=0 and top_p=1.0, output length 1024, and the same full GSM8K test set with 1319 samples for both vLLM and SpecForge.My intention was to compare chain-style speculative decoding baselines, not tree-style candidate expansion. For the EAGLE3 baseline, I launched vLLM with the following command: VLLM_CACHE_ROOT=/root/.cache/vllm_eagle3_argmax_gpu0 \\ SPEC_DECODE_MODE=baseline \\ CUDA_VISIBLE_DEVICES=0 vllm serve /root/autodl-tmp/Qwen3-32B \\ --served-model-name Qwen3-32B \\ --host 0.0.0.0 \\ --port 8000 \\ --trust-remote-code \\ --dtype bfloat16 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.85 \\ --max-model-len 4096 \\ --max-num-seqs 16 \\ --enable-chunked-prefill \\ --speculative-config '{ \"method\": \"eagle3\", \"model\": \"/root/autodl-tmp/Eagle3-Qwen3-32B-zh\", \"draft_tensor_parallel_size\": 1, \"num_speculative_tokens\": 3 }' The vLLM benchmark command was: vllm bench serve \\ --backend openai \\ --base-url http://127.0.0.1:8000 \\ --endpoint /v1/completions \\ --model Qwen3-32B \\ --tokenizer /root/autodl-tmp/Qwen3-32B \\ --dataset-name custom \\ --dataset-path /root/autodl-tmp/gsm8k_vllm_custom_mt1024.jsonl \\ --custom-output-len 1024 \\ --num-prompts 1319 \\ --request-rate inf \\ --max-concurrency 16 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,95,99 \\ --save-result \\ --save-detailed \\ --result-dir /root/vllm/results \\ --result-filename qwen32b_eagle3_argmax_chain3_gsm8k_custom_np1319_mt1024_bs16_temp0_topp1_gpu0_vllmbench.json \\ --extra-body '{\"temperature\":0,\"top_p\":1.0}' \"Image\" The vLLM result shows a mean acceptance rate of 55.08% and a mean accepted length of 2.65. For the same EAGLE3 baseline, I used SpecForge with the following command: CUDA_VISIBLE_DEVICES=0 python3 bench_eagle3.py \\ --model-path /root/autodl-tmp/Qwen3-32B \\ --speculative-draft-model-path /root/autodl-tmp/Eagle3-Qwen3-32B-zh \\ --port 30000 \\ --host http://127.0.0.1 \\ --trust-remote-code \\ --tp-size 1 \\ --attention-backend fa3 \\ --mem-fraction-static 0.7 \\ --config-list 16,3,1,4 \\ --benchmark-list gsm8k \\ --output-dir ./test \\ --dtype bfloat16 \"Image\" The SpecForge result shows a mean accepted length of 2.32, and the derived mean acceptance rate is around 44%. Therefore, for the EAGLE3 baseline, vLLM reports about 10 percentage points higher acceptance rate than SpecForge. For the standalone draft model baseline, I launched vLLM with the following command: VLLM_CACHE_ROOT=/root/.cache/vllm_qwen0p6b_argmax_gpu0 \\ SPEC_DECODE_MODE=baseline \\ CUDA_VISIBLE_DEVICES=0 vllm serve /root/autodl-tmp/Qwen3-32B \\ --served-model-name Qwen3-32B \\ --host 0.0.0.0 \\ --port 8000 \\ --trust-remote-code \\ --dtype bfloat16 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.85 \\ --max-model-len 4096 \\ --max-num-seqs 16 \\ --enable-chunked-prefill \\ --speculative-config '{ \"method\": \"draft_model\", \"model\": \"/root/autodl-tmp/Qwen3-0.6B\", \"draft_tensor_parallel_size\": 1, \"num_speculative_tokens\": 3 }' The vLLM benchmark command was: vllm bench serve \\ --backend openai \\ --base-url http://127.0.0.1:8000 \\ --endpoint /v1/completions \\ --model Qwen3-32B \\ --tokenizer /root/autodl-tmp/Qwen3-32B \\ --dataset-name custom \\ --dataset-path /root/autodl-tmp/gsm8k_vllm_custom_mt1024.jsonl \\ --custom-output-len 1024 \\ --num-prompts 1319 \\ --request-rate inf \\ --max-concurrency 16 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,95,99 \\ --save-result \\ --save-detailed \\ --result-dir /root/vllm/results \\ --result-filename qwen32b_qwen0p6b_ar_argmax_chain3_gsm8k_custom_np1319_mt1024_bs16_temp0_topp1_gpu0_vllmbench.json \\ --extra-body '{\"temperature\":0,\"top_p\":1.0}' \"Image\" The vLLM result shows a mean acceptance rate of 66.57% and a mean accepted length of 3.00. For the same standalone draft model baseline, I used SpecForge with the following command: CUDA_VISIBLE_DEVICES=0 python3 bench_speculative.py \\ --model-path /root/autodl-tmp/Qwen3-32B \\ --speculative-algorithm STANDALONE \\ --speculative-draft-model-path /root/autodl-tmp/Qwen3-0.6B \\ --port 30000 \\ --host http://127.0.0.1 \\ --trust-remote-code \\ --tp-size 1 \\ --attention-backend fa3 \\ --mem-fraction-static 0.7 \\ --config-list 16,3,1,4 \\ --benchmark-list gsm8k \\ --output-dir ./test \\ --dtype bfloat16 \"Image\" The SpecForge result shows a mean accepted length of 3.17, and the derived mean acceptance rate is around 72%. Therefore, for the standalone draft model baseline, vLLM reports about 6 percentage points lower acceptance rate than SpecForge. The direction of the difference is not consistent. For EAGLE3, vLLM reports higher acceptance than SpecForge. For the standalone draft model baseline, vLLM reports lower acceptance than SpecForge. I am mainly trying to understand whether the above difference is expected, or whether it suggests a configuration mismatch or a possible issue in the speculative decoding benchmark path. ### Your current environment (if you think it is necessary) ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by Anaconda, Inc. | (main, Mar 19 2026, 20:20:58) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-5.15.0-171-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H20 GPU 1: NVIDIA H20 GPU 2: NVIDIA H20 GPU 3: NVIDIA H20 GPU 4: NVIDIA H20 GPU 5: NVIDIA H20 GPU 6: NVIDIA H20 GPU 7: NVIDIA H20 Nvidia driver version : 580.105.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 224 On-line CPU(s) list: 0-223 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8480+ CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 56 Socket(s): 2 Stepping: 8 CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 5.3 MiB (112 instances) L1i cache: 3.5 MiB (112 instances) L2 cache: 224 MiB (112 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-55,112-167 NUMA node1 CPU(s): 56-111,168-223 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.590.48 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchvision==0.26.0 [pip3] transformers==5.6.2 [pip3] triton==3.6.0 [conda] flashinfer-python 0.6.8.post1 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.0.3 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.18.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.4.2 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.4.2 pypi_0 pypi [conda] nvidia-ml-py 13.590.48 pypi_0 pypi [conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] torch 2.11.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchvision 0.26.0 pypi_0 pypi [conda] transformers 5.6.2 pypi_0 pypi [conda] triton 3.6.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.2rc1.dev42+g98661fe01 (git sha: 98661fe01) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYS SYS 0-55,112-167 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE SYS SYS 0-55,112-167 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE 56-111,168-223 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE NODE 56-111,168-223 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE PIX 56-111,168-223 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE 56-111,168-223 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS NODE X PIX NODE SYS SYS NIC2 NODE NODE NODE NODE SYS SYS SYS SYS NODE PIX X NODE SYS SYS NIC3 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE X SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NIC5 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 ============================== Environment Variables ============================== PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_test001 ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/42508", "createdAt": "2026-05-13T08:50:11Z", "updatedAt": "2026-08-13T13:28:25Z", "timestamp": "2026-08-13T13:28:25Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "performance" ], "author": "Mterzzl", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f4661db93a6d7061421b", "signalId": "github:vllm-project/vllm:pull_request:52021", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve Anthropic disable_parallel_tool_use", "text": "## Purpose Anthropic’s `disable_parallel_tool_use` was silently discarded, leaving the converted OpenAI request with `parallel_tool_calls=True`. This preserves the field and maps it to the existing inverse OpenAI setting. ## Reproducer ```python request = AnthropicMessagesRequest( model=\"test\", messages=[{\"role\": \"user\", \"content\": \"Use a tool\"}], max_tokens=8, tool_choice={\"type\": \"auto\", \"disable_parallel_tool_use\": True}, ) converted = AnthropicServingMessages._convert_anthropic_to_openai_request(request) print(request.tool_choice.model_dump(exclude_none=True)) print(converted.parallel_tool_calls) ``` **On Main** ```text {'type': 'auto'} True ``` **On this branch** ```text {'type': 'auto', 'disable_parallel_tool_use': True} False ``` ## Test Plan and Results ```bash .venv/bin/python -m pytest tests/entrypoints/anthropic/test_anthropic_messages_conversion.py -q ``` ```text # 51 passed ```", "url": "https://github.com/vllm-project/vllm/pull/52021", "createdAt": "2026-08-12T16:49:55Z", "updatedAt": "2026-08-13T13:28:20Z", "timestamp": "2026-08-13T13:28:20Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "bug", "frontend", "ready" ], "author": "taneem-ibrahim", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:d27ccb847604e31e08c3", "signalId": "github:vllm-project/vllm:pull_request:52168", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Restore multimodal support on the plain \"vllm\" throughput backend", "text": "## Purpose `vllm bench throughput --backend vllm` crashes on any multimodal dataset (e.g. ASR/Whisper) with: ValueError: Multi-modal content is not supported on backend 'vllm'; use one of []. Regression from #50981. `get_requests()` computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handle: ```python mm_backends = (\"vllm-chat\",) if args.backend == \"vllm-chat\" else () ``` This is circular — it only lets the already-selected backend through the gate, and only if it's literally \"vllm-chat\". So --backend vllm always computes mm_backends=() and self-rejects, even though _run_vllm_requests (the function that executes --backend vllm requests) already forwards multi_modal_data straight into LLM.generate — it was always multimodal-capable, same as vllm-chat. Only hf (asserts prompts must be str) and mii (text-only) genuinely can't carry multimodal content, so the allowlist should be static rather than derived from the chosen backend: ```python mm_backends = (\"vllm\", \"vllm-chat\") ``` ## Test Plan - Added test_get_requests_allows_multimodal_on_plain_vllm_backend to tests/benchmarks/test_throughput_cli.py, mirroring the existing test_get_requests_resolves_mmvu pattern: stubs ASRDataset (network-free), parses --backend vllm --dataset-name hf --dataset-path openslr/librispeech_asr, asserts get_requests() resolves without raising. - Ran the exact vllm bench throughput command from the bug report end-to-end on real hardware (Intel XPU B70) before and after the fix. - ruff check / ruff format on both changed files. ##Test Result - New test: fails with the reported ValueError against pre-fix code, passes against post-fix code. - python -m pytest tests/benchmarks/test_throughput_cli.py -k \"allows_multimodal_on_plain_vllm_backend or resolves_mmvu\" → 2 passed. - End-to-end on B70: pre-fix reproduces the ValueError verbatim; post-fix completes cleanly (Throughput: 20.96 requests/s, 5386.74 total tokens/s). - ruff check: all checks passed. ruff format --diff: no changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52168", "createdAt": "2026-08-13T13:27:39Z", "updatedAt": "2026-08-13T13:28:16Z", "timestamp": "2026-08-13T13:28:16Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "performance" ], "author": "mganczarenko", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:2989e872a87419af00cd", "signalId": "github:vllm-project/vllm:pull_request:51182", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51182", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] [Build] Fix DeepGEMM SM90 paged mqa `prefix_sum` out of range", "text": "## Purpose Fix #51181. DeepGEMM SM90 paged mqa logit contains an out-of-index error which is easily hit in common device setup (DeepSeek V4 DSpark on H100 and common `max-num-seqs`). This PR prevents the out-of-index error (`CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS` on CUDA coredump) by building DeepGEMM from bugfixed branch (vllm-project/DeepGEMM#2). **TODO**: rollback the DeepGEMM repo to vllm-project after vllm-project/DeepGEMM#2 merged. ## Test Plan Run the reproducer in #51181 and check that CUDA graph capture runs successfully. ## Test Result I have checked the validity of the referred DeepGEMM fork branch by installing the source-built DeepGEMM wheel atop of latest(beca88e) main CI image and run the reproducer in #51181. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51182", "createdAt": "2026-08-05T17:22:31Z", "updatedAt": "2026-08-13T13:28:00Z", "timestamp": "2026-08-13T13:28:00Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "cjackal", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:ee85e2801cfb40ef16ea", "signalId": "github:vllm-project/vllm:pull_request:46994", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec][V2] Support MTP speculative decoding under pipeline parallelism", "text": "## Purpose MTP speculative decoding does not currently work under pipeline parallelism on the V2 model runner. This PR makes it functional for DeepSeek-family MTP drafts (DeepSeek-V3, GLM-5.2, Qwen3.5/3.6, …). Five independent issues, all on the PP>1 path. Fixes #1-#3 are DeepSeek-family-specific; fix #4 applies to all models using sparse MLA attention; fix #5 applies to Qwen3.5/3.6 MTP draft models on the last PP rank. **1. `DeepSeekMTP` does not implement `SupportsPP`** — DeepSeek-family specific. The engine refuses to build the draft model under PP at all: ``` NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface. [DeepSeekMTPModel] ``` The MTP draft runs only on the last PP stage, so it never actually consumes PP intermediate tensors, but the interface still requires the `make_empty_intermediate_tensors` factory. (This mirrors what #39704 does for the V1 runner.) **2. `PPHandler` sampled-token broadcast width mismatch (hang)** — affects all MTP under PP. `broadcast()` sends `sampled_token_ids` at its natural width — 1 on any step with no draft tokens (prefill, first decode), `num_spec+1` once rejection sampling has run — while `receive()` always posts a fixed `[num_reqs, max_sample_len]` buffer. NCCL `broadcast` doesn't negotiate element counts, so a width-1 send against a width-`max_sample_len` receive is a count mismatch that deadlocks the receiver. Fix: pad the source to `max_sample_len` (trailing `-1`, ignored by `post_update`). **3. Proposed draft tokens are never relayed to non-last PP ranks (garbage output / ~0 acceptance)** — affects all MTP under PP. `req_states.draft_tokens` is written only on the last rank (the `propose()` path); non-last ranks keep the zero-init buffer. `combine_sampled_and_draft_tokens` then embeds zeros at the draft positions on rank 0, so the verification input is wrong. Fix: coalesce a third broadcast (the proposed draft tokens) into the existing deferred `PPHandler` sibling-group broadcast, and scatter it into `req_states.draft_tokens` on consume. No new collective; gated identically to the sampled-token broadcast so per-step op counts stay matched. **4. Stale `topk_indices_buffer` reference in sparse MLA backends (the acceptance fix)** — affects all models using sparse MLA attention. Under MTP+PP, `FlashAttnMLASparseImpl.__init__` stored `indexer.topk_indices_buffer` at construction time. When `_maybe_share_lm_head` later replaced `Indexer.topk_indices_buffer` with the target model's buffer, the impl's reference was stale — still pointing to the draft model's original (uninitialized) buffer. This caused garbage DSA attention → degenerate \"repeat-the-current-token\" drafts → ~27-33% acceptance instead of ~85%. Fix: store `self._indexer = indexer` in `__init__`, read `self._indexer.topk_indices_buffer` dynamically in `forward_mqa`. Applied to all three sparse MLA backends: `flashattn_mla_sparse.py`, `flashmla_sparse.py`, `flashinfer_mla_sparse.py`. **5. Apply `fc` projection on last PP rank for Qwen3.5 MTP** — Qwen3.5/3.6-specific. Under MTP+PP, the Qwen3.5 MTP draft model on the last PP rank was using the target model's hidden_states directly, bypassing the `fc` projection entirely. This produced essentially random predictions (~1% acceptance) because the draft model's input was not properly projected. Fix: on the last PP rank, apply the same `fc` projection as the first rank (embed `input_ids`, normalize, concat with `hidden_states`, project through `fc`). This is the same pattern used on PP0 (first rank). ## Test Plan Serve a DeepSeek-family or Qwen3.5/3.6 MTP model under PP on the V2 runner and check it boots, produces correct output, and accepts drafts at a normal rate: ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve zai-org/GLM-5.2-FP8 \\ --tensor-parallel-size 4 --pipeline-parallel-size 2 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' ``` Unit tests (`tests/v1/worker/test_pp_utils.py`): - `test_deepseek_mtp_implements_supports_pp` — verifies Fix `#1` - `test_pphandler_broadcast_pads_to_max_sample_len` — verifies Fix `#2` - `test_sparse_mla_backend_reads_topk_indices_buffer_dynamically` — verifies Fix `#4` ## Test Result Validated on GLM-5.2-FP8 (GlmMoeDsaForCausalLM, DeepSeek-Sparse-Attention MoE), TP4/PP2, on a current-main base (0.23.1rc1.dev531), serving real traffic for 5+ hours at K=3: | Metric | Value | |---|---| | Drafts | 91,088 | | Draft tokens | 273,264 | | Accepted tokens | 230,722 | | **Overall acceptance** | **84.4%** | | pos0 | 92.1% | | pos1 | 84.0% | | pos2 | 77.2% | The stale-buffer fix (fix #4) lifts acceptance from ~27-33% (pre-fix, fixes #1-#3 only) to **84.4%** at K=3 over 5+ hours of real traffic — matching the non-PP TP8 baseline (~85%). ### Cross-model validation: fix #4 applies broadly The stale-buffer fix was further validated across 8 Qwen3.5/3.6 model variants (dense BF16, AWQ, MoE BF16, MoE AWQ, MoE GPTQ-Int4), each at MTP K=1/2/3, PP=2/TP=1. All show high acceptance (83-96%) that scales gracefully with K, confirming the fix is not architecture-specific. | Model | Quant | K=1 | K=2 | K=3 | |---|---|---|---|---| | Qwen3.5-27B-AWQ | AWQ 4-bit | 95.3% | 90.5% | 85.8% | | Qwen3.5-27B (BF16) | BF16 | 95.5% | 91.1% 86.0% | | Qwen3.5-35B-A3B (MoE, BF16) | BF16 | 93.6% | 88.0% | 82.0% | | Qwen3.6-27B (BF16) | BF16 | 95.2% | 91.1% | 86.0% | | Qwen3.6-27B-AWQ | AWQ 4-bit | 95.3% | 90.5% | 85.8% | | Qwen3.6-35B-A3B (BF16) | BF16 | 94.1% | 88.6% | 83.8% | | Qwen3.6-35B-A3B-AWQ | AWQ 4-bit | 94.0% | 88.8% | 83.0% | | Qwen3.6-35B-A3B-GPTQ-Int4 | GPTQ 4-bit | 94.1% | 88.9% | 83.5% | Full per-position breakdown and GSM8K accuracy available on request. ### Remaining stale-buffer backends Fix #4 was also applied to the three remaining sparse MLA backends that were not covered by the original fix commit (`c175667db`): - `flashinfer_mla_sparse_sm120.py` — `FlashInferMLASparseSM120Impl` (SM120 variant) - `rocm_aiter_mla_sparse.py` — `ROCmAiterMLASparseImpl` (ROCm) - `xpu_mla_sparse.py` — `XPUMLASparseImpl` (Intel XPU) These backends have the same stale `topk_indices_buffer` bug but are not reachable on our hardware (H200/SM90). The fix follows the exact same pattern as the already-validated fix: store `self._indexer = indexer` in `__init__`, read `self._indexer.topk_indices_buffer` dynamically in `forward_mqa`.", "url": "https://github.com/vllm-project/vllm/pull/46994", "createdAt": "2026-06-29T05:01:54Z", "updatedAt": "2026-08-13T13:27:46Z", "timestamp": "2026-08-13T13:27:46Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "rocm", "intel-gpu", "needs-rebase", "v1", "qwen", "deepseek", "nvidia", "mrv2" ], "author": "eastwood-c", "state": "open", "assignees": [ "njhill" ], "change": "new" } }, { "id": "event:5f9dad274fb73ea1394e", "signalId": "github:vllm-project/vllm:issue:52125", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52125", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: vLLM Native Reusable Engine Snapshot Sleep (L3)", "text": "## Motivation. A vLLM cold start includes model weight loading, EngineCore initialization, CUDA context and allocator creation, `torch.compile`, CUDA Graph capture, KV cache profiling, and warmup. These steps are repeated for every new instance. This impacts responsiveness in elastic scaling and multi-model deployment scenarios. When an instance is released and re-launched, or when a new replica is scaled out from an existing one, the full initialization pipeline runs again. During model switching, previously completed compilation and runtime state are also difficult to reuse directly. [vLLM #34303](https://github.com/vllm-project/vllm/issues/34303) framed this as CUDA Checkpoint/Restore for Near-Zero Cold Starts and proposed combining CRIU to persist full process state. This RFC continues along that CRIU-based persistence path, using CRIU and CUDA C/R to save an already-initialized vLLM Engine. vLLM's existing L1/L2 sleep modes primarily manage GPU allocations such as weights and KV cache, while the EngineCore process remains alive. They are suitable for short-term HBM release, but cannot cause EngineCore to exit, nor produce a reusable artifact that can be restored multiple times. Once the process is destroyed, the next launch must redo Engine initialization from scratch. This RFC proposes adding a reusable Engine Snapshot as an L3-level sleep mode in vLLM: save the initialized Engine and CUDA state, allow the source process to exit after the snapshot is committed, and restore directly back to a serving state on wake. ## Core Goals and Design Philosophy Container-level snapshot solutions such as Dynamo have already demonstrated that CRIU and CUDA Checkpoint can jointly save and restore inference processes containing GPU state. These solutions treat the entire container as the snapshot boundary, and the restore process depends on coordination among operators, node agents, and CRI runtimes — tightly coupled with the Kubernetes control plane and runtime environment, making them suitable for scenarios where Kubernetes uniformly manages instance lifecycles. This proposal defines the snapshot boundary at the EngineCore level within vLLM and integrates with the existing sleep/wake lifecycle. Before entering deep sleep, vLLM drains requests and brings workers and communicators into a saveable state. After the snapshot is saved, the EngineCore process exits. On wake, EngineCore is restored, necessary runtime connections are re-established, and the Engine is confirmed functional before accepting requests again. When to save, what internal state to prepare, and at what stage of restore the Engine is considered ready — all of these are controlled by vLLM itself. This proposal provides deeper sleep capability than L2, and users can invoke it according to their own scheduling policies. Within this framework, generated snapshots can be restored repeatedly, and the approach can evolve to support cross-machine restore, 1→N, multi-GPU, and distributed scenarios. From the user's perspective, sleep depth is progressively layered: | Mode | State Retained After Sleep | |----|--------------------------| | L1 | EngineCore keeps running, weights retained as host-side backup, KV cache released | | L2 | EngineCore keeps running, weights and KV cache released, reloaded on wake | | Snapshot sleep (L3) | EngineCore saved as snapshot then exits, frontend and launcher continue running | L3 exposes two snapshot modes. `full` preserves weights, KV cache, and runtime state in the CUDA process image. `minimized` removes weights and KV cache before capture, then reloads weights from model files and rebuilds KV cache after restore. ## Design Phase 1 targets same-machine, single-GPU, TP=1 runtime sleep/wake. Deep sleep occurs while the service is running. After EngineCore exits, the service must still accept wake requests, maintain sleep state, and preserve the original HTTP listener. Based on this requirement, this proposal decouples the API frontend and EngineCore lifecycles: the frontend continues running during sleep, while `EngineCoreProc` exits and is restored as an independent snapshot object. ### Phase 1 Architecture ```text Before ====== ┌──────────────────────────────────────────┐ │ vllm serve main process │ │ │ │ HTTP / API │ │ MPClient │ │ CoreEngineProcManager │ └──────────────┬────────────────┬──────────┘ │ business ZMQ │ create / monitor │ │ ▼ ▼ ┌──────────────────────────────────────────┐ │ EngineCoreProc │ │ │ │ EngineCore │ │ UniProcExecutor + GPUWorker │ │ weights / KV / CUDA context / HBM │ └──────────────────────────────────────────┘ After Phase 1 ============= ┌──────────────────────────────────────────┐ │ API frontend │ │ │ │ HTTP listener / API │ │ admission / readiness │ │ MPClient │ └──────────────┬────────────────┬──────────┘ │ control │ business ZMQ │ │ ▼ │ ┌───────────────────────────────┼──────────┐ │ launcher │ │ │ │ │ │ CoreEngineProcManager │ │ │ · create / monitor │ │ │ · adopt restored process │ │ │ │ │ │ EngineSnapshotManager [new] │ │ │ · lifecycle state │ │ │ · artifact / provider │ │ └──────────────┬────────────────┘ │ │ │ │ manage + CRIU / CUDA C/R │ │ │ ▼ ▼ ┌──────────────────────────────────────────┐ │ EngineCoreProc subtree │ │ │ │ EngineCore │ │ UniProcExecutor + GPUWorker │ │ weights / KV / CUDA context / HBM │ │ │ │ entire subtree captured / restored │ └──────────────────────────────────────────┘ After deep sleep: launcher continues running API frontend continues running, retains HTTP listener EngineCoreProc saved as snapshot, then exits ``` `HTTP/API` and `MPClient` are split from the original main process into the persistent frontend. EngineCore process management stays in the launcher, with added capability to adopt restored processes. `EngineSnapshotManager` is a new component responsible for snapshot state and provider invocation. The internal structure of `EngineCoreProc` remains unchanged; the entire subtree is saved and restored as a unit. A control channel between the API frontend and launcher handles sleep, wake, and status queries. `MPClient` and EngineCore continue using business ZMQ. The business connection is torn down when EngineCore exits and re-established after restore. ### Phase 1 Lifecycle ```text READY → DRAINING → PREPARING → SNAPSHOTTING → HIBERNATED HIBERNATED → RESTORING → ATTACHING → VERIFYING → READY ``` During the sleep phase, the frontend first closes the request admission gate and confirms EngineCore is idle. EngineCore prepares the state to be saved according to the resource policy, then the provider executes CUDA checkpoint and CRIU dump. After the snapshot is committed, the source `EngineCoreProc` exits, and the system enters `HIBERNATED`. During the wake phase, the system first validates the snapshot and runtime environment, then restores `EngineCoreProc` and CUDA state. The launcher adopts the restored process, and the frontend re-establishes the business connection. After Engine state and inference result checks pass, the service returns to `READY`. ### Future Evolution Phase 1 delivers same-machine, TP=1 EngineCore snapshot and restore. Phase 2 addresses cross-machine snapshot usage. Phase 3 extends to multi-GPU, distributed topologies, and elastic restore. #### Phase 2: Single-Machine, Single-GPU Cross-Machine Restore (with 1→N Support) When moving across machines, state elements carrying source-machine identity — the original filesystem, `/dev/shm`, GPU, and communication addresses — become invalid and must be handled. 1. **vLLM runtime files.** a. **Compilation artifacts.** vLLM uses TorchInductor, AOTAutograd, and Triton. `VLLM_CACHE_ROOT/torch_compile_cache` contains generated Python modules, JIT `.so` files, Triton kernels, `cubin`, and compilation metadata. FlashInfer additionally produces JIT `.so` files, Ninja build files, and autotune results. b. **Shared memory.**`/dev/shm` may contain `psm_*` objects created by Python `SharedMemory`, `torch_*` objects used by PyTorch tensor IPC, and shared files that have been unlinked from the directory but are still held by the process. c. **Hardcoded paths and open files.** The model directory must be available in the target container. EngineCore may also retain absolute paths for cache, working directories, logs, temporary files, and Unix sockets. 2. **Restore fixup.** a. **Environment variables and in-process caches.** Values such as `VLLM_HOST_IP`, `CUDA_VISIBLE_DEVICES`, and `VLLM_RPC_BASE_PATH` may differ on the target instance. Some of these have already been absorbed into `VllmConfig`, worker configuration, or other runtime objects. Modifying `os.environ` alone will not synchronize already-created state; fixup must cover this in-process data. b. **Instance and network identity.** The target instance acquires a new IP, port, GPU UUID, and IPC paths. The ZMQ address between the API frontend and EngineCore, the worker's `distributed_init_method`, and cached device information in the process all need to be rechecked.c. **Prefill-Decode disaggregation connections.** KV Connector handshake information, peer addresses, and transport-layer state all originate from the source instance. After restoring to a new machine, Prefill and Decode instances must rediscover peers and complete the handshake. Connectors such as NIXL, Mooncake, LMCache, and P2P NCCL must also determine which registration states can be preserved and which must be re-established. 3. **(External dependency) Environment compatibility.** a. **Software environment.** File paths, Python, PyTorch, CUDA user-space libraries, Triton, and FlashInfer can be made compatible by using the same container image. b. **Hardware environment.** The CPU must have compatible architecture and instruction sets. The GPU must use the same GPU model, compute capability, and memory specifications. #### Phase 3: Single-Machine Multi-GPU With TP\\>1, a single EngineCore manages multiple workers through the Executor. Each rank has its own process, CUDA context, and GPU state, while being linked together through communication resources such as NCCL and custom all-reduce. The snapshot boundary must expand from a single CUDA process to the entire EngineCore worker group. 1. **Multi-GPU boundaries for CUDA Checkpoint.** a. **Multiple CUDA processes.** vLLM's multiprocessing executor maps each TP rank to one worker process and one GPU. The full set of CUDA PIDs must be identified per rank. b. **Cross-process CUDA resources.** Current CUDA Checkpoint has good support for ordinary CUDA allocations, but CUDA IPC memory still has limitations. vLLM's NCCL P2P, cuMem/NVLS, symmetric memory, and custom all-reduce may all use cross-process GPU mappings. 2. **Communication resources.** a. **NCCL communicator.** Determine whether the communicator is preserved as part of process state, suspended before checkpoint and restored or rebuilt after restore. NCCL internals may also hold P2P mappings, shared memory, and network transport state; different transports have different restorability characteristics. b. **vLLM communicator lifecycle.** PR #46877 has already added `checkpoint_prepare` and `checkpoint_restore` hooks to the communicator, providing a unified entry point for each communication backend. PR #51485 is using `ncclCommSuspend`/`ncclCommResume` to handle NCCL buffers during ordinary sleep. c. **CUDA Graph.** May record collective kernels, communication buffer addresses, and communicator handles. If communication resources change after restore, the original graph may become invalid. #### Phase 4: Multi-Node Multi-GPU In multi-node deployments, workers may be launched by Ray or an external launcher and are no longer part of a single local process tree on the machine where EngineCore resides. The snapshot object must be extended into a cross-node snapshot group. 1. **Cross-node snapshot generation.** a. **Multi-rank synchronization.** When producing a snapshot, all workers must enter the same checkpoint barrier. On restore, each rank must complete process restore, CUDA restore, and communication state handling before EngineCore can resume scheduling requests. b. **Cross-node processes.** Each node can only save its own worker processes and CUDA state. The management model for EngineCore, driver worker, remote workers, and DP coordinator must be clearly defined. 2. **Post-restore topology and identity correction.** a. **Rank placement.** On restore, the original TP/PP/DP ranks must be mapped to target nodes and GPUs, and `CUDA_VISIBLE_DEVICES`, `LOCAL_RANK`, and the rank-to-logical-device mapping must be regenerated. b. **Distributed addresses.**`MASTER_ADDR`, `MASTER_PORT`, `VLLM_HOST_IP`, `distributed_init_method`, DP coordinator addresses, and various ZMQ endpoints may all change. Ray actors are also associated with node IDs, GCS, and actor runtime; processes managed by an external launcher depend on external rendezvous. c. **NCCL network state.** Cross-node NCCL communicators hold bootstrap addresses, sockets, InfiniBand/RDMA queue pairs, GIDs, and memory registrations. ## Relationship to PR #51360 We started this work before PR #51360 came to our attention. Both efforts follow the same basic approach: using CRIU and CUDA C/R to capture an initialized Engine and restoring it from a snapshot to skip Engine initialization. Both currently target the same scope: same-host, single-GPU, TP=1. The main difference lies in when the snapshot is used and the semantics exposed to users. This leads to different snapshot boundaries: 1. **PR #51360 targets instance startup.** It launches a snapshot server from the command line and captures the entire vLLM process tree after Engine initialization but before the HTTP listener is bound. A new instance restores the frontend and EngineCore together, then creates the HTTP listener. Its broader snapshot boundary makes the restore path more direct, since the EngineCore does not need to reconnect to an existing frontend. 2. **This proposal targets runtime deep sleep.** A service can run normally and enter sleep through an HTTP API when it becomes idle. The frontend and HTTP listener remain alive during sleep to handle status and wake requests. The EngineCore is captured and exits, then is restored and reattached to the existing frontend on wake-up. The direct benefit of this proposal is that Engine Snapshot can be exposed as a native vLLM sleep level. A higher-level system can decide when to put a running instance to sleep based on actual load, regardless of how long the service has already been running. The HTTP listener and service address remain unchanged across sleep/wake cycles, so there is no need to restart the service process or recreate the control endpoint. These semantics are better suited to reclaiming resources from running instances while reusing vLLM’s existing sleep/wake API and state management. Both approaches restore an initialized Engine, so we do not expect a fundamental performance difference. PR #51360 benefits from a broader process boundary and a simpler restore path. This proposal handles EngineCore detachment, restoration, and reattachment in exchange for allowing a running service to sleep and wake at any time. We have discussed the overlap with the author of PR #51360 and reached a shared understanding of the first-phase scope. The subsequent work—cross-host restore, GPU relocation, 1→N, multi-GPU, and distributed restore—focuses on snapshot migration and coordinated restore, and does not directly depend on which frontend boundary is chosen in the first phase. ## 5\\. Related Issues, RFCs, and PRs - [#34303](https://github.com/vllm-project/vllm/issues/34303) is the RFC this document directly continues. It proposed using CUDA Checkpoint/Restore to shorten cold starts and listed CRIU-based full-process state persistence as a future direction. This document advances the full-process snapshot path within that framework. - [#51360](https://github.com/vllm-project/vllm/pull/51360) has implemented same-machine, TP=1 initialized-engine snapshot, directly overlapping with Phase 1 of this proposal. The two efforts differ in snapshot boundary and runtime lifecycle; see the previous section for details. - [#35934](https://github.com/vllm-project/vllm/pull/35934), [#37921](https://github.com/vllm-project/vllm/pull/37921), and [#37925](https://github.com/vllm-project/vllm/pull/37925) are earlier CRIU-safe suspend/resume and CUDA checkpoint prototypes, covering scheduler, worker, communicator, and HTTP API prepare and restore. These PRs are older but their lifecycle handling remains a useful reference. - [#44074](https://github.com/vllm-project/vllm/pull/44074) introduced a pluggable `SleepModeBackend`. [#46877](https://github.com/vllm-project/vllm/pull/46877) added communicator checkpoint prepare/restore hooks. When subsequently integrating CUDA checkpoint backends and handling NCCL and other communication resources, these common interfaces should continue to be used. - [Dynamo Snapshot](https://github.com/ai-dynamo/dynamo/tree/main/deploy/snapshot) provides a complete implementation for Kubernetes scenarios, including operator, node agent, CRIU/CUDA C/R, container environment recording, artifact storage, and restore cleanup. It serves as an important reference for external orchestration and can directly invoke vLLM's Engine Snapshot primitives in the future. ## Current Progress and Experimental Results The Phase 1 prototype has completed same-machine, single-GPU, TP=1 sleep/wake validation. Both public modes, `full` and `minimized`, have passed real CRIU/CUDA capture and restore. Two additional internal resource ablations were tested earlier for engineering comparison only and are not part of the user-facing interface. ### Test Environment - GPU: NVIDIA H20, single GPU exposed inside the container - Model: Qwen3-8B, BF16 - Topology: TP=1, PP=1, DP=1, single API server - Engine: non-eager, `max_model_len=8192`, `max_num_seqs=16`, `gpu_memory_utilization=0.50` - Software: PyTorch `2.13.0+cu130`, CUDA 13.0, NVIDIA driver/cuda-checkpoint `580.105.08`, CRIU 4.1 `20c6014e1` - Storage: model, compilation cache, CRIU images, and snapshots all on tmpfs - Snapshot config: `page_cache + optimistic`, skip SHA-256 computation for large files on restore - Sampling: 6 runs per group, first run as warmup, last 5 runs used for statistics. Times reported in the table are medians. `minimized` was run independently in three groups, totaling 15 steady-state runs. L3 restore time is measured from the `/wake_up` request to the first non-empty token received in a fixed streaming request, with final response content confirmed correct. ### Performance Results | L3 Policy | Runs | Checkpoint Time | Restore Time | Artifact Size | | --- | ---: | ---: | ---: | ---: | | `minimized` | 15 | 2.837 s | **5.948 s** | 4.251 GiB | | `full` | 5 | 27.242 s | 22.990 s | 49.659 GiB | Baseline results from the same batch: | Path | Runs | Sleep | Wake / Start | First Inference | |----|----|-----|------------|---------------| | Fresh Engine | 5 | — | 35.396 s to `/ready` | 0.054 s | | CuMem L1 | 5 | 0.340 s | 0.407 s | 0.044 s | | CuMem L2 | 5 | 0.047 s | 2.907 s | 0.049 s | ### Restore Breakdown Phase-by-phase timing for `minimized`: | Phase | Median | |-----|------| | CRIU restore | 1.861 s | | CUDA restore-TID discovery | 0.335 s | | CUDA restore action | 0.553 s | | CUDA unlock | 0.021 s | | Provider total | 2.810 s | | Worker restore | 3.147 s | | Post-restore TTFT | 0.014 s | ### Correctness and Resource Results - All 30 formal L3 runs restored successfully and passed fixed semantic checks. - Model page residency rate was 100% before every restore. - Engine `read_bytes` delta was 0 for all runs. - Across 30 `HIBERNATED` checks, project process HBM and full-GPU HBM were both 0 MiB. - Ruff and Python 3.10 mypy passed. - Snapshot tests: `75 passed`. - CLI parser tests: `3 passed, 30 deselected`. ### Feedback Period. _No response_ ### CC List. _No response_ ### Any Other Things. ## Discussion Topics 1. **Snapshot boundary.** This proposal preserves the frontend and only saves/restores EngineCore. #51360 saves the entire initialized process tree. We would like to discuss which scenarios each approach is best suited for, and the trade-offs around HTTP lifecycle, restore adoption, and persistent overhead. 2. **User interface.** Should Engine Snapshot be integrated as a new sleep level within `sleep/wake`, or exposed as a standalone `snapshot create/restore` interface? Which lifecycle and state definitions can be shared between the two entry points? 3. **Future evolution.** This proposal plans to progress through cross-machine 1→N, single-machine multi-GPU, and multi-node multi-GPU. We would like to discuss whether this roadmap aligns with the community's plans, which scenarios should be prioritized, and what additional considerations are needed for cross-machine and distributed restore. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52125", "createdAt": "2026-08-13T08:25:08Z", "updatedAt": "2026-08-13T13:26:59Z", "timestamp": "2026-08-13T13:26:59Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "RFC" ], "author": "fuhefei", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:fe6b140b846a3a0f0d4c", "signalId": "github:vllm-project/vllm:issue:52167", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52167", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Extended online quantization roadmap", "text": "### Motivation. Continuation of https://github.com/vllm-project/vllm/issues/32029. Online quantization API is now relatively stable at https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online and documented at https://docs.vllm.ai/en/stable/features/quantization/online/. This RFC summarizes and motivates proposed features that would be helpful to make online quantization to be more flexible and more widely compatible with models, including pre-quantized ones. ### Proposed Change. ### 1. Support activation quant key override Implemented in https://github.com/vllm-project/vllm/pull/51808. This is necessary to dispatch on specific linear/moe backends that can be picked only through the `activation_quant_key` (see e.g. https://github.com/vllm-project/vllm/blob/c4e969294ecab9ffefb995b734303f13f62b723f/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L441-L445) ### 2. Targeted online quantization configuration based on user patterns Implemented in https://github.com/vllm-project/vllm/pull/51285, RFC in https://github.com/vllm-project/vllm/issues/50281. The goal is simply to extend the UI to support e.g. `--quantization-config.targets '{\"re:.*o_proj.*\":\"mxfp8\"}'`, e.g. to target a single layer similar to e.g. https://github.com/vllm-project/vllm/blob/dc9f845ddc54c1df38fdbce5afe03f9fd15813bd/vllm/model_executor/layers/attention/mla_attention.py#L935-L948 (that is currently hard-coded and may be causing accuracy issues if done silently https://github.com/vllm-project/vllm/pull/48051), or e.g. `--quantization-config.targets '{\"re:.*shared_expert.*\":\"fp8_per_block\"}'` to enable fused shared expert. ### 3. Support online quantization with partially pre-quantized checkpoints Implemented in https://github.com/vllm-project/vllm/pull/51392. This is necessary to combine online quantization with already partially quantized checkpoints, which is notoriously the case recently. This is e.g. useful to test the accuracy/speed tradeoff of quantizing certain additional layers, or useful to e.g. release a single model compatible with fused shared expert / unfused shared expert. This has e.g. been requested in separate PRs: - https://github.com/vllm-project/vllm/pull/48983 - https://github.com/vllm-project/vllm/pull/49313 ### 4. Requantize already-quantized layers to different precision Not implemented yet, closest prototype is https://github.com/vllm-project/vllm/pull/48427. This is similar to https://github.com/sgl-project/sglang/pull/28291 & https://github.com/sgl-project/sglang/pull/29328 on sglang side. This idea here would be to allow loading through a certain quantization method, dequantize and override the quantization method on request. This has e.g. been requested in: - https://github.com/vllm-project/vllm/pull/48427 - https://github.com/vllm-project/vllm/pull/51274 ### Feedback Period. One month ### CC List. @BowenBao @mgoin @kylesayrs @vkuzo @maeehart @tanpinsiang @fanxingran @access2rohit ### Any Other Things. _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52167", "createdAt": "2026-08-13T13:25:24Z", "updatedAt": "2026-08-13T13:25:34Z", "timestamp": "2026-08-13T13:25:34Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "RFC", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:41e9428d00ba179b21e1", "signalId": "github:vllm-project/vllm:pull_request:52166", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52166", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Zen5 image build reuse cpu", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52166", "createdAt": "2026-08-13T13:23:34Z", "updatedAt": "2026-08-13T13:24:11Z", "timestamp": "2026-08-13T13:24:11Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "ci/build", "cpu" ], "author": "Chinmay-Kulkarni-AMD", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:9528514d21651ce78ce0", "signalId": "github:vllm-project/vllm:pull_request:52019", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52019", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Skip stage2 for single-split Triton MLA decode", "text": "## Purpose When grouped Triton MLA uses one KV split, its second-stage reduction has one input and is mathematically an identity. This PR lets stage1 write the final output and LSE directly and skips the stage2 launch. The fast path is restricted to MLA shapes for which the normal performance heuristic selects one split. Multi-split attention, MHA/GQA, and long contexts where batch invariance alone forces one split keep the existing two-stage path. The existing scratch workspace remains reserved, so this reduces HBM traffic and one launch when eligible but does not reduce reserved memory capacity. The final implementation reuses pointer, stride, and mode slots already present in the grouped kernel. This keeps the fallback Triton launch parameter list identical to the pre-patch implementation. For empty/padded rows, the direct path writes the attention-combine neutral values `output=0` and `LSE=-inf`. The old identity stage happened to produce `NaN/-inf` for those rows; they are not consumed as ordinary outputs, and the new values are safe for DCP merging. ## Duplicate-work check I searched open PRs for Triton MLA, single split, direct output, and stage2 and found no implementation of this optimization. Related work such as #33529 (split-selection/query-load tuning) and #42885 (full CUDA graph capture) is complementary; neither writes single-split MLA results directly nor skips the identity stage2 reduction. ## Test plan ```bash .venv/bin/python -m pytest -q tests/kernels/attention/test_triton_decode_attention.py .venv/bin/pre-commit run --files \\ vllm/v1/attention/ops/triton_decode_attention.py \\ vllm/v1/attention/backends/mla/triton_mla.py \\ tests/kernels/attention/test_triton_decode_attention.py git diff --check ``` The tests cover BF16/FP16 output, BF16/FP8 KV cache, active-row bitwise equivalence with the two-stage oracle, empty rows, grouped/multi-split fallback, and CUDA graph replay with changed sequence lengths. The behavior test was also run before implementation and failed because stage2 was still called once. ## Cross-platform validation The RTX PRO 6000 and H20 runs are independent, peer validations using DeepSeek-V2-Lite-Chat and `TRITON_MLA`. They use the same 15-shape kernel set: 13 shapes for which the production heuristic selects one split (`max_seq_len < 1024` on both GPUs), plus forced-single-split `B32/S2048` and `B64/S4096` controls. The tested revisions must be kept explicit: the RTX PRO 6000 was rerun on the current final source, `fa0bb68`; the H20 run evaluated the initial direct-output revision, `39b3e04`. The H20 results therefore validate the direct fast path on Hopper and identify the initial fallback-launch overhead, but they do not claim an H20 rerun of the final slot-reuse implementation. | | NVIDIA RTX PRO 6000 Blackwell Server Edition | NVIDIA H20 | | --- | --- | --- | | Architecture | sm120, 188 SMs | sm90, 78 SMs | | Driver | 580.159.03 | 535.161.08 | | PyTorch / Triton / CUDA | 2.11.0+cu130 / 3.6.0 / 13.0 | 2.11.0+cu130 / 3.6.0 / 13.0 | | Tested direct revision | **`fa0bb68` (final)** | **`39b3e04` (initial)** | | Fast-path baseline | true pre-patch two-stage source (`6accb779`) | true pre-patch kernel file matching `324f452f`, ported to the installed vLLM tree | | Microbenchmark repetition | 3 fresh processes; 12 balanced `A B C C B A` blocks per process | 12 balanced `A B C C B A` blocks | ### Result definitions - **Fast-path speedup** = `true pre-patch two-stage time / direct time`; values above 1 are faster. - **Fast-path saving** = `true pre-patch two-stage time - direct time`; positive values are faster. - **Fallback host delta** = `candidate two-stage host-enqueue time - true pre-patch host-enqueue time`; positive values are overhead. - **Fixed-work throughput change** is derived from inverse latency; it is not an online-serving throughput measurement. - RTX PRO 6000 kernel aggregates are the median of the three per-process paired medians. H20 aggregates are from its balanced three-arm run. Absolute latency is not compared across GPUs. For the three-arm microbenchmarks, `A` is true pre-patch two-stage, `B` is the tested revision with direct output disabled, and `C` is the tested revision with direct output enabled. ### Correctness | Check | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | Target kernel tests | 121 passed, 16 warnings | 121 passed | | Common 15-shape output and LSE vs true pre-patch | bitwise equal in 45/45 process-shape cases for both final arms | bitwise equal in 15/15 shapes | | Natural greedy generation, 16 prompts | identical token arrays across all three arms; 512 token IDs total, 209 distinct IDs | identical token IDs and text; 471 token IDs total, 248 distinct IDs | All applicable pre-commit hooks passed on the final source, and `git diff --check` passed. ### Kernel performance The graph region includes the same output-zeroing work used by the production path. | 13 production-eligible shapes | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | Graph speedup | **1.021x-1.152x** | **1.020x-1.144x** | | Graph saving | **1.667-2.365 us** | **1.584-3.309 us** | | Eager speedup | **1.044x-1.651x** | **1.023x-1.436x** | | Eager saving | **2.064-14.459 us** | **2.467-10.855 us** | The two control shapes were benchmarked with one split even though the production heuristic selects multiple splits: | Forced-single-split control | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | `B32/S2048`, graph / eager | 1.010x (+1.719 us) / 1.023x (+4.162 us) | 1.005x (+1.336 us) / 1.007x (+1.841 us) | | `B64/S4096`, graph / eager | 0.998x (-0.693 us) / 1.002x (+0.711 us) | 1.002x (+1.163 us) / 1.005x (+2.550 us) | For the RTX PRO 6000 `B64/S4096` graph control, per-process savings ranged from -1.262 to +0.359 us, so the small median regression is not treated as a stable architecture-level effect. The control results are not uniformly beneficial, and the production guard remains conservative rather than forcing one split for long contexts. ### End-to-end decode Both workloads used DeepSeek-V2-Lite-Chat, `TRITON_MLA`, `FULL_DECODE_ONLY`, `max_model_len=512`, batch 32, 64 forced decode tokens, 3 warmups, and 12 timed samples per process. The RTX PRO 6000 used two fresh processes per arm in `A-B-C-C-B-A` order; its estimate is the median of the two process medians and the bracketed values are their range. The H20 used an independent two-stage/direct/two-stage run. | Metric | NVIDIA RTX PRO 6000 (`fa0bb68`) | NVIDIA H20 (`39b3e04`) | | --- | ---: | ---: | | True pre-patch median estimate | 481.402 ms [479.585, 483.220] | 390.572 ms (midpoint of 389.971 and 391.172 ms) | | Final two-stage median estimate | 481.587 ms [480.752, 482.423] | not separately measured in E2E | | Direct median estimate | 476.560 ms [475.119, 478.001] | 385.849 ms | | Direct saving vs true pre-patch | **4.842 ms** | **4.723 ms** | | Latency change vs true pre-patch | **-1.006%** | **-1.209%** | | Fixed-work inverse-latency change | **+1.016%** | **+1.224%** | | Pre-patch process/arm drift | +0.758% | +0.308% | | Stage1 / stage2 launches | 1,701 / 1,701 pre-patch and final two-stage; 1,701 / 0 direct | 1,701 / 1,701 pre-patch; 1,701 / 0 direct | On the RTX PRO 6000, both mirrored halves independently favored direct output; direct versus the final-source two-stage arm was also **-1.044%**. Profiling measured target-kernel totals of 18.318 ms for true pre-patch, 18.406 ms for final two-stage, and 14.765 ms for direct output. The H20 direct samples were completely separated from its two baseline sample ranges. The forced-token arrays only establish identical fixed work because token ID 198 was the sole allowed token; natural-token correctness is reported separately above. Cross-GPU absolute latency differences are not used to infer the size or cause of the speedup. ### Fallback launch audit The initial revision added five direct-only runtime values to every stage1 launch. The H20 experiment found host-enqueue overhead on `write_direct=False`; the same issue was reproduced independently on the RTX PRO 6000. The final revision reuses existing grouped-kernel slots and restores the pre-patch runtime/constexpr signature. Values below are fallback host deltas as defined above, reported for `B32/S128`, `B32/S512`, and `B32/S2048` respectively: | Two-stage implementation | NVIDIA RTX PRO 6000 | NVIDIA H20 | | --- | ---: | ---: | | Initial `39b3e04` | **+2.681 / +2.726 / +2.821 us** | **+2.342 / +2.246 / +1.848 us** | | Final `fa0bb68` | **+0.084 / +0.208 / +0.187 us** | not rerun; no final-H20 claim | On the RTX PRO 6000, final-minus-pre-patch CUDA-event deltas for the GPU-bound S512/S2048 cases were +0.013/-0.010 us. On H20, the initial revision's two-stage graph-region deltas across the production shapes were -0.048 to +0.016 us. These measurements support a host argument-packing cause for the initial overhead, rather than a fallback-kernel GPU regression. On the PRO 6000, the final source reduces the central host deltas from about 2.7-2.8 us to at most 0.21 us. ## AI assistance OpenAI Codex assisted with code analysis, implementation, validation and benchmark scripting, and drafting this PR description. --- - [x] The purpose of the PR is described above. - [x] The test plan includes the commands and workloads used. - [x] Test, model-output, before/after performance, and fallback-regression results are included. - [x] Duplicate-work and AI-assistance disclosures are included. - [x] Documentation changes are not required because this does not change a user-facing API or supported-model surface.", "url": "https://github.com/vllm-project/vllm/pull/52019", "createdAt": "2026-08-12T16:31:12Z", "updatedAt": "2026-08-13T13:16:40Z", "timestamp": "2026-08-13T13:16:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "BabyDrangoner", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3327eb1932108c1ea020", "signalId": "github:vllm-project/vllm:pull_request:52085", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52085", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Warn when DeepSeek-V4 silently defaults reasoning_effort to \"high\"", "text": "## Purpose Fixes #52083. #50580 changed the DeepSeek-V4 chat encoding (both the Python tokenizer wrapper `vllm/tokenizers/deepseek_v4.py` and the Rust renderer `rust/src/chat/src/renderer/deepseek_v4/encoding.rs`) so that any request which ends up with thinking mode enabled but no explicit `reasoning_effort` now defaults to `reasoning_effort=\"high\"`. That happens both when a request omits `thinking`/`enable_thinking` **and** `reasoning_effort` entirely (the implicit default), and when a caller explicitly passes `thinking=True`/`enable_thinking=True` without also setting `reasoning_effort` (thinking was requested, but the effort default still silently changed). Either way it silently inserts a reasoning-effort prompt prefix that was previously omitted by default. This is a shared code path used by every DeepSeek-V4 checkpoint (Pro and Flash-0731 alike) — there's no per-checkpoint differentiation in the encoder/renderer. The change was an intentional, tested alignment with the DeepSeek-V4-Flash-0731 hosted API contract, but it silently changed the default request behavior for other DeepSeek-V4 deployments (e.g. DSV4-Pro) that relied on the old \"no reasoning-effort blurb unless requested\" default, causing a spike in truncations/reasoning length as reported in the issue. Given the encoder has no way to know which DeepSeek-V4 checkpoint variant is being served, reverting or branching the default behavior isn't a safe minimal fix — it could just as easily reintroduce the original 0731 bug #50580 fixed for other deployments. Per the issue's own suggestion (\"At the very least, we should issue a warning that the default behaviour has been changed\"), this PR adds a one-time warning in both encoders when this implicit default kicks in, telling users how to restore the previous behavior explicitly with `reasoning_effort=\"low\"`. The warning is keyed off the same `thinking_enabled`/resolved `ThinkingMode::Thinking` condition that actually selects `reasoning_effort=\"high\"`, not off \"thinking/enable_thinking were both omitted\" — an earlier version of this PR used the narrower condition and missed the explicit-`thinking=True`-without-`reasoning_effort` path, which hits the exact same silent default. ## Test Plan - Added `test_deepseek_v4_warns_when_defaulting_reasoning_effort` and `test_deepseek_v4_warns_when_thinking_explicit_without_reasoning_effort` (parametrized over `thinking=True` and `enable_thinking=True`) to assert the warning fires for both trigger paths, and `test_deepseek_v4_does_not_warn_when_explicit` (parametrized over `reasoning_effort=\"high\"`, `thinking=True` with an explicit `reasoning_effort=\"low\"`, and `thinking=False`/`enable_thinking=False`) to assert it stays silent when `reasoning_effort` is set or thinking is off. All use the existing `disable_log_dedup` fixture to observe `logger.warning_once`. - Verified the new `test_deepseek_v4_warns_when_thinking_explicit_without_reasoning_effort` test fails without the source fix: `git checkout HEAD -- vllm/tokenizers/deepseek_v4.py` (reverting to the narrower `thinking_unset`-gated condition), ran the test (failed as expected with `assert False` for both parametrizations), then restored the `thinking_enabled`-gated fix. - The Rust renderer change mirrors the same fix by checking the resolved `thinking_mode == ThinkingMode::Thinking` instead of `enable_thinking.is_none()`, gated by the same `std::sync::Once`; the existing 14 tests in `rust/src/chat/src/renderer/deepseek_v4/tests.rs` (including `omitted_thinking_and_effort_default_to_high`) continue to pass unchanged, confirming rendered output is untouched. No dedicated Rust test was added for the warning itself since this crate has no existing pattern for capturing `tracing` output in tests, and adding that infra seemed out of proportion for a log-only change. ### Commands run ``` .venv/bin/python -m pytest tests/tokenizers_/test_deepseek_v4.py -v # 33 passed cargo test -p vllm-chat renderer::deepseek_v4 # 14 passed pre-commit run --files vllm/tokenizers/deepseek_v4.py tests/tokenizers_/test_deepseek_v4.py rust/src/chat/src/renderer/deepseek_v4/encoding.rs # ruff-check, ruff-format, mypy, rust-cargo-fmt, etc. all passed ``` ## Duplicate Check Searched for open PRs referencing #52083 or similar titles — none found. #50684 (\"Fix DeepSeek-V4 reasoning_effort \\\"high\\\" tier and message-level tools preservation\") addresses a related but distinct bug (the \"high\" tier text was mislabeled/missing, and message-level `tools` were dropped for `role == \"system\"`) in the Python encoder only; it does not touch the default-value regression described in #52083, and doesn't touch the Rust renderer. ## AI Assistance AI assistance (Claude) was used to bisect the regression to #50580, compare the Python tokenizer wrapper and Rust renderer default-value logic, implement the warning in both, and write/run the tests above. All changes were reviewed and the listed commands were run and their output verified by the submitting human.", "url": "https://github.com/vllm-project/vllm/pull/52085", "createdAt": "2026-08-13T03:16:29Z", "updatedAt": "2026-08-13T13:15:58Z", "timestamp": "2026-08-13T13:15:58Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "deepseek", "rust" ], "author": "chelsealong", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:bf3840ec1c563ccd88bb", "signalId": "github:vllm-project/vllm:issue:50690", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50690", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: gpt-oss chat completions return 500 \"Unexpected token 200002 while expecting start token 200006\" when ignore_eos=true", "text": "### Your current environment - vLLM **0.26.0** and **0.18.0** (both reproduce), official `vllm/vllm-openai` images - `openai-harmony` 0.0.8 - Model: `openai/gpt-oss-120b`, 1x H200, TP=1 - `vllm serve openai/gpt-oss-120b --max-model-len 32768 --tensor-parallel-size 1` ### 🐛 Describe the bug A Chat Completions request with `ignore_eos: true` returns **HTTP 500** whenever the model finishes its Harmony turn before `max_completion_tokens` is reached: ```json {\"error\":{\"message\":\"Unexpected token 200002 while expecting start token 200006\",\"type\":\"InternalServerError\",\"param\":null,\"code\":500}} ``` (200002 = `<|return|>`, 200006 = `<|start|>`) **Minimal reproducer** — one request, no streaming: ```bash curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\": \"openai/gpt-oss-120b\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hi.\"}], \"max_completion_tokens\": 512, \"ignore_eos\": true }' ``` The prompt matters: the answer must be **shorter** than `max_completion_tokens`. A prompt whose answer fills the budget never reaches the terminator and succeeds, which makes this look intermittent. | arm | 0.18.0 | 0.26.0 | |---|---|---| | short answer + `ignore_eos` | 500 | 500 | | answer longer than `max_completion_tokens` + `ignore_eos` | OK | OK | | short answer, no `ignore_eos` | OK | OK | ### Why `<|return|>` / `<|call|>` are the Harmony assistant stop tokens. `ignore_eos=true` suppresses stopping on them (`SamplingParams.update_from_generation_config`, the `if not self.ignore_eos:` guards), so generation continues past the end of the turn. Those trailing tokens are then fed to the `StreamableParser`, which is a state machine: after `<|end|>` it expects `<|start|>`, gets `<|return|>`, and raises. The exception is not caught, so the whole request fails. The parser half reproduces **without a server or GPU**: ```python from openai_harmony import HarmonyEncodingName, Role, StreamableParser, load_harmony_encoding enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS) e = lambda s: enc.encode(s, allowed_special=\"all\") stream = e(\"<|channel|>\") + e(\"final\") + e(\"<|message|>\") + e(\"answer\") + e(\"<|end|>\") + e(\"<|return|>\") p = StreamableParser(enc, role=Role.ASSISTANT) for t in stream: p.process(t) # HarmonyError: Unexpected token 200002 while expecting start token 200006 ``` vLLM is on both sides of this: it permits `ignore_eos` to override its own Harmony stop tokens, then treats the resulting token stream as a parse error. ### Impact `ignore_eos` is how benchmark tools pin output length so ISL/OSL combinations are controlled — both `vllm bench serve` (`--ignore-eos`) and GuideLLM use it. It is therefore hard to benchmark gpt-oss over Chat Completions at all. Measured with GuideLLM against 0.18.0 at ISL/OSL 1024/1024: **65% of requests failed** (971 errors, all this message). At OSL 128 the error rate was 0%, because the model rarely finishes a turn that early — so the failure rate scales with output length and can easily be mistaken for server saturation. `/v1/completions` is unaffected (no Harmony parsing on that path). ### Related - #41731 \"[Bugfix] Respect ignore_eos for GPT-OSS Harmony stop tokens\" — describes this exact cause, closed without merging (went stale on merge conflicts) - #41950 \"[Bugfix] Stop Harmony stream parsing after parser errors\" — the parser-side guard, closed without merging - #44009 \"[Frontend] Clean up stop_token_ids override for Harmony\" — merged, removed the `default_sampling_params` stop-token injection, but **does not fix this**: 0.26.0 still returns 500 (table above) - #22519 — same parser error via the tool-call path Either half would fix it: honor the Harmony stop tokens even under `ignore_eos` (#41731), or stop feeding a failed parser and fall back to raw tokens (#41950). ### Before submitting a new issue... - [x] Made sure I searched the documentation and existing issues.", "url": "https://github.com/vllm-project/vllm/issues/50690", "createdAt": "2026-08-01T15:05:45Z", "updatedAt": "2026-08-13T13:15:58Z", "timestamp": "2026-08-13T13:15:58Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "rebel-joonhyunglee", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:64ebc3e92cb17218d6ef", "signalId": "github:vllm-project/vllm:issue:52083", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52083", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Breaking change to DeepSeek-V4-Pro default reasoning effort", "text": "### Your current environment vllm main, bisected to https://github.com/vllm-project/vllm/pull/50580 ### 🐛 Describe the bug This PR (https://github.com/vllm-project/vllm/pull/50580), which changes the DSV4 encoder for DSV4-Flash-0731 support, changes the default behaviour for DSV4-Pro. Previously, only reasoning_effort == \"max\" adds the reasoning blurb. See here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-DSpark/blob/main/encoding/encoding_dsv4.py#L262 However, the new default behaviour is for reasoning effort \"high\" to set that blurb, and introduces a newer blurb for \"max\" thinking mode. This means that DSV4-Pro deployments will now use (what used to be) maximum reasoning effort by default. This caused a massive spike in truncations and a drop in AL in my serving setup. At the very least, we should issue a warning that the default behaviour has been changed. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52083", "createdAt": "2026-08-13T02:51:04Z", "updatedAt": "2026-08-13T13:15:25Z", "timestamp": "2026-08-13T13:15:25Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "benchislett", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:8329121279496b3d51e9", "signalId": "github:vllm-project/vllm:pull_request:52157", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52157", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][Spec Decode] Support varlen trtllm-gen decode for adaptive verification", "text": "## Purpose - Resolves #51871: adaptive verification needs decode batches with per-request query lengths - Wire FlashInfer trtllm-gen decode to `cum_seq_lens_q`/`max_q_len`; `q_len_per_req` must be `None` for varlen - Flip `supports_device_cpu_query_lens_mismatch` on SM100: device `qo_indptr` is the source of truth, CPU lengths only an upper bound - Guard: adaptive verification without the trtllm-gen kernel now fails fast at init - Not a duplicate: no open PR touches trtllm-gen varlen decode ## Test Plan - `pytest tests/kernels/attention/test_flashinfer_trtllm_attention.py -k varlen` (B200) - Uniform decode regression subset on the same file - E2E: Gemma4-12B NVFP4 + DSpark (q=8) through the varlen path, GSM8K 50q - Kernel perf: varlen vs uniform-padded, profiler device time (B200) ## Test Result - 4 varlen + 16 uniform regression passed, 0 failed - E2E serves correctly: GSM8K 98%, 0% invalid, 17.6 q/s | shape | rows ragged/padded | varlen | padded | |---|---|---|---| | batch=8, q~U[1,8] | 38/64 | 28.0us | 22.2us | | batch=32, q~U[1,8] | 118/256 | 70.5us | 65.4us | | batch=32, q~U[1,4] | 74/128 | 61.4us | 54.2us | - Varlen carries a small fixed kernel overhead; it enables ragged batches rather than speeding uniform ones - Adaptive E2E is blocked on a public Gemma4 drafter with a confidence head; kernel tests cover the device-CPU mismatch semantics ---", "url": "https://github.com/vllm-project/vllm/pull/52157", "createdAt": "2026-08-13T11:12:00Z", "updatedAt": "2026-08-13T13:14:51Z", "timestamp": "2026-08-13T13:14:51Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "nvidia" ], "author": "guan404ming", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0e5b3d2e554271d30bc9", "signalId": "github:vllm-project/vllm:pull_request:52165", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52165", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc][Spec Decode] Detect DeepSeek-V4 DSpark checkpoints from config", "text": "## Purpose To close https://github.com/vllm-project/vllm/issues/52111 ## Test Plan Add a test case `tests/config/test_deepseek_v4_dspark_config.py` (TODO) ## Test Result Coming soon. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52165", "createdAt": "2026-08-13T13:13:35Z", "updatedAt": "2026-08-13T13:14:23Z", "timestamp": "2026-08-13T13:14:23Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "deepseek" ], "author": "xuebwang-amd", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:c469bbcdac5e61d07178", "signalId": "github:vllm-project/vllm:pull_request:50876", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50876", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual silu quant fusion", "text": "Adding more quant types to SiLU+quant activation. Made with the assistance of Claude, all generated code has been human-reviewed. ### Testing: #### Unit test ``` pytest tests/kernels/test_fused_quant_activation.py ``` #### E2E test example (fp8, static) - manually expect if outputs are sane ``` from vllm import LLM, SamplingParams # Sample prompts. prompts = [ \"Hello, my name is\", \"The president of the United States is\", \"The capital of France is\", \"The future of AI is\", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95, min_tokens=200, max_tokens=200) llm = LLM(model='neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8') outputs = llm.generate(prompts, sampling_params) # Print the outputs. for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f\"Prompt: {prompt!r}, Generated text: {generated_text!r}\") ``` #### Eval example (fp8, static) ``` lm_eval --model vllm \\ --model_args pretrained=neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \\ --tasks hellaswag,arc_easy \\ --batch_size auto ``` ### Eval results: (gsm8k with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8`) main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7710|± |0.0116| | | |strict-match | 5|exact_match|↑ |0.6922|± |0.0127| ``` this PR + https://github.com/vllm-project/vllm/pull/46864 ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.7582|± |0.0118| | | |strict-match | 5|exact_match|↑ |0.6816|± |0.0128| ```", "url": "https://github.com/vllm-project/vllm/pull/50876", "createdAt": "2026-08-03T13:50:48Z", "updatedAt": "2026-08-13T13:12:54Z", "timestamp": "2026-08-13T13:12:54Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "rust", "kimi", "k3" ], "author": "ElizaWszola", "state": "closed", "assignees": [ "sangstar" ], "change": "new" } }, { "id": "event:d3513a6514df1da054e0", "signalId": "github:vllm-project/vllm:issue:51964", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51964", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]:amd mi308x gpu, vllm 0.27.0~0.27.1, rocm 7.2.3, Kimi-K2.7-Coder start fails:AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942", "text": "### Your current environment
The output of python collect_env.py ```text (vllm) [fanlf@2025-h3c-83 tmp]$ python ./collect_env.py Collecting environment information... ============================== System Info ============================== OS : Rocky Linux 10.2 (Red Quartz) (x86_64) GCC version : (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4) Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+gitd0c8b1f Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Jul 10 2026, 00:00:00) [GCC 14.3.1 20251022 (Red Hat 14.3.1-4)] (64-bit runtime) Python platform : Linux-6.12.0-124.29.1.el10_1.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : AMD Instinct MI308X (gfx942:sramecc+:xnack-) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU(s) scaling MHz: 66% CPU max MHz: 3709.3569 CPU min MHz: 1500.0000 BogoMIPS: 4793.03 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap Virtualization: AMD-V L1d cache: 6 MiB (192 instances) L1i cache: 6 MiB (192 instances) L2 cache: 192 MiB (192 instances) L3 cache: 768 MiB (24 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-95,192-287 NUMA node1 CPU(s): 96-191,288-383 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.3.5 [pip3] onnx==1.22.0 [pip3] onnx-ir==0.2.1 [pip3] onnxscript==0.7.1 [pip3] onnxslim==0.1.95 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0+gitd0c8b1f [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.9.0+eaa9e4e [pip3] torchvision==0.24.1+d801a34 [pip3] transformers==5.15.0 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : 7.2.53211-c2d9476115 vLLM Version : 0.27.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 15 15 15 15 15 15 15 GPU1 15 0 15 15 15 15 15 15 GPU2 15 15 0 15 15 15 15 15 GPU3 15 15 15 0 15 15 15 15 GPU4 15 15 15 15 0 15 15 15 GPU5 15 15 15 15 15 0 15 15 GPU6 15 15 15 15 15 15 0 15 GPU7 15 15 15 15 15 15 15 0 ================================= Hops between two GPUs ================================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 1 1 1 1 1 1 1 GPU1 1 0 1 1 1 1 1 1 GPU2 1 1 0 1 1 1 1 1 GPU3 1 1 1 0 1 1 1 1 GPU4 1 1 1 1 0 1 1 1 GPU5 1 1 1 1 1 0 1 1 GPU6 1 1 1 1 1 1 0 1 GPU7 1 1 1 1 1 1 1 0 =============================== Link Type between two GPUs =============================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 XGMI XGMI XGMI XGMI XGMI XGMI XGMI GPU1 XGMI 0 XGMI XGMI XGMI XGMI XGMI XGMI GPU2 XGMI XGMI 0 XGMI XGMI XGMI XGMI XGMI GPU3 XGMI XGMI XGMI 0 XGMI XGMI XGMI XGMI GPU4 XGMI XGMI XGMI XGMI 0 XGMI XGMI XGMI GPU5 XGMI XGMI XGMI XGMI XGMI 0 XGMI XGMI GPU6 XGMI XGMI XGMI XGMI XGMI XGMI 0 XGMI GPU7 XGMI XGMI XGMI XGMI XGMI XGMI XGMI 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: 0 GPU[1] : (Topology) Numa Node: 0 GPU[1] : (Topology) Numa Affinity: 0 GPU[2] : (Topology) Numa Node: 0 GPU[2] : (Topology) Numa Affinity: 0 GPU[3] : (Topology) Numa Node: 0 GPU[3] : (Topology) Numa Affinity: 0 GPU[4] : (Topology) Numa Node: 1 GPU[4] : (Topology) Numa Affinity: 1 GPU[5] : (Topology) Numa Node: 1 GPU[5] : (Topology) Numa Affinity: 1 GPU[6] : (Topology) Numa Node: 1 GPU[6] : (Topology) Numa Affinity: 1 GPU[7] : (Topology) Numa Node: 1 GPU[7] : (Topology) Numa Affinity: 1 ================================== End of ROCm SMI Log =================================== ============================== Environment Variables ============================== VLLM_ROCM_USE_AITER=1 PYTORCH_ROCM_ARCH=gfx942 TORCH_BLAS_PREFER_HIPBLASLT=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_fanlf ```
### 🐛 Describe the bug 1. install： uv pip install vllm==\"0.27.0+rocm723\" --extra-index-url https://wheels.vllm.ai/rocm --no-cache 2. startup and error log： ``` log (vllm) [fanlf@2025-h3c-83 vllm]$ export AITER_ROCM_ARCH=\"gfx942\" export PYTORCH_ROCM_ARCH=\"gfx942\" export MORI_GPU_ARCHS=\"gfx942\" export VLLM_ROCM_USE_AITER=1 export TORCH_BLAS_PREFER_HIPBLASLT=1 export HIP_FORCE_DEV_KERNARG=1 (vllm) [fanlf@2025-h3c-83 vllm]$ vllm serve \\ /pool/models/moonshotai/Kimi-K2.7 \\ --served-model-name kimi \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --mm-encoder-tp-mode data \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.6 \\ --port 8003 \\ --host 0.0.0.0 \\ --tool-call-parser kimi_k2 \\ --enable-auto-tool-choice \\ --reasoning-parser kimi_k2 \\ --kv-cache-dtype fp8 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █ █ █▄ ▄█ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.0 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █▄█▀ █ █ █ █ model /pool/models/moonshotai/Kimi-K2.7 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:273] non-default args: {'model_tag': '/pool/models/moonshotai/Kimi-K2.7', 'enable_auto_tool_choice': True, 'tool_call_parser': 'kimi_k2', 'host': '0.0.0.0', 'port': 8003, 'model': '/pool/models/moonshotai/Kimi-K2.7', 'trust_remote_code': True, 'max_model_len': 262144, 'served_model_name': ['kimi'], 'reasoning_parser': 'kimi_k2', 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'mm_encoder_tp_mode': 'data'} ...... (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] WorkerProc hit an exception. (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] [vllm-rocm.md.txt](https://github.com/user-attachments/files/30973739/vllm-rocm.md.txt) ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51964", "createdAt": "2026-08-12T08:49:47Z", "updatedAt": "2026-08-13T13:12:53Z", "timestamp": "2026-08-13T13:12:53Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "rocm", "kimi" ], "author": "badrecover", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:523310c63adac3192c58", "signalId": "github:vllm-project/vllm:issue:51181", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51181", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeekV4 DSpark CUDA graph capture failure on H100", "text": "### Your current environment
The output of python collect_env.py ```textCollecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.14.0-284.118.1.el9_2.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H100 80GB HBM3 GPU 1: NVIDIA H100 80GB HBM3 GPU 2: NVIDIA H100 80GB HBM3 GPU 3: NVIDIA H100 80GB HBM3 GPU 4: NVIDIA H100 80GB HBM3 GPU 5: NVIDIA H100 80GB HBM3 GPU 6: NVIDIA H100 80GB HBM3 GPU 7: NVIDIA H100 80GB HBM3 Nvidia driver version : 580.173.02 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6430 CPU family: 6 Model: 143 Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 76% CPU max MHz: 3400.0000 CPU min MHz: 800.0000 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities L1d cache: 3 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 128 MiB (64 instances) L3 cache: 120 MiB (2 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 NUMA node2 CPU(s): 16-23 NUMA node3 CPU(s): 24-31 NUMA node4 CPU(s): 32-39 NUMA node5 CPU(s): 40-47 NUMA node6 CPU(s): 48-55 NUMA node7 CPU(s): 56-63 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.15.post1 [pip3] helion==1.1.0 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.26.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.14.1 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.1rc1.dev385+g7fd4af585 (git sha: 7fd4af585) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX SYS SYS SYS SYS 0-7 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PXB SYS SYS SYS SYS 0-7 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS PXB NODE SYS SYS 16-23 2 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS PIX NODE SYS SYS 16-23 2 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS PXB SYS 32-39 4 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS PIX SYS 32-39 4 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS PXB 48-55 6 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS PIX 48-55 6 N/A NIC0 PIX PXB SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS NIC1 SYS SYS PXB PIX SYS SYS SYS SYS SYS X NODE SYS SYS NIC2 SYS SYS NODE NODE SYS SYS SYS SYS SYS NODE X SYS SYS NIC3 SYS SYS SYS SYS PXB PIX SYS SYS SYS SYS SYS X SYS NIC4 SYS SYS SYS SYS SYS SYS PXB PIX SYS SYS SYS SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=/var/run/nvidia-container-devices CUDA_COREDUMP_SHOW_PROGRESS=1 CUDA_COREDUMP_GENERATION_FLAG=skip_nonrelocated_elf_images,skip_global_memory,skip_shared_memory,skip_local_memory,skip_constbank_memory VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=1024 LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_NO_USAGE_STATS=1 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=1024 VLLM_ENFORCE_STRICT_TOOL_CALLING=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root VLLM_WORKER_MULTIPROC_METHOD=spawn ```
### 🐛 Describe the bug While launching DeepSeek V4 0731 with DSpark speculative decoding on, the server crashes on the CUDA graph capture stage. Exact reproduction requires single H100 * 8 machine. # Launch script ```shell vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --kv-cache-dtype fp8 \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --all2all-backend deepep_v2 --max-num-seqs 32 \\ --tokenizer-mode deepseek_v4 \\ --speculative-config.method dspark \\ --speculative-config.num_speculative_tokens 5 \\ --reasoning-parser deepseek_v4 \\ --enable-auto-tool-choice \\ --tool-call-parser deepseek_v4 ``` # Error traceback ```log ... Capturing CUDA graphs (PIECEWISE): 65%|███████ | 28/43 [01:16<00:20, 1.38s/it] [19:25:48.403492] coredump: Starting GPU coredump generation ... ... ... [19:25:48.404432] coredump: Detected an exception of type CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS (5) [19:25:48.404435] coredump: - Device: 4 [19:25:48.404437] coredump: - SM: 124 [19:25:48.404439] coredump: - Warp: 1 [19:25:48.404440] coredump: - PC 0x7ef928bc46b0 [19:25:48.404688] coredump: Stack trace (lane masks: active 0xFFFFFFFF, valid 0xFFFFFFFF): [19:25:48.404692] coredump: # 0 0x7ef928bc46b0 _ZN9deep_gemm5sched30sm90_paged_mqa_logits_metadataILj96ELj256ELj132ELb0EEEvjjbPKjS3_Pj ... ``` # Root Cause Analysis As printed in coredump, the DeepGEMM version built in latest main has an OOB bug. A fix is submitted on deepseek-ai/DeepGEMM#399 , but this bug is really easy to hit in practice - the dummy inputs used for CUDA graph capture is pretty small so that the OOB edge condition (`q_idx == batch_size`) is easily met. I'd rather suggest pulling the PR into vllm built-in DeepGEMM instead of waiting for the review upstream. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51181", "createdAt": "2026-08-05T16:33:30Z", "updatedAt": "2026-08-13T13:12:44Z", "timestamp": "2026-08-13T13:12:44Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug" ], "author": "cjackal", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:a8ee642cd76e5020323a", "signalId": "github:vllm-project/vllm:issue:51912", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51912", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Allow logging model output text without output token IDs", "text": "### 🚀 The feature, motivation and pitch In vLLM v0.27.1, `--enable-log-outputs` logs the generated text, `output_token_ids`, and finish reason together in the same INFO record: ```text Generated response : output: '...', output_token_ids: [...], finish_reason: stop ``` For production troubleshooting, I want to keep the human-readable generated output and finish reason while omitting output token IDs. Long token ID lists add substantial log volume, and token IDs are not anonymized data because they can be decoded back into text. I propose adding a BooleanOptionalAction-style frontend option: ```text --enable-log-output-token-ids --no-enable-log-output-token-ids ``` Suggested behavior: - Preserve current behavior by default (`enable_log_output_token_ids=True`). - The option is relevant only when `--enable-log-outputs` is enabled. - With `--no-enable-log-output-token-ids`, omit the entire `output_token_ids` field from output log records rather than logging `output_token_ids: None`. - Apply the behavior consistently to streaming and non-streaming output logging across supported serving endpoints. - When token ID logging remains enabled, continue applying `--max-log-len` as today. Prompt logging at INFO is intentionally outside this issue's scope because it is already covered by #42051 and #38583. The broader request/output logging use case is also discussed in #40155. ### Alternatives - Removing `--enable-log-outputs` also removes the generated text needed for troubleshooting. - `--max-log-len` truncates both the desired output text and token IDs, and it does not eliminate per-delta streaming log records. - A custom `--log-config-file` can change logger levels, but standard Python `dictConfig` cannot remove only `output_token_ids` because it is embedded in the same log message and argument tuple as the output and finish reason. Doing so requires a custom Python filter/formatter. - Filtering the field in an external logging pipeline is possible but brittle and makes the textual log format an implicit parsing contract. ### Additional context Relevant v0.27.1 implementation: - `RequestLogger.log_outputs`: https://github.com/vllm-project/vllm/blob/v0.27.1/vllm/entrypoints/serve/utils/request_logger.py#L70-L98 - Chat completion output logging: https://github.com/vllm-project/vllm/blob/v0.27.1/vllm/entrypoints/openai/chat_completion/serving.py#L1106-L1136 This request is based on an observed v0.27.1 deployment using `--enable-log-requests --enable-log-outputs`. ### Before submitting a new issue... - [x] I searched existing and past issues, open pull requests, and the official documentation for an existing output-token-ID logging control.", "url": "https://github.com/vllm-project/vllm/issues/51912", "createdAt": "2026-08-12T02:28:33Z", "updatedAt": "2026-08-13T13:11:53Z", "timestamp": "2026-08-13T13:11:53Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "ruanwenjun", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:9754ab1f66a801e3fa10", "signalId": "github:vllm-project/vllm:pull_request:52098", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52098", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Log output token IDs at DEBUG level", "text": "## Purpose Allow operators to keep human-readable generated output logs without emitting output token IDs at the default INFO level. Following maintainer feedback, this now mirrors the existing request-input logging split instead of adding a new CLI flag: - INFO keeps generated text and the finish reason. - DEBUG additionally logs output token IDs. - `--max-log-len` continues to truncate both output text and token IDs at their respective levels. - Streaming delta, streaming-complete, and non-streaming output logs share the same behavior. Fixes #51912. Duplicate check: I rechecked the issue discussion and open PRs by issue number and output-token-ID logging keywords. No other open PR implements this behavior. No model behavior or evaluation output changes. Model evals are not applicable because this only changes request log formatting and CLI help text. AI assistance disclosure: OpenAI Codex assisted with implementation, review-feedback handling, and test execution. The human submitter must review every changed line, understand the end-to-end change, and take responsibility for the PR before merge. ## Test Plan - Execute the existing `tests/entrypoints/serve/utils/test_request_logger.py` unit tests, covering INFO/DEBUG separation, streaming and non-streaming logs, truncation, and edge cases. - Run Ruff check and format validation on all three net-changed files. - Run the repository Python 3.10 mypy hook on all three net-changed files. - Run `validate-config`, applicable commit hooks, and `git diff --check`. ## Test Result - Request logger unit tests: 9 passed in 0.12s. The local Windows environment did not contain the full vLLM runtime/test dependency set, so the existing test file was run with `--noconftest` and minimal import stubs; the real `RequestLogger` implementation and all tests in that file were executed. - Ruff check and format: passed. - Python 3.10 mypy on the three net-changed files: passed. - `validate-config`: passed with UTF-8 console mode. - Applicable commit hooks passed, including typos, SPDX, forbidden imports, configuration validation, and DCO sign-off. - `git diff --check`: passed. - `actionlint` and `update-dockerfile-graph` were skipped because no workflow or Dockerfile changed. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test commands. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] Documentation impact considered. CLI help text now describes the INFO/DEBUG split; no separate documentation page is needed.
", "url": "https://github.com/vllm-project/vllm/pull/52098", "createdAt": "2026-08-13T04:51:35Z", "updatedAt": "2026-08-13T13:11:52Z", "timestamp": "2026-08-13T13:11:52Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "frontend", "ready" ], "author": "ruirui6946", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e6db5b0b022b05a491d7", "signalId": "github:vllm-project/vllm:pull_request:37925", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:37925", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] CUDA Checkpoint/Restore — Phase 2: Engine/Executor/API Integration", "text": "## Summary Phase 2 of CUDA checkpoint/restore (RFC #34303). Wires `suspend`/`resume` through the full engine stack and exposes REST endpoints. **Depends on:** #37921 (Phase 1: C extension, Python wrapper, worker methods) ### Changes - **Executor** (`vllm/v1/executor/abstract.py`): `suspend()`/`resume()` via `collective_rpc`, `is_suspended` state tracking - **Engine Core** (`vllm/v1/engine/core.py`): `suspend(mode)`/`resume()`/`is_checkpoint_suspended()` with scheduler pausing - **Protocol** (`vllm/engine/protocol.py`): Abstract methods for checkpoint operations - **AsyncLLM** (`vllm/v1/engine/async_llm.py`): Async implementations - **Core Client** (`vllm/v1/engine/core_client.py`): Methods in all 4 client classes (base, InprocClient, SyncMPClient, AsyncMPClient) - **REST API** (`vllm/entrypoints/serve/checkpoint/`): `POST /suspend`, `POST /resume`, `GET /is_suspended` (dev mode only) - **Router registration** (`vllm/entrypoints/serve/__init__.py`) ### Architecture Follows the exact same wiring pattern as existing `sleep`/`wake_up`/`is_sleeping`: ``` API Router → AsyncLLM → CoreClient → EngineCore → Executor → Worker ``` ## Test plan - REST endpoints gated behind `VLLM_SERVER_DEV_MODE` (same as sleep) - Integration tests to be added in follow-up - `ruff check` and `mypy` pass on all changed files ## Notes - AI assistance was used (Claude). All changes reviewed by human submitter. - Not duplicating any existing PR — this is new functionality per RFC #34303. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/37925", "createdAt": "2026-03-23T20:24:34Z", "updatedAt": "2026-08-13T13:11:47Z", "timestamp": "2026-08-13T13:11:47Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "frontend", "v1", "nvidia" ], "author": "elizabetht", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:520a76f6f42891ca0861", "signalId": "github:vllm-project/vllm:pull_request:51415", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51415", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual act quant fusion llama", "text": "Starts the `ActivationQuantFusionPass` manual-fusion migration (RFC #43224, specific tracker https://github.com/vllm-project/vllm/issues/43501) on the producer side of the `QuantizedActivation` contract (#44260). Adds `maybe_fused_act_quant`: given an activation and the linear it feeds, it emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kernel when the linear advertises a consumable `input_quant_key`, and falls back to the plain activation otherwise. Llama's MLP forward calls it for `down_proj` — the model-code change is one line. When manual fusion fires the `silu_and_mul` pattern is already consumed, so the compiler `ActivationQuantFusionPass` finds nothing and the two never double-fuse. Only `(SiluAndMul, kFp8StaticTensorSym)` is registered for now; other schemes/activations are one table row each. Not a duplicate: builds on the landed contract (#44260) and is the activation-quant counterpart to the AR+RMSNorm manual fusion in #45855; no open PR covers the producer side. ## Test Smoke tested on `RedHatAI/Llama-3.2-1B-Instruct-FP8` (default compiled + CUDA-graph path): all 16 `mlp.down_proj` layers take the fused branch (`input_quant_key == kFp8StaticTensorSym`) and generations are coherent — the `QuantizedActivation` flows through the compiled `down_proj.forward` into `apply_weights` with no graph break. TP2 coherence and an A/B vs. the compiler pass are follow-ups. AI assistance (Claude) was used; all changed lines are human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/51415", "createdAt": "2026-08-07T15:44:04Z", "updatedAt": "2026-08-13T13:10:59Z", "timestamp": "2026-08-13T13:10:59Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "llama", "quantization" ], "author": "mgoin", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:a6378a3af2a0d764fbad", "signalId": "github:vllm-project/vllm:issue:51303", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51303", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Tracking][Spec Decode] Adaptive DSpark Bring-Up Tracker", "text": "Tracking issue to organize the adaptive verification support for DSpark speculative decoding ## Backend Support Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, the following backends are known to support variable lengths: - FLASH_ATTN - DSV4 Attention **TODO**: - Hybrid / Linear attention (requires significant effort) - - Mamba (for NVIDIA Nemotron models) - - GDN (for Qwen3.x models) - - Kimi KDA (for Kimi K3) - DSV3-style MLA - FlashInfer GQA: - - TRTLLM kernel support exists, just needs to be plugged into vLLM. Minor change. - - CUTLASS backend of FlashInfer may need additional support to avoid the cpu query lengths ## Online Profiling The initial version does profiling at startup to estimate the verification step time by replaying the recorded CUDA graphs. This is known to have some drift: \"Image\" We want to extend this to fully online profiling in the future. However, there are some challenges that make this tricky: - Online profiling may lead to \"dead spots\": one-time latency spikes make a certain batch size appear unattractive, leading to the adaptive verifier to never select that batch size, which leads to negative feedback and stale data limiting our range of profiled values. Generally there is an \"exploration vs exploitation\" problem here, as well as a requirement for some way to detect and eliminate outliers (such as those caused by JIT compilation) - Variable sequence lengths make it hard to measure a single reliable estimate of execution time for a given number of tokens. In practice we will want some way to account for this: either tracking measured points in 2D (num_tokens, max_seq_len or mean_seq_len), or estimating the overhead added per-token of KV reads ## Tuning Confidence Scores The confidence scores are not always perfectly calibrated. In particular, it seems like the deeper positions have higher variance and some bias in their expectation. The DSpark paper suggests [Sequential Temperature Scaling (STS)](https://arxiv.org/html/2607.05147v1) as an online solution to calibrating these estimates. I did not see much benefit when adding this feature, but we should experiment more thoroughly and assess the potential benefit of improved confidence estimates. Here are the results from one run with DSV4-Flash-0731: \"Image\" ## First-Class Observability ### Exposing Timings and Adaptive-SD Metrics Now that we are profiling different phases of the worker execution (model runner, proposer), and tracking the step time by batch size, we will want to expose these to the user for observability. This will help with performance tuning and development, as well as for monitoring of production deployments at scale. We will want to propagate the metrics to both the prometheus / frontend, as well as to the iteration-level logging feature. We will also want some way to report our other metrics for adaptive verification: confidence success rate (mismatch between \"expected\" AL and \"achieved\" AL), performance model reliability (difference between \"expected\" step time and \"actual\" step time), etc. ### Benchmarking Compatibility Adaptive speculative decoding will change what it means for speculative decoding to have high acceptance rate. AL is no longer a \"golden\" metric for success. As a start, this PR enables detailed, per-request and per-step speculative decoding metrics to be exposed in the OpenAI response. This can be used to report exact speculative decoding statistics and aggregated on the client to fit the use-case. https://github.com/vllm-project/vllm/pull/48915 ## Feature Compatibility - [ ] Logprob extraction support ([see here](https://github.com/vllm-project/vllm/pull/47808#discussion_r3723768178))", "url": "https://github.com/vllm-project/vllm/issues/51303", "createdAt": "2026-08-06T19:27:44Z", "updatedAt": "2026-08-13T13:07:09Z", "timestamp": "2026-08-13T13:07:09Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "speculative-decoding", "DSv4", "dflash", "kimi", "k3" ], "author": "benchislett", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:bce7eb30bbce1994762e", "signalId": "github:vllm-project/vllm:pull_request:49544", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] gfx942: use FlyDSL fp8 MQA logits kernel (ROCm/aiter#3913)", "text": "On gfx942, replace the vendored Triton `fp8_mqa_logits` with `aiter.ops.flydsl.flydsl_fp8_mqa_logits` from ROCm/aiter#3913. Drop-in replacement (identical args/semantics), gated behind `_ON_GFX942`. gfx950 and other paths are untouched. ## Results: GLM-5.2-FP8, 8× MI325X TP8 ### ISL=128K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 42,839 | 21,871 | **−49%** | | Median TPOT (ms) | 109.11 | 68.76 | **−37%** | | Output tok/s | 52.44 | 85.34 | **+63%** | ### ISL=8K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 918 | 863 | −6% | | Median TPOT (ms) | 19.79 | 19.68 | −0.6% | | Output tok/s | 382.2 | 386.3 | +1.1% | No regression at short context; gains scale with input length as seen above ## Accuracy | Test | Filter | Result | |------|--------|--------| | GSM8K (5-shot, full 1319 samples) | flexible-extract | 0.9416 ± 0.0065 | | | strict-match | 0.9424 ± 0.0064 | | NIAH (128K tokens, 10 needles, 10 trials) | — | 8/10 perfect, 2/10 miss one needle |
Serve command 
export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export HSA_NO_SCRATCH_RECLAIM=1 vllm serve zai-org/GLM-5.2-FP8 --block-size 1 --compilation-config '{\"pass_config\": {\"fuse_norm_quant\": true, \"fuse_act_quant\": true}, \"custom_ops\": [\"+rms_norm\", \"+silu_and_mul\"]}' --disable-log-stats --enable-auto-tool-choice --no-enable-expert-parallel --gpu-memory-utilization 0.80 --hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": 8}' --host 0.0.0.0 --kv-cache-dtype fp8 --max-model-len 263168 --max-num-batched-tokens 16384 --max-num-seqs 128 --no-enable-prefix-caching --port 8000 --reasoning-parser glm45 --tensor-parallel-size 8 --tool-call-parser glm47 --trust-remote-code
Bench command 
vllm bench serve --backend vllm --model zai-org/GLM-5.2-FP8 --host localhost --port 8000 --dataset-name random --random-input-len 131072 --random-output-len 1024 --max-concurrency 8 --num-prompts 20 --num-warmups 2 --seed 4 --ignore-eos --trust-remote-code
", "url": "https://github.com/vllm-project/vllm/pull/49544", "createdAt": "2026-07-23T07:20:08Z", "updatedAt": "2026-08-13T13:01:38Z", "timestamp": "2026-08-13T13:01:38Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "v1" ], "author": "akii96", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5778331eeee4693fc843", "signalId": "github:vllm-project/vllm:pull_request:52156", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52156", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Apply attention sinks in the Transformers backend", "text": "## Purpose vLLM PR #48270 adds native `GraniteSWAForCausalLM` / `GraniteMoeSWAForCausalLM` because \"the transformers backend does NOT handle these models correctly. Sink tokens get dropped silently and so the model gives wrong output.\" That is accurate, and it is a bug in the Transformers modeling backend rather than something inherent to those models, so this PR fixes it there and every current and future sink model works on the backend. Transformers models with learnable attention sinks (GPT-OSS, GraniteSWA, GraniteMoeSWA, DeepSeek V4, MiMo-V2-Flash, ...) hand the per-head sink to the attention interface as `s_aux`. Only the attention impl can fold a sink into the softmax denominator, so vLLM has to pass it to `Attention` at construction time. `vllm_attention_forward` accepted it in `**kwargs` and dropped it, so every softmax in the model was wrong. Reproduced with `ibm-granite/granite-swash-2b` (greedy, transformers backend): | prompt | before | after | | --- | --- | --- | | `The capital of France is` | `''` | `' Paris.\\nThe capital of France is the largest city...'` | | `def fibonacci(n):` | `'::::::::::::::::::::::::'` | a correct fibonacci body | | `The boiling point of water at sea level is` | `'\\n\\n\\n at at at at at'` | `' 100 degrees Celsius (212 degrees Fahrenheit).'` | Two defects were involved: 1. **Sinks were dropped.** `Base.find_sinks` now locates the parameter Transformers keeps the sinks in, `Base.init_sink` materializes it as this rank's slice of the heads with a sharded weight loader, and `create_attention_instances` passes it as `Attention(sinks=...)`. `vllm_attention_forward` now raises instead of silently ignoring `s_aux` if a model applies sinks that we failed to find. 2. **`Attention` instances were invisible to `named_modules()`.** They are only stored in a plain dict, so `process_weights_after_loading` never ran for them, which is where FlashInfer upcasts sinks to the fp32 tensor its kernel requires. They are now registered in an `nn.ModuleList`, which also replaces the narrower `_vllm_mla_attn` workaround that existed for MLA layers. ## Test plan New tests in `tests/models/transformers/test_backend.py`: - `test_sinks` loads `tiny-random/gpt-oss-bf16` on the backend and asserts every attention layer has a sink and that the tensor it was handed matches the checkpoint. Skipped below SM90, where no backend supports sinks. - `check_implementation` now also asserts that attention instances are registered submodules, covering defect 2 for every model it is used with. Commands and results (1x GB200, transformers 5.15.0): ``` pytest tests/models/transformers/test_backend.py -k \"sinks\" # 1 passed pytest tests/models/transformers/test_backend.py -k \"test_mla or hybrid\" # 2 passed ``` ## Model evaluation Prompt logprobs for a 55-token paragraph, `ibm-granite/granite-swash-2b`, bf16, against HF eager as the reference (mean logprob -1.5943): | build | mean logprob | mean abs delta | max abs delta | | --- | --- | --- | --- | | main | -74.0868 | 72.4925 | 175.4590 | | this PR | -7.8748 | 6.8942 | 81.0792 | Greedy continuations become token-identical to HF with this PR. The residual logprob gap is a second, unrelated bug in the backend - Granite's `logits_scaling` divisor is ignored - fixed separately in #52158, which takes the same measurement to a mean absolute delta of 0.0165. ## Notes - Not a duplicate: no open PR addresses sinks in the Transformers backend (checked `is:pr is:open` for \"transformers backend\" and \"sinks\"). #48270 adds native implementations for two Granite variants; this fixes the backend for all sink models. The two changes are compatible. - Tensor parallel sink sharding follows the existing `sharded_weight_loader(0)` pattern (as in `gpt_oss.py`) but was not exercised, since only one GPU was available. Worth running `test_distributed`-style coverage before merge. - AI assistance was used for this change (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/52156", "createdAt": "2026-08-13T11:09:08Z", "updatedAt": "2026-08-13T13:01:02Z", "timestamp": "2026-08-13T13:01:02Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "tdoublep", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9c67526ceab23b720dab", "signalId": "github:vllm-project/vllm:pull_request:51793", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Remove dead `QuantizationConfig.is_mxfp4_quant`", "text": "## Purpose This is dead code following https://github.com/vllm-project/vllm/pull/37128. This was originally added in https://github.com/vllm-project/vllm/pull/29008 that supported padding for gpt-oss / certain MXFP4 backends, see: https://github.com/xuebwang-amd/vllm/blob/c62f664e97977ee54ab1d1c77604ebb45081bc06/vllm/model_executor/layers/fused_moe/layer.py#L259-L277 This is now handled in: https://github.com/vllm-project/vllm/blob/490259c1f63faf025b8050504db63d81c817d781/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L640 ## Test Plan N/A ## Test Result N/A", "url": "https://github.com/vllm-project/vllm/pull/51793", "createdAt": "2026-08-11T09:46:17Z", "updatedAt": "2026-08-13T13:00:34Z", "timestamp": "2026-08-13T13:00:34Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:a66abdb0a858c046ce4c", "signalId": "github:vllm-project/vllm:pull_request:52145", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52145", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add missing return type annotations in outputs.py", "text": "### Purpose Add missing return type annotations to from_base() static methods and PoolingRequestOutput.__repr__() in vllm/outputs.py. All from_base() static methods on EmbeddingOutput, ClassificationOutput, ScoringOutput, EmbeddingRequestOutput, ClassificationRequestOutput, and ScoringRequestOutput lacked return type annotations. PoolingRequestOutput.__repr__() was missing -> str, making it the only __repr__ in the file without one — every other __repr__ in the same file already has it. No logic changes. ### Test Plan pre-commit run ruff-check --files vllm/outputs.py pre-commit run mypy-3.10 --files vllm/outputs.py --hook-stage manual ### Test Result ruff check...........Passed Run mypy for Python 3.10...........Passed", "url": "https://github.com/vllm-project/vllm/pull/52145", "createdAt": "2026-08-13T10:04:56Z", "updatedAt": "2026-08-13T12:58:46Z", "timestamp": "2026-08-13T12:58:46Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "verified" ], "author": "vineetatiwari27", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:620a8091acf7b1f47f9f", "signalId": "github:vllm-project/vllm:pull_request:46638", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46638", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Add FlashInfer CuteDSL non-causal decode path for DFlash", "text": "## Purpose Part of the DFlash bring-up tracker #46105 (\"Attention backend → CuteDSL via FlashInfer\", contribution-welcome item). vLLM currently runs non-causal DFlash queries through the FlashInfer *prefill* wrapper (`BatchPrefillWithPagedKVCacheWrapper` with `causal=False`), because the decode/TRTLLM paths cannot express non-causal attention. FlashInfer's CuteDSL paged decode wrapper (`BatchDecodePagedCuteDSLWrapper`) supports `q_len_per_req > 1` and `is_causal=False`, which maps directly onto the DFlash decode workload (uniform `1 + num_speculative_tokens` query tokens per request). This PR adds a `CuteDSLDecode` dispatch mode to the FlashInfer backend: - `VLLM_FLASHINFER_CUTEDSL_DECODE` env flag, default off. - Static gate: SM100 only, full attention only (no SWA / soft cap / sinks), non-quantized KV cache (CuteDSL requires `kv_data_type == q_data_type`), page size in {8, 16, 32, 64}, no CUDA graph (Phase 2). - Per-batch gate: non-causal batches with a uniform query length per request; anything else falls back to the existing non-causal prefill path. ### FlashInfer version requirement The CuteDSL non-causal decode kernel had a boundary-mask bug (softmax normalized over padded KV tiles when `seq_len % 256 != 0`), fixed in flashinfer-ai/flashinfer#3717. The fix first shipped in the `0.6.14.dev20260706` nightly and landed in the `0.6.15` stable release. This PR bumps `requirements/cuda.txt` (and the Dockerfile pin) to `0.6.15` accordingly; the runtime helper only feature-detects the wrapper, so the default-off flag still guards correctness against any older FlashInfer install. Verified on B200 (SM100) with `0.6.14.dev20260706`: non-causal output matches reference for all tested `seq_len` including non-multiples of 256. ## Test Plan ```bash # Unit: CuteDSL decode vs SDPA reference (SM100 + flashinfer nightly required), # includes fallback coverage for non-uniform query lengths pytest tests/v1/attention/test_attention_backends.py -k cutedsl -v # Existing non-causal prefill path unaffected pytest tests/v1/attention/test_attention_backends.py -k non_causal -v ``` E2E (Qwen3-8B + DFlash draft, acceptance-length parity) and kernel microbench vs the non-causal prefill baseline: see Test Result below. ## Test Result Environment: NVIDIA B200 (SM100), torch 2.11.0, flashinfer 0.6.14.dev20260706, CUDA 13.0. **Unit (2026-07-07, run locally with an ungated same-architecture model, Qwen/Qwen3-0.6B, since this env has no HF token; the committed test uses Meta-Llama-3-8B as the rest of the file):** - `test_non_causal_cutedsl_decode_correctness`: 4/4 passed - `small_decode`, `small_prefill`, `medium_prefill`: routed through CuteDSL (wrapper-getter spy called), output matches SDPA reference (atol/rtol 1e-2) - `mixed_small` (non-uniform query lens): fell back to non-causal prefill, correct output, CuteDSL not invoked - seq_lens include non-multiples of 256, so the flashinfer-ai/flashinfer#3717 boundary-mask fix is exercised - Causal regression (FLASHINFER backend, flag on/off × small_decode/mixed_small): 4/4 passed - Kernel-level repro of the flashinfer bug against 0.6.14.dev20260706: ratio == 1.000 for all seq_lens (16–512) with `is_causal=False` — upstream fix confirmed **Kernel microbench (B200, DFlash shape: q_len_per_req=17, GQA 32/8, head_dim 128, page 16, bf16, HND layout, CUDA events, median of 50):** CuteDSL paged decode (`reduction=\"auto\"`) vs `BatchPrefillWithPagedKVCacheWrapper(causal=False)`: | batch | kv_len | prefill (ms) | cutedsl (ms) | speedup | |---|---|---|---|---| | 1 | 512 | 0.053 | 0.048 | 1.10x | | 1 | 2048 | 0.083 | 0.111 | 0.75x | | 1 | 8192 | 0.128 | 0.178 | 0.72x | | 4 | 2048 | 0.104 | 0.085 | 1.23x | | 8 | 2048 | 0.120 | 0.095 | 1.26x | | 16 | 2048 | 0.157 | 0.112 | 1.41x | | 16 | 8192 | 0.313 | 0.231 | 1.36x | | 32 | 8192 | 0.496 | 0.385 | 1.29x | - `reduction=\"kernel\"` (deterministic) is consistently *slower* than the prefill baseline (0.53–0.84x) — hence the integration uses `\"auto\"` (atomic; out buffer zero-initialized in `forward()`) and falls back to `\"kernel\"` only under `VLLM_BATCH_INVARIANT`. - Known weak spot: batch 1 × long context (0.72–0.75x). Everything else 1.1–1.4x. **E2E (Qwen/Qwen3-8B + z-lab/Qwen3-8B-DFlash-b16, num_spec_tokens=16, enforce_eager, 12 prompts × 512 tok, temp 0, 3 runs per config):** - CuteDSL path confirmed engaged when flag on (wrapper-getter called 362×; 0× when off). - Acceptance length: 3.7117 (off) vs 3.7214 (on) — reproducible per config across runs; drafts identical (1065). - 8/12 outputs bitwise identical; 4 diverge mid-generation into equally-plausible continuations (expected kernel-level numeric differences under temp 0). - Throughput (tok/s): off 651/644/730, on 612/709/741 — parity within machine noise (±7%) at this small scale; the kernel microbench above is the controlled performance signal, not this run. `vllm bench serve` (sonnet, 1024 in / 512 out): conc=1 median TPOT 7.92 ms (off) vs 9.79 ms (on); conc=8 742 vs 713 tok/s. Acceptance lengths differed between runs (2.75/2.53 vs 2.57/2.59) due to random sonnet sampling, so these numbers are noisy and not used as the performance evidence for this PR — included for completeness only. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/46638", "createdAt": "2026-06-24T16:29:05Z", "updatedAt": "2026-08-13T12:55:37Z", "timestamp": "2026-08-13T12:55:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ci/build", "v1", "nvidia" ], "author": "elwhyjay", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ede9c5bed608a95257b6", "signalId": "github:vllm-project/vllm:pull_request:48247", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48247", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][ROCm] Add AITER custom AG/RS", "text": "## Purpose Part of #48255. Only activated under uniform batches. Enabled by default when using DP attention. Same as ATOM. Disable with `VLLM_ROCM_USE_AITER_CUSTOM_AR=0`. **Perf gain:** **~3% improved TPOT on 1k/1k** (1-256 conc) ### Technical details - AG: Runs as 3 separate calls for hidden, topk_weights, and topk_ids. AITER doesn't have a grouped AG unfortunately. \"image\" - Zooming in we see these kernels now being run in uniform decode batches: \"image\" \"image\" - Mixed batches still use NCCL/RCCL: \"image\" ## Test Plan Tested on 8xMI300. 1. Run DSv3 with DPA+TP: ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` Compare vs nightly `vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9`. 2. Bench serve sweep, 1k/1k and 8k/1k ISL/OSL - 10x as many prompts as the concurrency level; 2x number of warmups. ```bash vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 \\ --num-warmups 256 ``` 3. GSM8k accuracy non-regression ```bash lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result ### 1. Performance Only decode path affected, so focus on TPOT. #### 1k/1k (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 262.54 | 21.64 | **+12.46%** | | 1 | nightly | 0.04 | 273.16 | 24.72 | - | | 2 | this branch | 0.08 | 439.28 | 23.60 | **+6.79%** | | 2 | nightly | 0.08 | 298.14 | 25.32 | - | | 4 | this branch | 0.16 | 481.21 | 24.02 | **+12.50%** | | 4 | nightly | 0.14 | 366.23 | 27.45 | - | | 8 | this branch | 0.29 | 701.59 | 26.51 | **+16.98%** | | 8 | nightly | 0.24 | 664.92 | 31.93 | - | | 16 | this branch | 0.44 | 1065.97 | 34.56 | **-5.37%** | | 16 | nightly | 0.46 | 851.06 | 32.80 | - | | 32 | this branch | 0.68 | 2125.91 | 44.33 | **-2.57%** | | 32 | nightly | 0.70 | 1698.74 | 43.22 | - | | 64 | this branch | 1.32 | 2901.05 | 44.66 | **+2.19%** | | 64 | nightly | 1.29 | 2528.00 | 45.66 | - | | 128 | this branch | 2.17 | 4486.44 | 53.32 | **+2.82%** | | 128 | nightly | 2.10 | 4543.42 | 54.87 | - | | 256 | this branch | 3.48 | 6351.00 | 65.49 | **+3.49%** | | 256 | nightly | 3.37 | 6386.54 | 67.86 | - | #### 8k/1k Not as clear improvement, a bit variable. Expecting 1k/1k results to be stronger since that's more decode-heavy, which this PR should affect. (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.0420 | 1690.7196 | 21.6517 | **+3.80%** | | 1 | nightly | 0.0404 | 1681.0387 | 22.5068 | - | | 2 | this branch | 0.0705 | 1872.6645 | 25.4534 | **-11.53%** | | 2 | nightly | 0.0766 | 1872.0056 | 22.8210 | - | | 4 | this branch | 0.1306 | 3409.2517 | 27.2208 | **+3.05%** | | 4 | nightly | 0.1241 | 3439.4700 | 28.0764 | - | | 8 | this branch | 0.2267 | 4008.7127 | 29.8010 | **+2.15%** | | 8 | nightly | 0.2214 | 3836.1143 | 30.4567 | - | | 16 | this branch | 0.3372 | 5931.2343 | 39.9316 | **+9.28%** | | 16 | nightly | 0.3135 | 6404.6625 | 44.0170 | - | | 32 | this branch | 0.4641 | 8843.5975 | 59.0122 | **-7.02%** | | 32 | nightly | 0.4905 | 9516.1175 | 55.1427 | - | | 64 | this branch | 0.7066 | 11619.6178 | 77.3325 | **+2.10%** | | 64 | nightly | 0.6972 | 11196.8358 | 78.9877 | - | | 128 | this branch | 0.9199 | 14325.6379 | 122.2971 | **-0.14%** | | 128 | nightly | 0.9067 | 13800.9142 | 122.1202 | - | ### 2. Accuracy #### This branch |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9469|± |0.0062| | | |strict-match | 5|exact_match|↑ |0.9462|± |0.0062| #### Nightly |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48247", "createdAt": "2026-07-10T11:36:54Z", "updatedAt": "2026-08-13T12:54:34Z", "timestamp": "2026-08-13T12:54:34Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "rocm", "nvidia" ], "author": "simondanielsson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dc04aafea1be129c4cf7", "signalId": "github:vllm-project/vllm:pull_request:51695", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51695", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MOE] Standardize and abstract fused shared expert optimization selection", "text": "## Disclosure AI assistance was used. The changes were reviewed and tested manually. ## Purpose Standardize fused shared-expert (FSE) detection so model construction and checkpoint loading use the same quantization-compatible decision **throughout all models implementing FSE**. This is e.g. useful for `shared_expert` quantization through various quantization methods (e.g. online quantization in https://github.com/vllm-project/vllm/pull/51285 / https://github.com/vllm-project/vllm/pull/51392), so that FSE compatibility can be detected more largely based on arbitrary quantization config across all model archs, and not simply based on https://github.com/vllm-project/vllm/blob/fac808b36f502d0d992509a187dca94c68b0360a/vllm/model_executor/models/qwen3_next.py#L81-L98 (qwen3 specific, quark specific) This is also useful e.g. for https://github.com/vllm-project/vllm/issues/50963 that is NOT rocm/aiter specific. The shared resolver validates FSE against the quantization configuration, rejects an incompatible requested FSE configuration, and ensures all MoE layers in a model agree on the resolved state. The mixed-state error reports the counts of enabled and disabled layers. Behavior change compared with `main`: - When `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS` is enabled but quantization is incompatible, model construction now raises `ValueError` instead of falling back to the non-fused shared-expert path. - Models with mixed resolved FSE states now raise `NotImplementedError`. FSE is only supported/tested for all (or none) layers currently. Left to do: - [ ] Maybe left to an other PR: Should `not get_current_vllm_config().parallel_config.enable_expert_parallel` be promoted globally (from `vllm/models/minimax_m3/amd/model.py`) - [ ] Maybe left to an other PR: Should `on_gfx950()` from minimax_m3 be promoted (// backend specific?)? - [x] update `deepseek_v4/amd/model.py` - [ ] Reliance on `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS` for non-AITER backends to turn on FSE in MiniMax-M3 on AMD https://github.com/vllm-project/vllm/blob/0e2d78028c473bb04fc74bf945eb32da29239b0c/vllm/models/minimax_m3/amd/model.py#L112-L127 ==> does not require `VLLM_USE_AITER=1`, does not use `rocm_aiter_ops.is_fusion_moe_shared_experts_enabled()` - [ ] Check on `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` once https://github.com/vllm-project/vllm/pull/48223 is merged ## Test Plan - `pytest tests/model_executor/layers/test_fused_shared_expert.py -v` ## Test Result Tests pass.", "url": "https://github.com/vllm-project/vllm/pull/51695", "createdAt": "2026-08-10T15:04:56Z", "updatedAt": "2026-08-13T12:51:46Z", "timestamp": "2026-08-13T12:51:46Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "qwen", "deepseek", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:391793369b49e2ddaf56", "signalId": "github:vllm-project/vllm:pull_request:51851", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51851", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump the minor-update group across 1 directory with 174 updates", "text": "Bumps the minor-update group with 174 updates in the / directory: | Package | From | To | | --- | --- | --- | | [regex](https://github.com/mrabarnett/mrab-regex) | `2026.2.28` | `2026.7.19` | | [requests](https://github.com/psf/requests) | `2.32.3` | `2.34.2` | | [tqdm](https://github.com/tqdm/tqdm) | `4.67.3` | `4.70.0` | | [tokenizers](https://github.com/huggingface/tokenizers) | `0.22.2` | `0.23.1` | | [fastapi](https://github.com/fastapi/fastapi) | `0.136.3` | `0.141.1` | | [starlette](https://github.com/Kludex/starlette) | `1.3.1` | `1.6.0` | | [aiohttp](https://github.com/aio-libs/aiohttp) | `3.13.3` | `3.14.3` | | [openai](https://github.com/openai/openai-python) | `2.44.0` | `2.53.0` | | [pydantic](https://github.com/pydantic/pydantic) | `2.12.0` | `2.13.4` | | [prometheus-client](https://github.com/prometheus/client_python) | `0.22.0` | `0.26.0` | | [prometheus-fastapi-instrumentator](https://github.com/trallnag/prometheus-fastapi-instrumentator) | `8.0.2` | `8.1.0` | | [tiktoken](https://github.com/openai/tiktoken) | `0.12.0` | `0.13.0` | | [lark](https://github.com/lark-parser/lark) | `1.2.2` | `1.3.1` | | [typing-extensions](https://github.com/python/typing_extensions) | `4.15.0` | `4.16.0` | | [filelock](https://github.com/tox-dev/py-filelock) | `3.16.1` | `3.32.2` | | [jsonschema](https://github.com/python-jsonschema/jsonschema) | `4.23.0` | `4.26.0` | | [pybase64](https://github.com/mayeut/pybase64) | `1.4.3` | `1.5.0` | | [anthropic](https://github.com/anthropics/anthropic-sdk-python) | `0.112.0` | `0.121.0` | | [opentelemetry-sdk](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-api](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | opentelemetry-semantic-conventions-ai | `0.4.13` | `0.5.1` | | [grpcio](https://github.com/grpc/grpc) | `1.78.0` | `1.83.0` | | [grpcio-reflection](https://grpc.io) | `1.78.0` | `1.83.0` | | [numba](https://github.com/numba/numba) | `0.65.0` | `0.66.0` | | [peft](https://github.com/huggingface/peft) | `0.19.1` | `0.20.0` | | [tensorizer](https://github.com/coreweave/tensorizer) | `2.10.1` | `2.12.1` | | runai-model-streamer | `0.15.7` | `0.16.1` | | [conch-triton-kernels](https://github.com/stackav-oss/conch) | `1.2.1` | `1.3` | | [annotated-types](https://github.com/annotated-types/annotated-types) | `0.7.0` | `0.8.0` | | griffelib | `2.0.2` | `2.1.0` | | [idna](https://github.com/kjd/idna) | `3.10` | `3.18` | | [platformdirs](https://github.com/tox-dev/platformdirs) | `4.3.6` | `4.11.1` | | [pydantic-core](https://github.com/pydantic/pydantic) | `2.41.1` | `2.48.0` | | [pygments](https://github.com/pygments/pygments) | `2.18.0` | `2.20.0` | | [ruff](https://github.com/astral-sh/ruff) | `0.15.12` | `0.16.2` | | [six](https://github.com/benjaminp/six) | `1.16.0` | `1.17.0` | | [urllib3](https://github.com/urllib3/urllib3) | `2.2.3` | `2.7.0` | | [tblib](https://github.com/ionelmc/python-tblib) | `3.1.0` | `3.2.2` | | [absl-py](https://github.com/abseil/abseil-py) | `2.1.0` | `2.5.0` | | [accelerate](https://github.com/huggingface/accelerate) | `1.13.0` | `1.14.0` | | [aiohappyeyeballs](https://github.com/aio-libs/aiohappyeyeballs) | `2.6.1` | `2.7.1` | | [alembic](https://github.com/sqlalchemy/alembic) | `1.16.4` | `1.19.1` | | arctic-inference | `0.1.1` | `0.2.0` | | [argcomplete](https://github.com/kislyuk/argcomplete) | `3.5.1` | `3.7.2` | | [audioread](https://github.com/beetbox/audioread) | `3.0.1` | `3.1.0` | | [azure-core](https://github.com/Azure/azure-sdk-for-python) | `1.38.2` | `1.41.0` | | [azure-storage-blob](https://github.com/Azure/azure-sdk-for-python) | `12.28.0` | `12.30.0` | | [blobfile](https://github.com/blobfile/blobfile) | `3.0.0` | `3.2.0` | | [bm25s](https://github.com/xhluca/bm25s) | `0.2.13` | `0.3.10` | | [boto3](https://github.com/boto/boto3) | `1.35.57` | `1.43.67` | | [botocore](https://github.com/boto/botocore) | `1.35.57` | `1.43.67` | | [cffi](https://github.com/python-cffi/cffi) | `2.0.0` | `2.1.1` | | [cohere-melody](https://github.com/cohere-ai/melody) | `0.11.1` | `0.13.0` | | [colorlog](https://github.com/borntyping/python-colorlog) | `6.10.1` | `6.12.0` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.10.6` | `7.15.4` | | [cramjam](https://github.com/milesgranger/pyrus-cramjam) | `2.9.0` | `2.11.0` | | [cuda-bindings](https://github.com/NVIDIA/cuda-python) | `13.0.3` | `13.3.1` | | [cuda-pathfinder](https://github.com/NVIDIA/cuda-python) | `1.3.3` | `1.6.0` | | [cuda-toolkit](https://developer.nvidia.com/cuda-toolkit) | `13.0.3.0` | `13.3.1` | | [datamodel-code-generator](https://github.com/koxudaxi/datamodel-code-generator) | `0.26.3` | `0.72.2` | | [dataproperty](https://github.com/thombashi/DataProperty) | `1.0.1` | `1.1.1` | | [decorator](https://github.com/micheles/decorator) | `5.1.1` | `5.3.1` | | [dill](https://github.com/uqfoundation/dill) | `0.3.8` | `0.4.1` | | [distlib](https://github.com/pypa/distlib) | `0.3.9` | `0.4.3` | | [dnspython](https://github.com/rthalley/dnspython) | `2.7.0` | `2.8.0` | | [docker](https://github.com/docker/docker-py) | `7.1.0` | `7.2.0` | | [einx](https://github.com/fferflo/einx) | `0.3.0` | `0.4.3` | | [email-validator](https://github.com/JoshData/python-email-validator) | `2.2.0` | `2.3.0` | | [fastapi-cloud-cli](https://github.com/fastapilabs/fastapi-cloud-cli) | `0.21.0` | `0.23.0` | | [fonttools](https://github.com/fonttools/fonttools) | `4.55.0` | `4.63.0` | | [frozenlist](https://github.com/aio-libs/frozenlist) | `1.5.0` | `1.8.0` | | [genson](https://github.com/wolverdude/genson) | `1.3.0` | `1.4.0` | | [google-api-core](https://github.com/googleapis/google-cloud-python) | `2.24.2` | `2.34.0` | | [google-auth](https://github.com/googleapis/google-cloud-python) | `2.40.2` | `2.56.3` | | [google-cloud-core](https://github.com/googleapis/google-cloud-python) | `2.4.3` | `2.6.1` | | [google-cloud-storage](https://github.com/googleapis/google-cloud-python) | `3.4.0` | `3.13.1` | | [google-crc32c](https://github.com/googleapis/python-crc32c) | `1.7.1` | `1.8.0` | | [google-resumable-media](https://github.com/googleapis/google-cloud-python) | `2.7.2` | `2.10.1` | | [googleapis-common-protos](https://github.com/googleapis/google-cloud-python) | `1.70.0` | `1.75.1` | | [greenlet](https://github.com/python-greenlet/greenlet) | `3.2.3` | `3.5.4` | | [h11](https://github.com/python-hyper/h11) | `0.14.0` | `0.16.0` | | [h2](https://github.com/python-hyper/h2) | `4.3.0` | `4.4.1` | | [hiredis](https://github.com/redis/hiredis-py) | `3.0.0` | `3.4.1` | | [hpack](https://github.com/python-hyper/hpack) | `4.1.0` | `4.2.0` | | [httpx](https://github.com/encode/httpx) | `0.27.2` | `0.28.1` | | [humanize](https://github.com/python-humanize/humanize) | `4.11.0` | `4.16.0` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.131.0` | `6.165.2` | | [iniconfig](https://github.com/pytest-dev/iniconfig) | `2.0.0` | `2.3.0` | | [jiter](https://github.com/pydantic/jiter) | `0.15.0` | `0.16.0` | | [jmespath](https://github.com/jmespath/jmespath.py) | `1.0.1` | `1.1.0` | | [joblib](https://github.com/joblib/joblib) | `1.4.2` | `1.5.3` | | [jsonschema-rs](https://github.com/Stranger6667/jsonschema) | `0.46.5` | `0.49.9` | | [kiwisolver](https://github.com/nucleic/kiwi) | `1.4.7` | `1.5.0` | | [lazy-loader](https://github.com/scientific-python/lazy-loader) | `0.4` | `0.5` | | [librosa](https://github.com/librosa/librosa) | `0.10.2.post1` | `0.11.0` | | [llvmlite](https://github.com/numba/llvmlite) | `0.47.0` | `0.48.0` | | [mako](https://github.com/sqlalchemy/mako) | `1.3.10` | `1.4.1` | | [matplotlib](https://github.com/matplotlib/matplotlib) | `3.9.2` | `3.10.9` | | [mpmath](https://github.com/mpmath/mpmath) | `1.3.0` | `1.4.1` | | [msal](https://github.com/AzureAD/microsoft-authentication-library-for-python) | `1.34.0` | `1.37.0` | | [msgpack](https://github.com/msgpack/msgpack-python) | `1.1.0` | `1.2.1` | | [mteb](https://github.com/embeddings-benchmark/mteb) | `2.8.3` | `2.18.16` | | [multidict](https://github.com/aio-libs/multidict) | `6.1.0` | `6.7.1` | | [mypy-extensions](https://github.com/python/mypy_extensions) | `1.0.0` | `1.1.0` | | [networkx](https://github.com/networkx/networkx) | `3.2.1` | `3.4.2` | | [nltk](https://github.com/nltk/nltk) | `3.9.1` | `3.10.2` | | [nvidia-cublas](https://developer.nvidia.com/cuda-zone) | `13.1.1.3` | `13.6.1.10` | | [nvidia-cuda-cupti](https://developer.nvidia.com/cuda-zone) | `13.0.85` | `13.3.75` | | [nvidia-cuda-nvrtc](https://developer.nvidia.com/cuda-zone) | `13.0.88` | `13.3.33` | | [nvidia-cuda-runtime](https://developer.nvidia.com/cuda-zone) | `13.0.96` | `13.3.29` | | [nvidia-cudnn-cu13](https://developer.nvidia.com/cuda-zone) | `9.20.0.48` | `9.24.0.43` | | [nvidia-cufft](https://developer.nvidia.com/cuda-zone) | `12.0.0.61` | `12.3.0.29` | | [nvidia-cufile](https://developer.nvidia.com/cuda-zone) | `1.15.1.6` | `1.18.1.6` | | [nvidia-cusolver](https://developer.nvidia.com/cuda-zone) | `12.0.4.66` | `12.2.6.9` | | [nvidia-cusparse](https://developer.nvidia.com/cuda-zone) | `12.6.3.3` | `12.8.2.51` | | [nvidia-cusparselt-cu13](https://developer.nvidia.com/cusparselt) | `0.8.1` | `0.9.1` | | [nvidia-nccl-cu13](https://developer.nvidia.com/cuda-zone) | `2.29.7` | `2.30.7` | | [nvidia-nvjitlink](https://developer.nvidia.com/cuda-zone) | `13.0.88` | `13.3.33` | | [nvidia-nvshmem-cu13](https://developer.nvidia.com/cuda-zone) | `3.4.5` | `3.7.2` | | [nvidia-nvtx](https://developer.nvidia.com/cuda-zone) | `13.0.85` | `13.3.29` | | [opentelemetry-exporter-otlp-proto-common](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp-proto-grpc](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-otlp-proto-http](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-exporter-prometheus](https://github.com/open-telemetry/opentelemetry-python) | `0.56b0` | `0.65b0` | | [opentelemetry-proto](https://github.com/open-telemetry/opentelemetry-python) | `1.35.0` | `1.44.0` | | [opentelemetry-semantic-conventions](https://github.com/open-telemetry/opentelemetry-python) | `0.56b0` | `0.65b0` | | [pathvalidate](https://github.com/thombashi/pathvalidate) | `3.2.1` | `3.3.1` | | [perceptron](https://github.com/perceptron-ai-inc/perceptron) | `0.1.4` | `0.3.5` | | [pluggy](https://github.com/pytest-dev/pluggy) | `1.5.0` | `1.6.0` | | [polars](https://github.com/pola-rs/polars) | `1.29.0` | `1.43.2` | | [pooch](https://github.com/fatiando/pooch) | `1.8.2` | `1.9.0` | | [propcache](https://github.com/aio-libs/propcache) | `0.2.0` | `0.5.2` | | [proto-plus](https://github.com/googleapis/google-cloud-python) | `1.26.1` | `1.28.3` | | [pycryptodomex](https://github.com/Legrandin/pycryptodome) | `3.22.0` | `3.23.0` | | [pydantic-extra-types](https://github.com/pydantic/pydantic-extra-types) | `2.10.5` | `2.11.1` | | [pydantic-settings](https://github.com/pydantic/pydantic-settings) | `2.14.2` | `2.15.0` | | [pyjwt](https://github.com/jpadilla/pyjwt) | `2.11.0` | `2.13.0` | | [pyparsing](https://github.com/pyparsing/pyparsing) | `3.2.0` | `3.3.2` | | [pystemmer](https://github.com/snowballstem/pystemmer) | `3.0.0` | `3.1.0` | | [pytest-forked](https://github.com/pytest-dev/pytest-forked) | `1.6.0` | `1.7.5` | | [pytest-mock](https://github.com/pytest-dev/pytest-mock) | `3.14.0` | `3.15.1` | | [pytest-timeout](https://github.com/pytest-dev/pytest-timeout) | `2.3.1` | `2.4.0` | | [python-rapidjson](https://github.com/python-rapidjson/python-rapidjson) | `1.20` | `1.23` | | [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) | `3.12.1` | `3.14.5` | | [referencing](https://github.com/python-jsonschema/referencing) | `0.35.1` | `0.37.0` | | [responses](https://github.com/getsentry/responses) | `0.25.3` | `0.26.2` | | rignore | `0.7.6` | `0.8.1` | | runai-model-streamer-azure | `0.15.7` | `0.16.1` | | runai-model-streamer-gcs | `0.15.7` | `0.16.1` | | runai-model-streamer-s3 | `0.15.7` | `0.16.1` | | [s3transfer](https://github.com/boto/s3transfer) | `0.10.3` | `0.19.2` | | [sacrebleu](https://github.com/mjpost/sacrebleu) | `2.4.3` | `2.6.0` | | [schemathesis](https://github.com/schemathesis/schemathesis) | `4.21.6` | `4.24.3` | | [scikit-learn](https://github.com/scikit-learn/scikit-learn) | `1.5.2` | `1.7.2` | | [scipy](https://github.com/scipy/scipy) | `1.13.1` | `1.15.3` | | [sentence-transformers](https://github.com/huggingface/sentence-transformers) | `5.2.0` | `5.7.0` | | [sentry-sdk](https://github.com/getsentry/sentry-python) | `2.63.0` | `2.67.0` | | [soundfile](https://github.com/bastibe/python-soundfile) | `0.12.1` | `0.14.0` | | [sympy](https://github.com/sympy/sympy) | `1.13.3` | `1.14.0` | | [tabulate](https://github.com/astanin/python-tabulate) | `0.9.0` | `0.10.0` | | [termcolor](https://github.com/termcolor/termcolor) | `3.1.0` | `3.3.0` | | [threadpoolctl](https://github.com/joblib/threadpoolctl) | `3.5.0` | `3.6.0` | | [torchcodec](https://github.com/pytorch/torchcodec) | `0.14.0+cu130` | `0.15.0` | | [tritonclient](https://github.com/triton-inference-server/client) | `2.64.0` | `2.71.0` | | [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.1` | | [uvicorn](https://github.com/Kludex/uvicorn) | `0.35.0` | `0.52.1` | | [vector-quantize-pytorch](https://github.com/lucidrains/vector-quantizer-pytorch) | `1.21.2` | `1.31.1` | | [wcwidth](https://github.com/jquast/wcwidth) | `0.2.13` | `0.8.2` | | [xxhash](https://github.com/ifduyue/python-xxhash) | `3.5.0` | `3.8.1` | | [yarl](https://github.com/aio-libs/yarl) | `1.17.1` | `1.24.5` | | pynvvideocodec | `2.0.4` | `2.2.0` | | [nvidia-cutlass-dsl](https://github.com/NVIDIA/cutlass) | `4.6.2` | `4.7.0` | | [tokenspeed-mla](https://github.com/lightseekorg/tokenspeed) | `0.1.8` | `0.2.5` | Updates `regex` from 2026.2.28 to 2026.7.19
Changelog

Sourced from regex's changelog.

Version: 2026.8.12

Support Python 3.15.

Version: 2026.7.19

Git issue 607: SIGSEGV: NULL pointer dereference in `basic_match()` when `(?R)`/`(?0)` recursion is used inside a fuzzy-matching quantifier Git issue 608: SIGSEGV: out-of-bounds read in `bytes1_char_at()` with `DOTALL` + a fuzzy constraint + `(?r)` reverse-direction matching

Version: 2026.7.10

Clarified that the POSIX flag makes it look for only the longest overall match. 
Git issue 606: regex's compiled-pattern cache eviction is not free-threading-safe

Version: 2026.6.28

Git issue 604: regex fails on free-threading python because setlocale is not thread-safe

Version: 2026.5.9

Reverse matching with full unicode casefolding could lead to out-of-range string indexes.

Version: 2026.4.4

A fix for older Python versions before free-threading was supported.

Version: 2026.4.3

More fixes for free-threading.

Version: 2026.3.32

Fixed segfault.

Version: 2026.3.31

Fixed bug again.

Version: 2026.3.30

Fixed bug.

Version: 2026.3.28

Fixed version.

Version: 2026.3.27

Various fixes, including ones to improve free-threading support.

... (truncated)

Commits
  • 0525aff Git issue 608: SIGSEGV: out-of-bounds read in bytes1_char_at() with `DOTALL...
  • 07dfa3b Git issue 607: SIGSEGV: NULL pointer dereference in basic_match() when `(?R...
  • e7716da Python 3.15 not released yet.
  • e204ddb Git issue 606: regex's compiled-pattern cache eviction is not free-threadin...
  • d7e1927 Updated main.yml for Python 3.15.
  • b18daa6 Git issue 604: regex fails on free-threading python because setlocale is not ...
  • e57d185 Reverse matching with full unicode casefolding lead to out-of-range string in...
  • bc57b04 A fix for older Python versions before free-threading was supported.
  • 773e213 More fixes for free-threading.
  • 5d51c75 Fixed segfault.
  • Additional commits viewable in compare view

Updates `requests` from 2.32.3 to 2.34.2
Release notes

Sourced from requests's releases.

v2.34.2

2.34.2 (2026-05-14)

  • Moved headers input type back to Mapping to avoid invariance issues with MutableMapping and inferred dict types. Users calling Request.headers.update() may need to narrow typing in their code. (#7441)

Full Changelog: https://github.com/psf/requests/blob/main/HISTORY.md#2342-2026-05-14

v2.34.1

2.34.1 (2026-05-13)

Bugfixes

  • Widened json input type from dict and list to Mapping and Sequence. (#7436)
  • Changed headers input type to MutableMapping and removed None from Request.headers typing to improve handling for users. (#7431)
  • Response.reason moved from str | None to str to improve handling for users. (#7437)
  • Fixed a bug where some bodies with custom __getattr__ implementations weren't being properly detected as Iterables. (#7433)

New Contributors

Full Changelog: https://github.com/psf/requests/blob/main/HISTORY.md#2341-2026-05-13

v2.34.0

2.34.0 (2026-05-11)

Announcements

  • Requests 2.34.0 introduces inline types, replacing those provided by typeshed. Public API types should be fully compatible with mypy, pyright, and ty. We believe types are comprehensive but if you find issues, please report them to the pinned tracking issue.

    Special thanks to @​bastimeyer, @​cthoyt, @​edgarrmondragon, and @​srittau for helping review and test the types ahead of the release. (#7272)

Improvements

  • Digest Auth hashing algorithms have added usedforsecurity=False to clarify security considerations. (#7310)
  • Requests added support for Python 3.15 based on beta1. Downstream projects should be able to start testing prior to its release in October. (#7422)
  • Requests added support for Python 3.14t. (#7419)

Bugfixes

  • Response.history no longer contains a reference to itself, preventing accidental looping when traversing the history list. (#7328)
  • Requests no longer performs greedy matching on no_proxy domains. The

... (truncated)

Changelog

Sourced from requests's changelog.

2.34.2 (2026-05-14)

  • Moved headers input type back to Mapping to avoid invariance issues with MutableMapping and inferred dict types. Users calling Request.headers.update() may need to narrow typing in their code. (#7441)

2.34.1 (2026-05-13)

Bugfixes

  • Widened json input type from dict and list to Mapping and Sequence. (#7436)
  • Changed headers input type to MutableMapping and removed None from Request.headers typing to improve handling for users. (#7431)
  • Response.reason moved from str | None to str to improve handling for users. (#7437)
  • Fixed a bug where some bodies with custom __getattr__ implementations weren't being properly detected as Iterables. (#7433)

2.34.0 (2026-05-11)

Announcements

  • Requests 2.34.0 introduces inline types, replacing those provided by typeshed. Public API types should be fully compatible with mypy, pyright, and ty. We believe types are comprehensive but if you find issues, please report them to the pinned tracking issue.

    Special thanks to @​bastimeyer, @​cthoyt, @​edgarrmondragon, and @​srittau for helping review and test the types ahead of the release. (#7272)

Improvements

  • Digest Auth hashing algorithms have added usedforsecurity=False to clarify security considerations. (#7310)
  • Requests added support for Python 3.15 based on beta1. Downstream projects should be able to start testing prior to its release in October. (#7422)
  • Requests added support for Python 3.14t. (#7419)

Bugfixes

  • Response.history no longer contains a reference to itself, preventing accidental looping when traversing the history list. (#7328)
  • Requests no longer performs greedy matching on no_proxy domains. The proxy_bypass implementation has been updated with CPython's fix from bpo-39057. (#7427)
  • Requests no longer incorrectly strips duplicate leading slashes in URI paths. This should address user issues with specific presigned URLs. Note the full fix requires urllib3 2.7.0+. (#7315)

... (truncated)

Commits

Updates `tqdm` from 4.67.3 to 4.70.0
Release notes

Sourced from tqdm's releases.

tqdm v4.70.0 stable

  • contrib.concurrent: major improvements
    • support process_map(mp_context, max_tasks_per_child), thread_map(thread_name_prefix) (#1265)
    • fix total based on shortest iterable length (#1473)
    • use default max_workers (#1543 <- #1530, #1518)
    • support timeout, buffersize (#1576)
    • improve ETA (#1708 <- #1161)
    • update as_completed (#1709 <- #1565)
    • add tqdm.concurrent.intepreter_map (#1777)
  • asyncio: support iterables with only __aiter__ (#1714 <- #1686)
  • support reset(float("inf")) (#1783 <- #1781, #651)
  • framework: test & reduce wheel size (#1782)

tqdm v4.69.1 stable

tqdm v4.69.0 stable

  • add tqdm.asyncio.gather(..., return_exceptions=False) (#1776, #1671 <- #1286)
  • misc minor framework updates
    • bump workflow actions & pre-commit hooks

tqdm v4.68.4 stable

tqdm v4.68.3 stable

  • utils: delay os.get_terminal_size (#1763 <- #1760)
  • autonotebook: support QtConsole, Spyder, JupyterLite (#1763, #1628, #1559 <- #1283, #1098, #512)
  • minor docs updates
    • fix typo (#1762)
    • use git-fame
  • misc minor framework updates
    • fix & update CI build
    • pre-commit: add docs & metadata generation
    • move tox.ini -> pyproject.toml, move tox-gh-actions -> tox-gh
    • add Python 3.14, drop 3.7 support

tqdm v4.68.2 stable

  • revert accidental change to ascii default (fixes #1760)
    • UnicodeEncodeError: 'charmap' codec can't encode characters in position 6-7: character maps to <undefined> can be fixed by installing tqdm!=4.68.0,!=4.68.1
  • misc docs updates
    • fix links
    • replace stray rst -> md syntax
    • consistent "progress bar" terminology (#1737)

... (truncated)

Commits
  • 96f2e60 Merge pull request #1777 from shermansiu/feat/interpreter-pool
  • c27393e misc tidy
  • 061c623 Disable tqdm.monitor_interval for subinterpreters because they do not have ...
  • 9fc160b Update how we check for interpreter_map support in the tests
  • b42463a Ensure that subinterpreters can import tqdm while unpickling the initializer
  • 16d5486 Add support for nested progress bars
  • 9f5890f Add initial implementation for interpreter_map
  • 321f920 Merge pull request #1783 from LuShadowX/reset-inf-total
  • 4664b57 minor tidy
  • 426a098 Treat inf total as unknown in reset() too
  • Additional commits viewable in compare view

Updates `tokenizers` from 0.22.2 to 0.23.1
Release notes

Sourced from tokenizers's releases.

Release v0.23.1

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

🚨 Breaking changes

  • Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
  • add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
  • Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
  • 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

⚡ Performance — measured locally on this Mac, not lifted from PRs

Run with cargo bench --bench <name> -- --save-baseline v0_22_2 on v0.22.2, then --baseline v0_22_2 on v0.23.1. Numbers are point-in-time wall clock on a single laptop; relative deltas are what matters, absolute numbers will differ on CI hardware.

Added-vocabulary deserialize — the headline win (#1995, #1999)

bench: improve added_vocab_deserialize to reflect real-world workloads (#2000) is now representative of how transformers actually loads tokenizer.json files. The combined effect of daachorse for the matching automaton plus the normalize-on-insert refactor is enormous on this workload:

benchmark v0.22.2 v0.23.1 change
100k tokens, special, no norm ~410 ms 248 ms −40%
100k tokens, non-special, no norm ~7.1 s 273 ms −96%
100k tokens, special, NFKC ~395 ms 235 ms −40%
100k tokens, non-special, NFKC ~7.4 s 290 ms −96%
400k tokens, special, no norm ~15 s 980 ms −94%

Real-world impact: loading a Llama-3-style tokenizer with a large set of added tokens dropped from "noticeable pause" to "instant".

BPE encode

benchmark v0.22.2 v0.23.1 change
BPE GPT2 encode batch, no cache 530 ms 446 ms −16%
BPE GPT2 encode batch (cached) 690 ms 685 ms noise
BPE GPT2 encode (single) 1.95 s 1.94 s noise
BPE Train (small) 32.6 ms 31.5 ms −3%
BPE Train (big) 1.01 s 988 ms −2%

The BPE per-thread cache PR (#2028) shows much larger wins on highly-parallel workloads (+47–62% at 88+ threads on a server box, per the PR's own measurements on Vera). Single-thread batch numbers above are flat or slightly improved because cache-hit overhead was already low without contention.

Llama-3 encode

... (truncated)

Commits

Updates `fastapi` from 0.136.3 to 0.141.1
Release notes

Sourced from fastapi's releases.

0.141.1

Fixes

  • 🐛 Fix support for background tasks and headers from dependencies in app.frontend(). PR #16105 by @​tiangolo.

Docs

0.141.0

Features

  • ✨ Add app.frontend(check_dir="auto"), to make local development more convenient with fastapi dev. PR #16102 by @​tiangolo.

0.140.13

Fixes

Docs

0.140.12

Fixes

0.140.11

Fixes

  • 🐛 Fix response_model_* params ignored for non-generator endpoints with Iterable[..] return type. PR #15093 by @​YuriiMotov.

0.140.10

Fixes

Internal

0.140.9

Fixes

  • 🐛 Fix exclude_defaults not propagated to dict keys and values in jsonable_encoder. PR #16043 by @​MBGrao.

... (truncated)

Commits
  • 95f8322 🔖 Release version 0.141.1 (#16106)
  • f137944 📝 Update release notes
  • d623544 🐛 Fix support for background tasks and headers from dependencies in `app.fron...
  • 1d211b9 📝 Update release notes
  • 8a1f876 📝 Document FASTAPI_ENV in FastAPI CLI guide (#16104)
  • c7e7b65 🔖 Release version 0.141.0 (#16103)
  • 6bceb84 📝 Update release notes
  • 5429fed ✨ Add app.frontend(check_dir="auto"), to make local development more conven...
  • 628663f 🔖 Release version 0.140.13 (#16096)
  • 0b54fd0 📝 Update release notes
  • Additional commits viewable in compare view

Updates `starlette` from 1.3.1 to 1.6.0
Release notes

Sourced from starlette's releases.

Version 1.6.0

What's Changed

New Contributors

Full Changelog: https://github.com/Kludex/starlette/compare/1.5.1...1.6.0

Version 1.5.1

What's Changed

Full Changelog: https://github.com/encode/starlette/compare/1.5.0...1.5.1

Version 1.5.0

This release is all about giving GZipMiddleware some love. 🗜️

What's Changed

Full Changelog: https://github.com/encode/starlette/compare/1.4.1...1.5.0

Version 1.4.1

What's Changed

Full Changelog: https://github.com/Kludex/starlette/compare/1.4.0...1.4.1

Version 1.4.0

What's Changed

New Contributors

Full Changelog: https://github.com/Kludex/starlette/compare/1.3.1...1.4.0

Changelog

Sourced from starlette's changelog.

1.6.0 (August 8, 2026)

Added

  • Add max_body_size to Starlette and route classes #3431.
  • Expose http.response.debug information via response extensions #3130.

1.5.1 (August 8, 2026)

Fixed

  • Reject inverted single-byte ranges in FileResponse #3389.
  • Limit FileResponse to 100 ranges #3430.

1.5.0 (August 8, 2026)

Added

  • Add exclude_content_types parameter to GZipMiddleware #3418.

Changed

  • Expand default excluded content types in GZipMiddleware #3421.

Fixed

  • Flush GZip output for each streamed chunk #3419.
  • Skip compression of partial responses in GZipMiddleware #3420.

1.4.1 (August 5, 2026)

Fixed

  • Default thread_minimum_size to 128 KiB in GZipResponder, keeping it usable without the new keyword argument #3415.

1.4.0 (August 5, 2026)

Added

  • Offload large GZip compression to a worker thread, keeping the event loop responsive. GZipMiddleware accepts a new thread_minimum_size parameter (default 128 KiB) controlling the minimum body chunk size compressed in a thread #3410.

Changed

  • Use zlib.compressobj instead of GzipFile in GZipMiddleware, reducing memory usage during compression #3411.
  • Lazily allocate GZipMiddleware compression resources, avoiding compressor allocation for responses that are never compressed #3407.
Commits
  • 4f250d6 Version 1.6.0 (#3434)
  • 9eea41a Expose http.response.debug info via response extensions (#3130)
  • 38f8999 Add max_body_size to Starlette and route classes (#3431)
  • c41236c Version 1.5.1 (#3432)
  • 9c500db Limit FileResponse to 100 ranges (#3430)
  • 78ae82c Reject inverted single-byte Range like bytes=5-4 (#3389)
  • c1d6eda chore(deps): bump pymdown-extensions from 11.0 to 11.0.1 (#3429)
  • ee66ca4 chore(deps): bump the python-packages group across 1 directory with 8 updates...
  • 00d1016 fix(tests): skip test_staticfiles_filename_too_long on Windows where os.pathc...
  • d96887e Add Pydantic Logfire banner to the docs (#3428)
  • Additional commits viewable in compare view

Updates `aiohttp` from 3.13.3 to 3.14.3 Updates `openai` from 2.44.0 to 2.53.0
Release notes

Sourced from openai's releases.

v2.53.0

2.53.0 (2026-08-03)

Features

  • api: Add gpt-5.5 and tool name/namespace to Responses types (#3569) (dd1202d)

Bug Fixes

  • ci: avoid NumPy source builds and duplicate HTTPX coverage (#3573) (b58332f)

v2.52.1

2.52.1 (2026-07-31)

Full Changelog: v2.52.0...v2.52.1

Chores

  • ci: pin setup-uv v5 to its underlying commit (#3560) (cbdc98b)

v2.52.0

2.52.0 (2026-07-31)

Full Changelog: v2.51.0...v2.52.0

Features

  • api: content provenance checks (1d6c118)

Bug Fixes

  • client: honor Retry-After delays up to two minutes (#3555) (7fa7946)

Documentation

v2.51.0

2.51.0 (2026-07-30)

Full Changelog: v2.50.0...v2.51.0

Features

... (truncated)

Changelog

Sourced from openai's changelog.

2.53.0 (2026-08-03)

Features

  • api: Add gpt-5.5 and tool name/namespace to Responses types (#3569) (dd1202d)

Bug Fixes

  • ci: avoid NumPy source builds and duplicate HTTPX coverage (#3573) (b58332f)

2.52.1 (2026-07-31)

Full Changelog: v2.52.0...v2.52.1

Chores

  • ci: 1` and writes unreduced rows + reduces in Python (the same path already taken when a per-expert bias is present); TP1 keeps the fused fast path. Verified: the same model garbled at TP2 before this and is correct after. ## How fast it is All runs are on an RX 7900 XTX (gfx1100), ROCm 7.2.3, bf16, cudagraph on. Numbers are `vllm bench serve`, random 512-in / 256-out, `--ignore-eos` — output token throughput in tok/s. **GPT-OSS-20B (single GPU), native HIP MoE vs the Triton-unfused path it replaces.** This is the apples-to-apples one, since GPT-OSS is the only model here that *has* a ROCm fallback to compare against: | concurrency | Triton | HIP | speedup | |---|---|---|---| | 1 | 8.5 | 48.8 | 5.7× | | 8 | 82.1 | 206.8 | 2.52× | | 16 | 147.0 | 241.8 | 1.64× | | 32 | 230.0 | 382.6 | 1.66× | Single-stream latency (`vllm bench latency`, 128/256, batch 1) drops from 13.15 s to 2.43 s — about 5.4×. **`amd/Qwen3.5-35B-A3B-MXFP4` (AMD Quark, W4A4, 256-expert MoE, TP2)** — the second apples-to-apples case: this W4A4 checkpoint runs weight-only on gfx1100, native HIP vs the Triton-unfused fallback it otherwise gets. tok/s (mean TPOT ms): | concurrency | TRITON_UNFUSED | native RDNA3 | speedup | |---|---|---|---| | 1 | 26.6 (36.8) | 58.9 (14.9) | 2.21× | | 8 | 177.6 (42.1) | 255.6 (27.5) | 1.44× | | 16 | 279.9 (53.1) | 396.3 (36.4) | 1.42× | | 32 | 407.7 (72.7) | 559.0 (51.4) | 1.37× | **`pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4` (TP2)** — a real compressed-tensors MoE checkpoint (quantized from `Qwen/Qwen3.6-35B-A3B`). There's no baseline to compare against here because the model simply won't load on gfx1100 without these kernels: | concurrency | tok/s | mean TPOT (ms) | |---|---|---| | 1 | 63.3 | 15.3 | | 8 | 303.9 | 23.9 | | 16 | 434.7 | 33.0 | | 32 | 613.1 | 45.5 | **`kaitchup/Qwen3.5-27B-MXFP4A16` (TP2)** — a real third-party *dense* MXFP4 checkpoint, again with no ROCm baseline: | concurrency | tok/s | mean TPOT (ms) | |---|---|---| | 1 | 32.8 | 28.8 | | 8 | 127.4 | 55.0 | | 16 | 192.5 | 69.8 | | 32 | 251.0 | 104.4 |", "url": "https://github.com/vllm-project/vllm/pull/46676", "createdAt": "2026-06-25T01:41:16Z", "updatedAt": "2026-08-13T12:41:04Z", "timestamp": "2026-08-13T12:41:04Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "rocm", "ci/build", "gpt-oss", "quantization" ], "author": "JartX", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:d1aa876d9446fe4fdfc7", "signalId": "github:vllm-project/vllm:pull_request:50534", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50534", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B70", "text": "## Purpose Add tuned `selective_state_update` configs for **Intel Arc Pro B70 Graphics**, and fix `benchmarks/kernels/benchmark_selective_state_update.py` so it runs on an XPU-only build. The config directory holds AMD and NVIDIA devices only, so Mamba/hybrid models on Intel GPUs fall back to the heuristic. Four files, two shapes: | shape | cache_dtype | exercised by | | --- | --- | --- | | `headdim=128,dstate=256` | float32 + float16 | FalconH1 | | `headdim=64,dstate=128` | float32 + float16 | Mamba2 | `headdim=64,dstate={16,32,64,256}` was tuned and validated too, but no model on hand exercises those shapes, so per review they are dropped rather than shipped without E2E verification — they keep using the heuristic as they do today. - **`float16` is tuned, not copied from `float32`.** Without `--mamba-ssm-cache-dtype`, vLLM resolves `\"auto\"` → model dtype → the `float16` file. On a bfloat16 state the optimum differs at 9 of 12 grid points for `hd128 d256`; reusing `float32` values costs a median 79.6% at kernel level and is 1.10x *slower* than the heuristic at one point on `hd64 d128`. - **Grid starts at `effective_batch=8`.** Mamba2-2.7b at `concurrency=1` runs batch 1 × 80 heads; a grid starting at 128 was ~10% slower than the heuristic there. - **`hd64 d128` keeps some incumbent entries.** At `effective_batch=4096` the candidates were 0.1% apart in the microbenchmark and picking the winner cost 4.2% E2E — the tuner runs `ngroups=8`, models run `ngroups=1`. Re-tuned entries adopted only above a 10% margin. Script fixes, all needed on XPU: - `torch.cuda.get_device_capability()` → `current_platform.get_device_capability()` (`None` on XPU). Called unconditionally today, it raises `AssertionError: Torch not compiled with CUDA enabled` before tuning starts; also covers ROCm. - `torch.cuda.CUDAGraph()`/`torch.cuda.graph()` → `torch.accelerator.Graph()`. - `torch.cuda.Event` → `torch.Event`, as in `benchmark_moe.py`. The last two otherwise raise inside a `try/except` returning `None`, so every candidate is skipped and the sweep yields no timings. Note that `torch.accelerator.Graph()` is not used elsewhere in the repo — worth a check on CUDA. ## Test Plan Intel Arc Pro B70 (31.9 GiB), Triton 3.7.1, single card. Kernel sweep per shape and cache dtype; numerical validation against the CPU reference under each model's real `ngroups`/`nheads` with the heuristic as control; E2E `vllm bench serve` with a separate server launch and warm-up per measurement, on `tiiuae/Falcon-H1-7B-Instruct` and `AntonV/mamba2-1.3b-hf`. ## Test Result Kernel, speedup vs heuristic: | shape | 8 | 32 | 128 | 512 | 2048 | 16384 | peak | | --- | --- | --- | --- | --- | --- | --- | --- | | `hd128 d256` | 1.56x | 2.83x | 4.15x | 1.86x | 1.96x | 1.99x | **4.47x** @ 64 | | `hd64 d128` | 1.10x | 1.33x | 2.33x | 3.18x | 1.30x | 1.30x | **3.18x** @ 512 | E2E `in=128 out=1024`, tuned vs heuristic, explicit `--mamba-ssm-cache-dtype float32`: | model | conc | eff_batch | TPOT tuned | TPOT heur | TPOT gain | throughput gain | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | FalconH1-7B | 1 | 24 | 28.78 ms | 29.55 ms | 1.027x | 1.022x | | FalconH1-7B | 8 | 192 | 33.03 ms | 39.13 ms | 1.185x | 1.184x | | FalconH1-7B | 16 | 384 | 38.60 ms | 48.70 ms | 1.262x | 1.255x | | FalconH1-7B | 32 | 768 | 50.12 ms | 68.51 ms | 1.367x | **1.362x** | | FalconH1-7B | 64 | 1536 | 56.82 ms | 82.69 ms | **1.455x** | 1.213x | | Mamba2-1.3b | 1 | 64 | 15.00 ms | 15.36 ms | 1.023x | 1.021x | | Mamba2-1.3b | 8 | 512 | 15.26 ms | 15.70 ms | 1.029x | 1.029x | | Mamba2-1.3b | 16 | 1024 | 15.53 ms | 16.59 ms | 1.068x | 1.072x | | Mamba2-1.3b | 32 | 2048 | 19.00 ms | 25.01 ms | **1.316x** | **1.313x** | | Mamba2-1.3b | 64 | 4096 | 32.46 ms | 41.79 ms | 1.288x | 1.285x | FalconH1 throughput saturates by `concurrency=64` (631.5 tok/s at 32 vs 624.5 at 64), so it diverges from TPOT. On a `1024/256` shape: FalconH1 1.026x → 1.352x TPOT, Mamba2-1.3b 1.256x at 32 but 0.988x / 0.990x at 1 / 8 — ~1%, below the 1.5% threshold and single-shot. On the default path (no flag, so the `float16` file is loaded), tuning that file beats copying the `float32` values into it by 5.6–5.7% for FalconH1 at `concurrency` 16 and 64, and by 2.6% for Mamba2-1.3b at 128 — interleaved A/B with two warm-ups, run-to-run spread 0.02–0.23 ms. A null control at `concurrency=8`, where both variants resolve to the same entry, landed inside that spread. Validation passes 8/8 and 7/7, control clean. Mamba2-1.3b measured on two B70 machines with two vLLM builds; high-gain results agree to three decimals.", "url": "https://github.com/vllm-project/vllm/pull/50534", "createdAt": "2026-07-31T06:13:18Z", "updatedAt": "2026-08-13T12:36:56Z", "timestamp": "2026-08-13T12:36:56Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "performance", "intel-gpu", "ready", "verified" ], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:eb2c8dd612a6700f41dd", "signalId": "github:vllm-project/vllm:pull_request:48684", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48684", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Remove `override_attention_dtype`", "text": "## Purpose `override_attention_dtype` is only used for V0 and has been removd from https://github.com/vllm-project/vllm/pull/25351/ long time ago. It's safe to remove it now. ## Test Plan ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/48684", "createdAt": "2026-07-15T02:47:30Z", "updatedAt": "2026-08-13T12:32:57Z", "timestamp": "2026-08-13T12:32:57Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready" ], "author": "wangxiyuan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8376334a5463d0e0a23a", "signalId": "github:vllm-project/vllm:pull_request:51168", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Enable Speculative Decoding for NVIDIA-Nemotron-Parse-2.0", "text": "co autored with: @DenisOvchinnikov93 [NVIDIA-Nemotron-Parse-2.0](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0) ships: > Auxiliary Prediction Head: One training-time decoder prediction head is preserved separately in auxiliary_prediction_heads.safetensors.extra for future multi-token prediction research. Standard generation uses the tied decoder input/output embeddings; the default model.safetensors, Transformers examples, and vLLM examples do not load this auxiliary head. This PR leverages the auxiliary head to enable Speculative Decoding for Nemotron-Parse. Backward compatibility for older Nemotron-Parse versions is preserved (standard single-token decoding is only available for versions < 2.0). This PR also switches tests to track regressions on the latest model with speculative-decoding enabled. To measure throughput with vs without speculative decoding, I used ParseBench table and text_content (1,005 pages). That census has long target sequences, so MTP gains show up clearly. On H100, speculative-decoded vs baseline median output-throughput uplift was: Concurrency | speculative-decoding throughput gain vs single-token-prediction -- | -- 1 | 45.65% 8 | 40.85% 32 | 39.60% ## Purpose Enable spec-dec for nemotron-parse 2.0 ## Test Plan ``` .venv/bin/python -m pytest \\ tests/models/multimodal/generation/test_nemotron_parse.py \\ -vv --tb=short ``` ## Test Result ``` ================== 1 passed, 32 warnings in 74.73s (0:01:14) =================== ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51168", "createdAt": "2026-08-05T14:27:22Z", "updatedAt": "2026-08-13T12:32:26Z", "timestamp": "2026-08-13T12:32:26Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "multi-modality", "nvidia", "mrv2" ], "author": "mwawrzos", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8c5e70b5e25e99a51e55", "signalId": "github:vllm-project/vllm:pull_request:52162", "event": "discovered", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52162", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][PCP] Shard decode requests across PCP ranks", "text": "## Purpose PCP currently replicates every decode request on every PCP rank. For PCP-only deployments (`DCP == 1`), every rank already has a complete KV-cache replica, so this repeats the same decode computation `pcp_size` times. This PR: - assigns decode-only rows round-robin across PCP ranks on every scheduler step; - keeps the existing DualChunkSwap prefill partitioning unchanged; - gathers the owner-produced MLA and sparse-indexer KV updates so every PCP rank retains a complete cache replica; - restores hidden states in global batch order through the existing PCP gather; - preserves replicated decode when `DCP > 1`, because each DCP rank must consume its local KV shard. There is no new user-facing flag or API. Documentation is unchanged because this is an internal execution optimization with an automatic `DCP == 1` safety gate. ### Duplicate-work check I searched open PCP/decode issues and PRs before submission. This does not duplicate: - #49564, which adds FlashAttention/GQA PCP and PCP+DCP prefill support but does not shard PCP-only decode requests; - #49517, which replaces replicated KV-update collectives with opt-in, model-specific direct peer stores but does not reduce replicated decode computation; - #49741, which owner-shards persistent KV history for capacity rather than sharding decode compute. Related performance tracking: #31473. ### AI assistance OpenAI Codex was used to analyze, implement, test, benchmark, rebase, and prepare this draft. The commit includes an AI co-author trailer. This is intentionally a draft so the human submitter can review every changed line and the evidence before marking it ready. ## Test Plan Focused checks: ```bash uvx ruff==0.14.0 check \\ tests/v1/worker/test_gpu_pcp_manager.py \\ vllm/model_executor/layers/attention/mla_attention.py \\ vllm/model_executor/layers/attention/pcp.py \\ vllm/model_executor/layers/sparse_attn_indexer.py \\ vllm/v1/worker/gpu/pcp_manager.py uvx ruff==0.14.0 format --check \\ tests/v1/worker/test_gpu_pcp_manager.py \\ vllm/model_executor/layers/attention/mla_attention.py \\ vllm/model_executor/layers/attention/pcp.py \\ vllm/model_executor/layers/sparse_attn_indexer.py \\ vllm/v1/worker/gpu/pcp_manager.py .venv/bin/python -m pytest -q \\ --confcutdir=tests/v1/worker \\ tests/v1/worker/test_gpu_pcp_manager.py ``` The focused tests cover: - unique, balanced PCP-only decode ownership; - DCP fallback to replicated decode; - unchanged prefill partitioning in mixed batches; - gathered KV/slot mapping and hidden-state restoration order. End-to-end validation used 8x RTX 5090 with DeepSeek-V2-Lite-Chat: - functionality: concurrency 256, 4,096 output tokens, two rounds; - accuracy: full GSM8K test set, 1,319 examples, 5-shot, temperature 0, seed 42; - performance: 100K input / 1K output, 99% prefix-cache hit, concurrency 32, 128 requests, three retained PCP baseline/candidate runs. ## Test Result ### Current rebased HEAD - Rebased cleanly onto `upstream/main@5fee0a872d`. - `ruff check`: passed. - `ruff format --check`: 5 files already formatted. - `git diff --check upstream/main...HEAD`: passed. - Diff scope: 5 files, 219 insertions, 8 deletions. - The focused pytest suite passed `4 passed` on the pre-rebase candidate. It has not yet been rerun after the final rebase. ### Functionality and model evaluation All TP8, baseline PCP8, and candidate PCP8 functional runs completed: - `512/512` requests per state; - every request generated 4,096 tokens; - 2,097,152 output tokens per state; - zero errors and zero aborted requests. GSM8K over three runs: | State | Correct answers | Mean accuracy | Std. dev. | |---|---:|---:|---:| | Baseline PCP8 | 852, 852, 851 | 0.645691 | 0.000438 | | Candidate PCP8 | 844, 853, 869 | 0.648471 | 0.009600 | There is no stable accuracy decrease, but the candidate variance is materially higher. Accuracy is therefore classified as **inconclusive / nondeterministic**, not passed. ### PCP decode performance Each retained measurement completed 128/128 requests with exactly 1,024 output tokens and a measured prefix-cache token hit rate of 98.993%. | Metric | Baseline PCP8 | Candidate PCP8 | Change | |---|---:|---:|---:| | Request throughput | 0.3810 req/s | 0.5102 req/s | +33.91% | | Output throughput | 390.15 tok/s | 522.45 tok/s | +33.91% | | Total-token throughput | 39,407.53 tok/s | 52,771.39 tok/s | +33.91% | | Mean TTFT | 11,153 ms | 11,328 ms | +1.57% | | Mean TPOT | 71.16 ms | 50.20 ms | -29.46% | | Mean ITL | 71.62 ms | 50.54 ms | -29.42% | Throughput CV was 0.25% for baseline and 0.34% for candidate. A candidate-only 100K/1-token guardrail measured TP4+EP4 at 4,887.26 ms and PCP4+EP4 at 5,019.78 ms (+2.71% PCP TTFT). The multi-GPU results above were collected before the final rebase. The runtime diff applied cleanly to current main and current-head static checks pass, but the full multi-GPU matrix has not yet been repeated on `0b478f031e`. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52162", "createdAt": "2026-08-13T12:29:43Z", "updatedAt": "2026-08-13T12:30:22Z", "timestamp": "2026-08-13T12:30:22Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "mrv2" ], "author": "pisceskkk", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:752beb210cdc636137cc", "signalId": "github:vllm-project/vllm:pull_request:52033", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52033", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][ROCm] Dual-stream decode with hipgraphs", "text": "## Changes since reverted in https://github.com/vllm-project/vllm/pull/52024 Re-opened https://github.com/vllm-project/vllm/pull/48223 with fix for Qwen3.5-style models (CI failed test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] after merging this PR previously). **Changes made since revert:** https://github.com/vllm-project/vllm/pull/52033/changes/50e80bb721d43c0d032d0a71d44546e3cdedcc6c. Issue stemmed from fact that Qwen3.5 uses the same tensor for hidden state and shared expert inputs, and later on mutates one of them in the routed experts. That causes a race condition during multi-stream. **Validation:** Re-ru nthe test with this config: https://github.com/vllm-project/vllm/blob/fe889ac9255491e82e58012d7787e920c6c7fbc2/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml#L6-L9 ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --max-model-len 4096 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --trust-remote-code \\ --disable-uvicorn-access-log ``` ``` docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c \"pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-35B-A3B,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k\" ``` **Results:** Running GSM8k on Qwen3.5 on MI350 with DP2EP gives (test pass requires acc >= 0.84 - 0.03 = 0.81) |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8522|± |0.0098| | | |strict-match | 5|exact_match|↑ |0.8362|± |0.0102| _From original PR:_ ## Purpose Fixes #48111. Enables (1) dual-stream decode for CUDA-like platforms with proper overlap (2) make them hip/cudagraph compatible. Only enabled on ROCm when using DP, as we observed [performance regression under TP](https://github.com/vllm-project/vllm/pull/48223#issuecomment-4993855510). Mutually exclusive with `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS`. Disable with `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` as usual. **Gain:** About -3-4% TPOT on 1k/1k and 8k/1k when using DPA vs default. **Note:** When Fused Shared experts (FSE) can be used, it provides even better TPOT than dual-stream. However, FSE cannot be used everywhere (like with MoRI a2a kernels). ### Technical details - Simply allowing `MULTI_STREAM_OVERLAPPED` for CUDA-like platforms is not sufficient for performant dual-stream decode (on ROCm, at least), because the shared expert was launched on the aux stream only _after_ the routed experts had completed. That leads to a sequential launch with no overlap: \"image\" - Instead, we need to launch the shared expert in the aux stream _before_ the dispatch. #### After this PR we get the following traces - Pure decodes: properly overlapped \"image\" - Mixed batches: no aux stream \"image\" - Pure prefills: no aux stream \"image\" ## Test Plan Tested on 8xMI300. 1. Run DSv3 with DPA+TP: ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` 2. Bench serve sweep, 1k/1k and 8k/1k ISL/OSL - 10x as many prompts as the concurrency level; 2x number of warmups. - Comparing to nightly `vllm/vllm-openai-rocm:nightly-a16dbd5b8572d4128be9f10b9dcff4999b594b25` with fused shared experts ```bash vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 \\ --num-warmups 256 ``` 4. GSM8k accuracy non-regression ```bash lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result - Couple % improved TPOT ### 1k/1k (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 232.57 | 23.67 | **+4.24%** | | 1 | nightly | 0.04 | 273.16 | 24.72 | - | | 2 | this branch | 0.08 | 258.38 | 24.16 | **+4.58%** | | 2 | nightly | 0.08 | 298.14 | 25.32 | - | | 4 | this branch | 0.15 | 397.90 | 25.52 | **+7.03%** | | 4 | nightly | 0.14 | 366.23 | 27.45 | - | | 8 | this branch | 0.25 | 602.93 | 30.84 | **+3.41%** | | 8 | nightly | 0.24 | 664.92 | 31.93 | - | | 16 | this branch | 0.44 | 1268.65 | 35.02 | -6.77% | | 16 | nightly | 0.46 | 851.06 | 32.80 | - | | 32 | this branch | 0.71 | 1830.04 | 42.03 | **+2.75%** | | 32 | nightly | 0.70 | 1698.74 | 43.22 | - | | 64 | this branch | 1.34 | 3228 | 43.51 | **+4.71%** | | 64 | nightly | 1.29 | 2528 | 45.66 | - | | 128 | this branch | 2.15 | 4487.07 | 53.85 | **+1.86%** | | 128 | nightly | 2.10 | 4543.42 | 54.87 | - | | 256 | this branch | 3.46 | 6554.86 | 65.74 | **+3.12%** | | 256 | nightly | 3.37 | 6386.54 | 67.86 | - | Note: TTFT numbers seems highty variable. Ran 64 concurrency yet another time, and then I got 1902 ms TTFT @ 1.37 QPS, substantially better than the first time. This PR should not affect prefills anyhow so not focusing too much on TTFT here anyways. ### 8k/1k | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 1868.16 | 25.22 | **+3.59%** | | 1 | nightly | 0.04 | 1860 | 26.16 | - | | 2 | this branch | 0.07 | 2034.06 | 25.42 | **+5.47%** | | 2 | nightly | 0.07 | 2014.35 | 26.89 | - | | 4 | this branch | 0.11 | 3800.62 | 31.35 | **+2.82%** | | 4 | nightly | 0.11 | 3687 | 32.26 | - | | 8 | this branch | 0.19 | 4105.90 | 37.01 | **+5.54%** | | 8 | nightly | 0.18 | 4122.95 | 39.18 | - | | 16 | this branch | 0.31 | 6355 | 45.37 | -3.07% | | 16 | nightly | 0.31 | 6321 | 44.02 | - | | 32 | this branch | 0.45 | 9616.31 | 60.48 | **+11.44%** | | 32 | nightly | 0.41 | 9173.59 | 68.29 | - | | 128 | this branch | 0.85 | 15579.93 | 131.24 | -3.71% | | 128 | nightly | 0.89 | 13966.30 | 126.55 | - | ### GSM8k This branch: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9416|± |0.0065| | | |strict-match | 5|exact_match|↑ |0.9409|± |0.006| Nightly: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52033", "createdAt": "2026-08-12T18:30:25Z", "updatedAt": "2026-08-13T12:29:35Z", "timestamp": "2026-08-13T12:29:35Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm" ], "author": "simondanielsson", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:d1cf716247e04de7a5d3", "signalId": "github:vllm-project/vllm:pull_request:43091", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Support spec decode with draft model", "text": "## Purpose Support spec decode with draft models in model runner v2 ## Test Plan ``` python examples/features/speculative_decoding/spec_decode_offline.py \\ --method draft_model \\ --model-dir /shared/models/modelscope/models/Qwen/Qwen3-32B \\ --draft-model /shared/models/modelscope/models/Qwen/Qwen3-1.7B \\ --num-spec-tokens 3 \\ --dataset-name hf \\ --dataset-path philschmid/mt-bench \\ --num-prompts 400 \\ --temp 0 \\ --output-len 256 \\ --max-model-len 10000 ``` ## Test Result **Model Runner V1:** ``` -------------------------------------------------- total_num_output_tokens: 102355 num_drafts: 36593 num_draft_tokens: 109779 num_accepted_tokens: 65181 mean acceptance length: 2.78 -------------------------------------------------- acceptance at token 0: 0.75 acceptance at token 1: 0.57 acceptance at token 2: 0.45 ``` **Model Runner V2:** ``` -------------------------------------------------- total_num_output_tokens: 102312 num_drafts: 36394 num_draft_tokens: 109182 num_accepted_tokens: 65379 mean acceptance length: 2.80 -------------------------------------------------- acceptance at token 0: 0.76 acceptance at token 1: 0.58 acceptance at token 2: 0.46 ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/43091", "createdAt": "2026-05-19T11:35:21Z", "updatedAt": "2026-08-13T12:26:46Z", "timestamp": "2026-08-13T12:26:46Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "ready", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "wxsIcey", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7f9eb3b45ff4aa44a87a", "signalId": "github:vllm-project/vllm:pull_request:51350", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51350", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "support rl feature : weight checker", "text": "## Purpose This PR extends the POST /weight_checker test suite so the weight-checker flow is exercised under both single-GPU (TP=1) and tensor-parallel (TP=2) configurations)Skip all cases now run for both TP settings via pytest.mark.parametrize. The coverage includes snapshot, compare, checksum, reset, and invalid-request error handling, plus a new assertion that the server reports the expected world_size for each TP configuration. ## Test Plan A single parameterized test file drives the dev server for each TP setting and verifies: -snapshot returns a structured response with a positive tensor count, and repeated snapshots are consistent. -compare requires a prior snapshot (400 otherwise), matches when weights are unchanged, and still matches after a sleep/wake cycle and after a generate request. -checksum returns per-tensor 64-character SHA-256 digests, is stable across calls, and does not clobber an existing snapshot. -reset rewrite the model weight to check update is correct. -error handling rejects missing/unknown actions cleanly while /health stays up. -world size matches the configured tensor_parallel_size. | Priority| Test Case | Main Coverage | | --- | --- | --- | | P0 | test_compare_without_snapshot_returns_400 | 400 when comparing without a snapshot; server stays healthy | | P0 | test_snapshot_then_compare_matches | Full match after snapshot (match=True, no mismatches) | | P0 | test_world_size_matches_tp | Server world_size equals the target TP (1/2) | | P0 | test_snapshot_returns_n_tensors | snapshot returns positive integer n_tensors, status=snapshotted | | P0 | test_compare_is_one_shot | compare is one-shot: a second compare (no new snapshot) returns 400 | | P1 | test_snapshot_twice_overwrites | Repeated snapshot overwrites the old one; n_tensors stays consistent | | P1 | test_checksum_returns_dict_of_hex | checksum returns a dict of 64-char hex SHA-256 digests | | P1 | test_checksum_stable_across_calls | checksum is idempotent/stable while weights are unchanged | | P1 | test_checksum_does_not_affect_snapshot | checksum does not write to or corrupt the stored snapshot | | P1 | test_sleep_wake_does_not_change_weights | Weights unchanged across sleep/wake (compare still matches) | ## Test Result default test tp=1 and 2 VLLM_SERVER_DEV_MODE=1 python -m pytest tests/entrypoints/serve/dev/test_weight_checker.py -v result： ```bash 52.35s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 35.10s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 4.07s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 4.06s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 3.47s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 2.95s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 2.83s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 2.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 2.71s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 2.66s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 2.58s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] 2.57s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 2.22s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 1.96s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 1.88s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 1.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 1.73s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 1.70s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 1.40s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 1.37s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 1.35s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 1.33s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 1.33s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 1.03s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.84s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.48s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.37s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 0.32s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.28s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[2] 0.27s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 0.27s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 0.25s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.24s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.24s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 0.19s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.17s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[2] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[2] ``` or you can test tp =1 faster VLLM_TEST_TP=1 python -m pytest tests/entrypoints/serve/dev/test_weight_checker.py result ```bash 36.11s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 3.94s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 3.93s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 3.37s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 2.87s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 2.74s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 2.64s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 2.64s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 2.63s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 1.40s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 1.35s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 1.32s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 1.29s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.45s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.34s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_without_snapshot_returns_400[1] 0.27s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.25s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.23s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.22s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.21s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.20s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.19s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.18s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.17s teardown tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.01s call tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_twice_overwrites[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_returns_n_tensors[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_world_size_matches_tp[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_changes_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_does_not_affect_snapshot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_compare_is_one_shot[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_stable_across_calls[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_checksum_returns_dict_of_hex[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_missing_action_returns_400[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_sleep_wake_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_generate_does_not_change_weights[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightChecker::test_snapshot_then_compare_matches[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerReset::test_reset_is_idempotent[1] 0.00s setup tests/entrypoints/serve/dev/test_weight_checker.py::TestWeightCheckerErrors::test_unknown_action_returns_400[1] ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51350", "createdAt": "2026-08-07T03:32:38Z", "updatedAt": "2026-08-13T12:24:41Z", "timestamp": "2026-08-13T12:24:41Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "frontend", "needs-rebase" ], "author": "shiyuan680", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:911d158a5a0e1f5bdefd", "signalId": "github:vllm-project/vllm:pull_request:50174", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50174", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup", "text": "## Description This PR extends the shared JIT warmup infrastructure with provider registration and centralized orchestration. It builds on #49315 and the contract described in #47456. For more details, see parent (draft) PR: https://github.com/vllm-project/vllm/pull/49627 and tracking list issue https://github.com/vllm-project/vllm/issues/49349 ``` JIT kernel warmup (5 compile keys): 100%|██████████████████████████████████| 1/1 [00:00<00:00, 259.93kernel/s, ComputeSlotMappingKernel (5 keys)] (Worker_TP0 pid=425015) INFO 08-11 15:43:58 [kernel_warmup.py:120] JIT kernel warmup finished in 0.01s. ``` ## Motivation - Discover warmup providers from the model and backend objects actually instantiated. - Avoid model-name-based lists and warming kernels for inactive backends. - Run compilation under `kernel_warmup()` logging, ordering, progress reporting, and exception handling. - Respect `enable_jit_warmup`. - Keep model construction and runtime execution separate from startup compilation. ## What Changed - Added a scoped registry used while model and KV-cache components are constructed. - Added centralized provider deduplication and warmup execution. - Added kernel and compile-key progress reporting. - Added shared Triton helpers and an initial block-table migration. - Added contributor documentation for the warmup contract. - Added registry, orchestration, and AST-expansion tests.", "url": "https://github.com/vllm-project/vllm/pull/50174", "createdAt": "2026-07-28T19:24:41Z", "updatedAt": "2026-08-13T12:23:04Z", "timestamp": "2026-08-13T12:23:04Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "v1", "cpu", "mrv2" ], "author": "LopezCastroRoberto", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0d49e8926cfd69a6439e", "signalId": "github:vllm-project/vllm:pull_request:45899", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45899", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit tests for the chunked scaled-dot KKT kernels", "text": "## Purpose Adds unit tests for both chunked scaled-dot K@K^T kernels in `tests/kernels/test_chunk_scaled_dot_kkt.py`, neither of which had coverage: - `chunk_scaled_dot_kkt_fwd` (GDN/FLA) — scalar gate, also serves grouped K heads - `chunk_kda_scaled_dot_kkt_fwd` (KDA) — per-channel gate, also returns the non-strictly causal `Aqk` ## Test Plan 16 cases (8 per kernel, bfloat16 and float32) against a chunk-wise PyTorch reference, covering partial chunks, gated and ungated paths, GQA head expansion and ragged varlen input. Sequences are laid out either as a padded batch or as a packed varlen run, so both encodings meet the same reference. Inputs follow the production layers: l2-normalised q/k, fp32 sigmoid `beta`, and a `chunk_local_cumsum` log-gate damped so the causal block does not underflow to zero. ## Test Result All 16 pass on IntelB70 and H200", "url": "https://github.com/vllm-project/vllm/pull/45899", "createdAt": "2026-06-17T06:38:34Z", "updatedAt": "2026-08-13T12:22:21Z", "timestamp": "2026-08-13T12:22:21Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:083e306ac1d42a83d5c9", "signalId": "github:vllm-project/vllm:pull_request:48712", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48712", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Only run FP8 AITER MLA prefill when using FP8 KV", "text": "## Purpose Even when KV is not FP8, the FP8 PS ASM prefill kernel's persistent buffers are both (a) initialized/reserved and (b) filled with metadata (through `_build_fp8_prefill_ps_metadata`) in every forward. Specifically the latter leads to performance degradation when using BF16 as `_build_fp8_prefill_ps_metadata` is a host-side function with several sync memcpys in it, causing a unnecessary H->D sync before every forward. Co-authored with @olka-amd. ### Before: big bubble in start of every forward, even when using BF16 \"image\" ### After: bubble gone \"image\" ## Test Plan 1. Runs as expected on gsm8k with BF16 KV ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` ``` lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result This branch: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9466|± |0.0065| | | |strict-match | 5|exact_match|↑ |0.9469|± |0.006| Nightly: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/48712", "createdAt": "2026-07-15T07:24:10Z", "updatedAt": "2026-08-13T12:20:28Z", "timestamp": "2026-08-13T12:20:28Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "bug", "rocm", "v1" ], "author": "simondanielsson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:941fd03634356d610d24", "signalId": "github:vllm-project/vllm:issue:51873", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51873", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable logprobs with adaptive verification", "text": "### 🚀 The feature, motivation and pitch Follow-up leftover from #47808. See: https://github.com/vllm-project/vllm/pull/47808#discussion_r3723768178 ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51873", "createdAt": "2026-08-11T20:26:26Z", "updatedAt": "2026-08-13T12:19:43Z", "timestamp": "2026-08-13T12:19:43Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "benchislett", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:433bc64b0d90801f2a85", "signalId": "github:vllm-project/vllm:pull_request:51692", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51692", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM", "text": "## Purpose Activated when shapes allow for it and configs are tuned. **Implications (DSv3 1k/1K):** - TP8+DPA: +4-8% QPS - TP8+EP: +0-4% QPS ## Test Plan Bench serve & accuracy validation with DSv3 on (1) TP8+DPA (2) TP8+EP on 8xMI350. (note need to run with `VLLM_ROCM_USE_AITER_FP8BMM=0` until https://github.com/vllm-project/vllm/issues/51957 is resolved) ```bash VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_FP8BMM=0 \\ vllm serve deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ ``` Sweep concurrencies 1-128, 1k/1k ```shell vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 --num-warmups 256' ``` ```shell pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k ``` ## Test Result ### Profiling This branch: Using DPA, the MLA decode o-proj is using preshuffled weights. No other GEMM is changed. 21us in trace. \"image\" Main branch: ck-tile kernel for o-proj. 52us in trace, so **more than 2x slower that of the bpreshuffled version!** \"image\" ### TP8+DPA | Concurrency | Branch | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | |-------------|-------------|---------------|---------------|--------|----------------| | 1 | This branch | 156.9816 | 18.2976 | 0.0529 | +7.30 | | 1 | Nightly | 158.3445 | 19.7720 | 0.0493 | | | 2 | This branch | 161.0075 | 18.5311 | 0.1049 | +8.14 | | 2 | Nightly | 198.6506 | 19.9552 | 0.0970 | | | 4 | This branch | 220.7625 | 18.7688 | 0.2054 | +7.48 | | 4 | Nightly | 284.2663 | 20.1512 | 0.1911 | | | 8 | This branch | 557.7459 | 22.6129 | 0.3479 | +7.61 | | 8 | Nightly | 317.1527 | 23.8614 | 0.3233 | | | 16 | This branch | 581.6531 | 23.5224 | 0.6528 | +5.96 | | 16 | Nightly | 609.4239 | 24.8873 | 0.6161 | | | 32 | This branch | 767.6995 | 23.7180 | 1.2794 | +4.61 | | 32 | Nightly | 787.4866 | 24.8746 | 1.2230 | | | 64 | This branch | 1449.0858 | 24.3342 | 2.4326 | +5.50 | | 64 | Nightly | 1394.4616 | 25.7091 | 2.3057 | | | 128 | This branch | 2018.3187 | 25.9321 | 4.4837 | +4.90 | | 128 | Nightly | 2201.5545 | 27.2313 | 4.2742 | | ### TP8+EP | Concurrency | Branch | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | |-------------|-------------|---------------|---------------|--------|----------------| | 1 | This branch | 108.3783 | 12.6642 | 0.0765 | -0.39 | | 1 | Nightly | 113.4111 | 12.6244 | 0.0768 | | | 2 | This branch | 172.7154 | 13.6376 | 0.1416 | +0.14 | | 2 | Nightly | 178.4821 | 13.6537 | 0.1414 | | | 4 | This branch | 280.6493 | 13.3944 | 0.2861 | +0.42 | | 4 | Nightly | 287.1797 | 13.446 | 0.2849 | | | 8 | This branch | 424.7564 | 13.5723 | 0.5592 | +0.81 | | 8 | Nightly | 427.5746 | 13.6773 | 0.5547 | | | 16 | This branch | 479.1289 | 14.4931 | 1.0434 | +0.69 | | 16 | Nightly | 484.925 | 14.6155 | 1.0363 | | | 32 | This branch | 817.9601 | 15.6236 | 1.9015 | +3.95 | | 32 | Nightly | 821.0856 | 15.8782 | 1.8292 | | | 64 | This branch | 938.6096 | 21.6289 | 2.7676 | +9.01 | | 64 | Nightly | 932.3192 | 23.9109 | 2.5388 | | | 128 | This branch | 1064.6245 | 27.3408 | 4.4065 | +4.04 | | 128 | Nightly | 1079.7563 | 28.087 | 4.2355 | | ### GSM8k This branch (TP8+DPA) |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9477|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9484|± |0.006 Nightly |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9439|± |0.0063| | | |strict-match | 5|exact_match|↑ |0.9431|± |0.0064| ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51692", "createdAt": "2026-08-10T14:54:38Z", "updatedAt": "2026-08-13T12:19:17Z", "timestamp": "2026-08-13T12:19:17Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "simondanielsson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5e46a8b899339e5025d6", "signalId": "github:vllm-project/vllm:pull_request:52159", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Hardware][Intel] Disable compiled grouped_topk on XPU", "text": "## Purpose Unconditional `@torch.compile` on `grouped_topk` emits an XPU Inductor kernel that gathers `scores[topk_ids]` with **out-of-range expert ids**: ``` Assertion `index out of bounds: 0 <= tmp0 < 128` failed. ``` This fires at engine-init warmup for any path that uses the Python grouped-topk router on XPU (NemotronH / Nemotron-3.5 Lightning MTP draft MoE, and any fallback that is not fused inside `XPUExpertsWNA16`). The fused base WNA16 MoE kernel does **not** execute this Python router, which is why the backbone can load while the draft / compiled router dies. This is the same class of compiled-reduction bug vLLM already documents for other backends (see the PowerPC argmax note in `topk_topp_sampler.py`). **Fix:** decorate `grouped_topk` with `torch.compiler.disable` **only when** `current_platform.is_xpu()`. CUDA keeps `@torch.compile` + the existing fused / fallback path. `torch.compiler.disable` is a graph break at the router only; the rest of the model can stay compiled / XPU-graph-captured. This is complementary to https://github.com/vllm-project/vllm/pull/39145 (fused XPU `grouped_topk` registration). That PR is the right long-term kernel path. This PR unblocks the eager/native router that still runs when that kernel is not selected (draft MoE, fused-flag off, or registration not yet landed). It does **not** add a second routing algorithm, expert-id clamps, or a CUDA-breaking top-level XPU import. Related kernels PR: https://github.com/vllm-project/vllm-xpu-kernels/pull/524 — zero the Xe2 grouped-GEMM scheduler counter (`at::empty` → `at::zeros`). Without that, graph-mode MoE can still be non-deterministic even if this router is correct. ## Test Plan ```bash pytest tests/kernels/moe/test_grouped_topk_xpu.py ``` The new file is skipped on non-XPU. On XPU it: - calls `grouped_topk` for softmax/sigmoid × bias/no-bias with `num_experts=128`, `num_expert_group=8`, `topk=8`, `topk_group=4`; - asserts every id is in `[0, num_experts)` and weights are finite; - wraps the same call in `torch.compile` and asserts it does **not** device-assert. Existing CUDA `tests/kernels/moe/test_grouped_topk.py` is unchanged. Manual serve repro (Intel XPU, NemotronH / Nemotron-3.5 Lightning with a draft that hits Python `grouped_topk`): ``` vllm serve --quantization gptq ... ``` Pre-fix: EngineCore dies during warmup with `0 <= tmp0 < 128`. Post-fix: router no longer asserts. ## Test Result On Intel Arc Pro B70: - Pre-fix: compiled `grouped_topk` device-asserts `0 <= tmp0 < 128` at engine-init warmup for NemotronH MTP draft MoE. - Post-fix: same serve path no longer asserts in the router. `torch.compile(grouped_topk)` on-device returns in-range ids. - CUDA path is a no-op (decorator still `@torch.compile`). - Full-eager + this router is deterministic at temp-0. Compiled/graph mode can still diverge in *other* XPU compiled kernels (sampler / GEMM without the kernels `at::zeros` fix). This PR does **not** claim “XPU is now deterministic.” No MTP prenorm, V1-bypass, W4A8, converters, or expert-id clamps in this PR.", "url": "https://github.com/vllm-project/vllm/pull/52159", "createdAt": "2026-08-13T11:20:36Z", "updatedAt": "2026-08-13T12:18:09Z", "timestamp": "2026-08-13T12:18:09Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "SergiioB", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3fe0870b7b912acecabf", "signalId": "github:vllm-project/vllm:pull_request:51809", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51809", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Enable Kimi K3 KDA kernel tests on XPU", "text": "## Purpose `gather_initial_states()` asserted `state.is_cuda`, which is False for XPU tensors, so it blew up before reaching the Triton kernel; widened to `state.is_cuda or state.is_xpu`. `test_kda.py` hardcoded `DEVICE = \"cuda\"`, now taken from `current_platform.device_type` - the CUDA-only cases already self-gate. This gives XPU coverage of the KDA Triton kernels: chunked prefill (`chunk_kda`, plus the fused gate/cumsum variant) and the fused-recurrent packed-decode and spec-decode paths. ## Test Plan ```bash python -m pytest tests/models/kimi_k3/test_kda.py -v ``` ## Test Result 45 passed, 6 skipped on Intel Arc Pro B70. Tested also on H200 - 51 passed", "url": "https://github.com/vllm-project/vllm/pull/51809", "createdAt": "2026-08-11T11:12:28Z", "updatedAt": "2026-08-13T12:16:56Z", "timestamp": "2026-08-13T12:16:56Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "intel-gpu", "ready", "kimi", "k3" ], "author": "pmanczak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d478ee587d85c7b40207", "signalId": "github:vllm-project/vllm:issue:51986", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51986", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: mnnvl allreduce workspace init hangs 30s and leaks GPU memory on IB-only multi-node", "text": "### Your current environment
    The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-124-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA B300 SXM6 AC GPU 1: NVIDIA B300 SXM6 AC GPU 2: NVIDIA B300 SXM6 AC GPU 3: NVIDIA B300 SXM6 AC GPU 4: NVIDIA B300 SXM6 AC GPU 5: NVIDIA B300 SXM6 AC GPU 6: NVIDIA B300 SXM6 AC GPU 7: NVIDIA B300 SXM6 AC Nvidia driver version : 580.173.02 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) 6776P BIOS Model name: Intel(R) Xeon(R) 6776P CPU family: 6 Model: 173 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 Stepping: 1 CPU max MHz: 4600.0000 CPU min MHz: 800.0000 BogoMIPS: 4600.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 6 MiB (128 instances) L1i cache: 8 MiB (128 instances) L2 cache: 256 MiB (128 instances) L3 cache: 672 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-31,128-159 NUMA node1 CPU(s): 32-63,160-191 NUMA node2 CPU(s): 64-95,192-223 NUMA node3 CPU(s): 96-127,224-255 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS Not affected; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.1 vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: \u001b[4mGPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 CPU Affinity NUMA Affinity GPU NUMA ID\u001b[0m GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PXB NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS 0-31,128-159 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS 0-31,128-159 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS 32-63,160-191 1 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS 32-63,160-191 1 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PXB NODE NODE NODE SYS SYS 64-95,192-223 2 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE PXB NODE NODE SYS SYS 64-95,192-223 2 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB NODE 96-127,224-255 3 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PXB 96-127,224-255 3 N/A NIC0 PXB NODE SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NIC1 NODE PXB SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NIC2 NODE NODE SYS SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS SYS SYS SYS SYS SYS SYS NIC3 NODE NODE SYS SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS SYS SYS SYS SYS SYS SYS NIC4 SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE SYS SYS SYS SYS SYS SYS NIC5 SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS NODE X SYS SYS SYS SYS SYS SYS NIC6 SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE SYS SYS NIC7 SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE SYS SYS NIC8 SYS SYS SYS SYS NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS NIC9 SYS SYS SYS SYS NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS NIC10 SYS SYS SYS SYS SYS SYS PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE NIC11 SYS SYS SYS SYS SYS SYS NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_5 NIC2: mlx5_6 NIC3: mlx5_7 NIC4: mlx5_8 NIC5: mlx5_9 NIC6: mlx5_10 NIC7: mlx5_11 NIC8: mlx5_12 NIC9: mlx5_13 NIC10: mlx5_14 NIC11: mlx5_15 ============================== Environment Variables ============================== NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 CUDA_VERSION=13.0.2 LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 NVIDIA_VISIBLE_DEVICES=GPU-e713112b-69d2-1c97-3e3f-1d4928b14252,GPU-30ad23ee-a87d-946c-008c-0f642a30b03b,GPU-4be8db1a-84d2-ef7e-a378-908a70631412,GPU-328baf3a-8c49-14dd-9968-319bbb890442,GPU-d03d7060-7248-aeab-14d2-683c340e2be8,GPU-a9a9aff2-1e7b-b90d-9259-e5195fecde40,GPU-198f35fd-27f1-5bb4-f593-20bfb9cc5c4e,GPU-5278c0f8-77d8-3711-e6f8-399512c0d3cc NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_ENABLE_CUDA_COMPATIBILITY=0 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0 VLLM_USAGE_SOURCE=production-docker-image VLLM_BUILD_COMMIT=6e448d0ea9bf3d88d898b65449ca6dc2aec170ac VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/5006 VLLM_IMAGE_TAG=vllm/vllm-openai:v0.27.1 NCCL_SOCKET_IFNAME=bond0 NCCL_IB_PKEY=2 NCCL_IB_HCA=mlx5_0,mlx5_5,mlx5_8,mlx5_9,mlx5_10,mlx5_11,mlx5_14,mlx5_15 VLLM_ENGINE_READY_TIMEOUT_S=14400 NCCL_DMABUF_ENABLE=0 VLLM_ALLREDUCE_USE_FLASHINFER=0 VLLM_USE_V2_MODEL_RUNNER=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root VLLM_WORKER_MULTIPROC_METHOD=spawn ```
    ### 🐛 Describe the bug ## What happens Since the auto backend resolution started defaulting to `mnnvl` (#47219 / #47589), startup on non-fabric multi-node clusters attempts mnnvl workspace creation, which: 1. Stalls ~30s per workspace attempt (up to 2: non-quant + quant), then logs: - `Failed to initialize FlashInfer All Reduce workspace: [fd_exchange] timed out connecting to /tmp/cuda_fd_xchg_...` - `Failed to initialize FlashInfer Allreduce norm fusion workspace with backend=mnnvl` 2. I'm not sure why but we subsequently hit OOM on worker nodes. Disabling the pass avoids both: `--compilation-config '{\"pass_config\": {\"fuse_allreduce_rms\": false}}'` On \"vllm/vllm-openai:kimi-k3\" this did not occur because the mnnvl workspace was not attempted on this topology. ## Expected vLLM should pre-check fabric support before attempting multi-node mnnvl workspace creation, and skip cleanly on IB-only topologies instead of relying on a 30s timeout inside flashinfer. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51986", "createdAt": "2026-08-12T11:49:24Z", "updatedAt": "2026-08-13T12:16:30Z", "timestamp": "2026-08-13T12:16:30Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "kimi" ], "author": "appleparan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1acc92dc2f96bd3b5a25", "signalId": "github:vllm-project/vllm:pull_request:51757", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51757", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Fix] Support heterogeneous models (Gemma 4 / Qwen) failing with AmbiguousGlobalPerLayerAttributeError", "text": "### What this PR does This PR fixes severe crashes that occur when loading heterogeneous models (such as **Gemma 4** and **Qwen MTP**) with recent versions of `transformers` (>=5.15.0). These models possess varying `head_dim` and `num_key_value_heads` across layers. The new `transformers` library raises an `AmbiguousGlobalPerLayerAttributeError` (which does **not** inherit from `AttributeError`) when attempting to access per-layer attributes on the global config object. ### The Fix This PR addresses all 3 crash points during model initialization and weight loading: 1. **Config Initialization**: Adds a `_safe_getattr()` helper to `ModelArchConfigConvertorBase` that catches the `AmbiguousGlobalPerLayerAttributeError` exception when `get_head_size()` and others are evaluated globally, falling back safely to `per_layer_config[0]`. 2. **Layer Initialization**: `gemma4_mtp.py` now consults `config.per_layer_config[layer_idx]` during `__init__` to instantiate attention modules with the correct dimensional parameters rather than the potentially incorrect global defaults. 3. **Parameter Weight Loading**: Introduces `_safe_narrow()` into `BasevLLMParameter` (with a fast-path for homogeneous layouts) and adds 1D clipping fallback in `weight_utils.py` to prevent tensor slicing bounds overflow when merging parameters of disparate layer topologies. ### Validation Tested extensively in a clustered environment on H200 and A100 setups provisioning Gemma 4 31B and Qwen MTP models. *(Reference implementations and our runtime patcher used to validate this in production can be found at [quivent/vllm-heterogeneous](https://github.com/quivent/vllm-heterogeneous))* ### Checkboxes - [x] Tested locally - [x] Does not break standard/homogeneous models", "url": "https://github.com/vllm-project/vllm/pull/51757", "createdAt": "2026-08-11T03:33:25Z", "updatedAt": "2026-08-13T12:12:04Z", "timestamp": "2026-08-13T12:12:04Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen" ], "author": "quivent", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a3eb672ce3958ac97dbc", "signalId": "github:vllm-project/vllm:pull_request:52144", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52144", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] Add pause/resume E2E tests", "text": "## Purpose One part of the RL CI Matrix for vLLM proposed by #45585. This PR adds end-to-end tests for the pause/resume API used in RL rollout, covering: 1. is_paused state & idempotency: repeated pause/resume, multiple cycles 2. Invalid mode behavior — return 400; state unchanged 3. Behavior of keep/wait/abort on in-flight and new requests, and behavior after resume 4. clear_cache=true/false behavior — golden output comparison ## Test Plan ### Test Matrix Priority | Test Case | Main Coverage -- | -- | -- P0 | test_pause_resume_state_is_idempotent_across_cycles | is_paused, repeated calls, multiple cycles P0 | test_invalid_pause_mode_preserves_state | Status code and unchanged state for illegal mode P0 | test_pause_mode_request_lifecycle[mode] | abort/wait/keep for in-flight, new requests, and resume P1 | test_clear_cache_preserves_output_and_controls_prefix_cache | Golden output and cache retention/clearing ### Request Lifecycle Across the Three Modes Expected behavior matrix: | mode | in-flight request at pause return | new request submitted after pause | after resume | | ----- | ------------------------------------------------------------ | --------------------------------- | ----------------------------------------------------------- | | abort | already finished, `finish_reason == \"abort\"` | accepted but stays pending | new request completes normally; old request does not resume | | wait | already finished normally, typically `finish_reason == \"length\"` | stays pending | new request completes normally | | keep | not yet finished, generation progress frozen | stays pending | both old and new requests complete normally | ### Speed & Stability Use a class/module-scoped `server_url` fixture to start the server once and reuse it across all tests, calling `resume()` before and after each test to restore state. The cache test should perform its own `clear_cache=True` at the start to avoid cache pollution from shared server state. abort/wait/keep and multi-request behavior already have fairly comprehensive coverage in lower-level tests. The HTTP E2E layer just needs to preserve the key mappings above. ## Test Result H100 single card: ```sh pytest --durations=0 -sv test_pause_resume.py =============================== test session starts =============================== platform linux -- Python 3.11.15, pytest-9.1.1, pluggy-1.6.0 -- /opt/miniconda/envs/y00806874/bin/python3.11 cachedir: .pytest_cache rootdir: /home/c00944022/tests plugins: typeguard-4.5.2, anyio-4.14.1 collected 6 items test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles PASSED test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] PASSED test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] PASSED test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache PASSED ================================ slowest durations ================================ 36.12s setup test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles 10.06s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] 5.56s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 1.02s teardown test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache 0.73s call test_pause_resume.py::TestPauseResume::test_clear_cache_preserves_output_and_controls_prefix_cache 0.64s call test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.60s call test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles 0.10s call test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[keep] 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 0.01s teardown test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[wait] 0.01s setup test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.01s teardown test_pause_resume.py::TestPauseResume::test_mode_request_lifecycle[abort] 0.01s teardown test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s setup test_pause_resume.py::TestPauseResume::test_invalid_mode_preserves_state 0.01s teardown test_pause_resume.py::TestPauseResume::test_state_and_idempotency_across_cycles (2 durations < 0.005s hidden. Use -vv to show these durations.) =============================== 6 passed in 54.96s ================================ ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52144", "createdAt": "2026-08-13T10:03:59Z", "updatedAt": "2026-08-13T12:09:50Z", "timestamp": "2026-08-13T12:09:50Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "floatlibai", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6bd612e802bc47a9b9d1", "signalId": "github:vllm-project/vllm:pull_request:51968", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51968", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][Tests] Make tests device-agnostic", "text": "## Purpose Two kernel tests hardcoded a CUDA device, so they gave no coverage on XPU even though the kernels they exercise are available there. This makes both resolve the device via `current_platform` instead. **`test_fused_recurrent_packed_decode.py`** — gated on `torch.cuda.is_available()`, so it skipped on every non-CUDA accelerator. Both `fused_recurrent_gated_delta_rule_packed_decode` and its `fused_recurrent_gated_delta_rule` reference are pure Triton with no CUDA-only intrinsics, so the gate was stricter than the code requires. **`test_merge_attn_states.py`** — hardcoded `device=\"cuda\"` in eleven places and gated `test_merge_attn_states` on `current_platform.is_cuda()`. `test_merge_attn_states_both_empty` had no guard at all, so on XPU it did not skip but failed outright with `AssertionError: Torch not compiled with CUDA enabled`. vllm-project/vllm-xpu-kernels#467 (\"align merge_attn_states kernel interface\") registers `merge_attn_states` for `torch::kXPU` under the same `_C` schema and argument order that `vllm._custom_ops.merge_attn_states` already calls, and `vllm/platforms/xpu.py` imports `vllm_xpu_kernels._C` at module load. The op therefore resolves on XPU with no test-side dispatch, so the only platform-specific thing left in that test was the device string. No separate code path, no per-platform reference and no FP8 skip are needed: the XPU kernel supports fp32/fp16/bf16 inputs and the FP8-output path (static per-tensor `output_scale`), matching the CUDA coverage. No production code changed, and CUDA behaviour is unchanged. ## Test Plan ``` pytest tests/kernels/test_fused_recurrent_packed_decode.py -v pytest tests/kernels/attention/test_merge_attn_states.py -v ``` ## Test Result Intel Arc Pro B70 (torch 2.13.0+xpu, Triton 3.7.2): | Test | Before | After | | --- | --- | --- | | `test_fused_recurrent_packed_decode` | 6 skipped | **6 passed** | | `test_merge_attn_states` | 6 failed, 2592 skipped | **2598 passed** | The 6 failures before are `test_merge_attn_states_both_empty`, which had no skip guard and hit `Torch not compiled with CUDA enabled`. No regression on NVIDIA L40S: `test_merge_attn_states` is 2598 passed both before and after; `test_fused_recurrent_packed_decode` was also checked on L40S. The `use_fp8=True` x `head_size=613` subset (216 cases) passes on XPU, confirming both the FP8 output path and the `head_size % pack_size` constraint.", "url": "https://github.com/vllm-project/vllm/pull/51968", "createdAt": "2026-08-12T09:09:05Z", "updatedAt": "2026-08-13T12:07:57Z", "timestamp": "2026-08-13T12:07:57Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "intel-gpu" ], "author": "pmanczak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5a29164324bf9f30ed1d", "signalId": "github:vllm-project/vllm:pull_request:50429", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50429", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] HunyuanOCR - handle duplicate image_start tokens in xdrope positions", "text": "## Purpose Fix IndexError crash in HunyuanOCR multimodal inference. `get_xdrope_input_positions` loops over `` token count but indexes actual image tensor array. HunyuanOCR tokenizer inserts duplicate tokens → out-of-bounds access → EngineCore crash on first multimodal request. ## Test Plan **Reproduce (before fix):** ```bash # Start vLLM server with HunyuanOCR python3 -m vllm.entrypoints.openai.api_server \\ --model tencent/HunyuanOCR \\ --dtype bfloat16 \\ --max-model-len 8192 \\ --gpu-memory-utilization 0.85 \\ --trust-remote-code # Send multimodal request curl -X POST http://localhost:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{ \"model\": \"tencent/HunyuanOCR\", \"messages\": [{ \"role\": \"user\", \"content\": [ {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mNk+M9Qz0AEYBxVSF+FABJADveWkH6oAAAAAElFTkSuQmCC\"}}, {\"type\": \"text\", \"text\": \"What color is this?\"} ] }], \"temperature\": 0, \"max_tokens\": 50 }' ``` Expected: IndexError in hunyuan_vision.py:851 → EngineDeadError Verify fix: Same commands → successful 200 response with generated text ## Test Result Before (IndexError crash): ```bash (EngineCore pid=440) ERROR [core.py:1332] File \".../hunyuan_vision.py\", line 851, in get_xdrope_input_positions (EngineCore pid=440) ERROR [core.py:1332] t, h, w = image_grid_thw[image_index] (EngineCore pid=440) ERROR [core.py:1332] ~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore pid=440) ERROR [core.py:1332] IndexError: list index out of range (EngineCore pid=440) INFO [xpu_worker.py:167] XPUWorker shutdown: cleaning up HTTP 500: {\"error\": {\"message\": \"EngineCore encountered an issue.\", \"type\": \"InternalServerError\", ...}} ``` After (successful generation): ```bash { \"id\": \"chatcmpl-a1177725ffaacc5a\", \"object\": \"chat.completion\", \"created\": 1785411625, \"model\": \"tencent/HunyuanOCR\", \"choices\": [{ \"index\": 0, \"message\": { \"role\": \"assistant\", \"content\": \"To determine the color of the image, observe the visual elements: the entire image is a solid, uniform green. There are no other colors present. \\n\\nSo, the color of the image is green.\" }, \"finish_reason\": \"stop\" }], \"usage\": {...} } ``` ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/50429", "createdAt": "2026-07-30T12:06:07Z", "updatedAt": "2026-08-13T12:05:45Z", "timestamp": "2026-08-13T12:05:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "mganczarenko", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e451cbf58483a0d9e0ae", "signalId": "github:vllm-project/vllm:pull_request:51998", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51998", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "chore: Upstream Cohere parser fixes + tests", "text": "Adding some local fixes for Cohere parsers alongside corresponding tests", "url": "https://github.com/vllm-project/vllm/pull/51998", "createdAt": "2026-08-12T14:04:35Z", "updatedAt": "2026-08-13T12:05:37Z", "timestamp": "2026-08-13T12:05:37Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "ci/build", "tool-calling", "cpu", "nvidia" ], "author": "jasonozuzu-cohere", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e868522e10cc39a2f226", "signalId": "github:vllm-project/vllm:pull_request:51605", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51605", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Size batched Triton MoE tiles from per-expert rows", "text": "## Purpose `get_default_config` divides `M` by `E` to estimate rows per expert, but the expert-batched layout passes `max_num_tokens`, which is already per-expert — so the fallback over-sizes tiles for the E small GEMMs `batched_triton_kernel` launches. Platform-agnostic as it is a layout mismatch. Only the heuristic fallback changes; override and tuned-file paths are untouched. Also makes the tile index the fastest-varying grid axis, matching `fused_moe`. ## Test Plan `pytest tests/kernels/moe/test_batched_moe.py`, plus kernel A/B and e2e base-vs-patched from paired worktrees at the same commit with identical settings. Measured on XPU only so far; NVIDIA 2x H200 numbers to follow before this leaves draft. ## Test Result *XPU Intel Arc B70, 2x, TP=2* Kernel A/B, 40 cells: geomean 1.075x, min 0.97x, max 1.43x. E2E, 24 cells, all passing. `rows_per_expert = max_num_batched_tokens * topk / E` decides whether the heuristic engages: | regime | geomean | min | max | n | |---|---|---|---|---| | heuristic active | 1.026x | 1.008x | 1.042x | 6 | | defers to shared config | 0.999x | 0.967x | 1.040x | 9 | E2E pending *NVIDIA H200, 2x, TP=2* Pending — same kernel A/B and e2e pair, to be filled in before this leaves draft.
    Full numbers and limitations - E2E coverage is bf16 only: WNA16 rejects `moe_backend=batched_triton`, and fp8 only offers BATCHED_TRITON via auto-selection under a batched activation format. Quantized guards are unit-tested. - `benchmarks/kernels/benchmark_moe.py` covers dense shapes only and is CUDA-only (CUDA graphs, Ray `num_gpus`), so the kernel A/B above was run out-of-tree.
    The grid reorder comes from [intel-xpu-backend-for-triton#6200](https://github.com/intel/intel-xpu-backend-for-triton/pull/6200) (commit `afa5150`) by @Egor-Krivov. A follow-up will add `@triton.autotune` to this kernel for both CUDA and XPU, which needs the reorder as a prerequisite: once the autotuner picks the block sizes, the host can no longer derive the grid from the config dict. The default config stays the fallback for `VLLM_TRITON_FORCE_FIRST_CONFIG`, `VLLM_BATCH_INVARIANT`, and wna16. Claude was used for the analysis, benchmarking, and edits; all changes were reviewed by me.", "url": "https://github.com/vllm-project/vllm/pull/51605", "createdAt": "2026-08-10T03:16:08Z", "updatedAt": "2026-08-13T12:00:58Z", "timestamp": "2026-08-13T12:00:58Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "oonyshch", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b97928bf103f83e49269", "signalId": "github:vllm-project/vllm:pull_request:52158", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52158", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Apply logits scaling in the Transformers backend", "text": "## Purpose Transformers scales the logits in the `ForCausalLM` wrapper. This backend only loads the decoder and provides its own head and logits processor, so whatever the wrapper does after the decoder has to be reproduced here. Only `logit_scale` was, which means: - Granite-family and MiniCPM3 models returned logprobs `logits_scaling`x too confident: 9x for `ibm/PowerLM-3b`, 10x for `ibm-granite/granite-swash-2b`. - HyperCLOVAX silently dropped its muP multiplier. Scaling does not reorder the logits, so the generated tokens are unaffected. That is why this went unnoticed, and it means the impact is on consumers of logprob values: sampling with a temperature or penalties, logprob-based evals, speculative decoding acceptance, and anything ranking sequences by score. There is no config field or API to read the scale from, and no single convention for it either — `logit_scale` multiplies (Cohere), `logits_scaling` divides (Granite) except under muP where it multiplies (HyperCLOVAX), `logits_mup_width_multiplier` divides the hidden states (Inkling), `dim_model_base` implies a divisor (MiniCPM3), `lm_head_multiplier` multiplies (FalconH1). Rather than enumerate that, this PR measures it: at model init, build the model's own `ForCausalLM` class with its decoder and head replaced by stubs (no weights, no checkpoint), run its `forward` over known hidden states, and take the ratio. Whatever the wrapper does to the logits is then the only thing left in the output, and the value is inherited from Transformers instead of transcribed from it. If the ratio is not constant across the hidden states, the transform is not a scale (final logit softcapping) and cannot be folded into a logits processor, so we warn and leave the logits alone — as today. Same if the probe cannot be run at all. Neither is a regression, but both are now visible in the log rather than silent. Found while fixing attention sinks on the same backend (#52156), but independent of it. ## Test plan Three new tests in `tests/models/transformers/test_backend.py`: - `test_get_logit_scale` pins the measured scale for one config per convention (Granite, HyperCLOVAX, Cohere, MiniCPM3, Inkling), plus Llama for no scaling and Gemma2 to assert softcapping is *not* folded in as a scale. Pure CPU, no weights, runs in seconds. - `test_probe_logit_scale_runs_for_scaled_models` is the canary for the approach: it sweeps `MODEL_FOR_CAUSAL_LM_MAPPING`, sets every float config field that looks like a logit multiplier to a non-default value, and asserts the probe can still be run for each. If a Transformers refactor breaks the probe for a model that does scale its logits, we would silently fall back to 1.0; this fails CI instead. - `test_logits_scaling` compares the spread of the top-5 logprobs between the native and Transformers implementations of `ibm/PowerLM-3b`. Logprob *values* have to be compared because `check_logprobs_close` only compares token ids, which a missing division cannot change. Commands and results (1x GB200, transformers 5.15.0): ``` pytest tests/models/transformers/test_backend.py -k \"logit_scale\" # 8 passed pytest tests/models/transformers/test_backend.py -k \"logits_scaling\" # 1 passed ``` Mutation checks that the new tests bite: - Reverting the fix entirely: `test_logits_scaling` fails on 8/8 positions, max difference 48.06 against a 0.2 tolerance. - Dropping the config scalars from the decoder stub: the sweep fails on FalconH1, which reads its multiplier off the decoder. - Dropping `weight` from the head stub: the sweep fails on xLSTM, which reads the head dtype off it. What the probe finds across the 173 entries of `MODEL_FOR_CAUSAL_LM_MAPPING`, with default configs: 135 apply no scaling, 5 scale (Cohere/Cohere2/Cohere2Moe 0.0625, Inkling 1/24, MiniCPM3 0.1), 7 are nonlinear (Gemma2, Gemma3n, VaultGemma, RecurrentGemma, NanoChat, xLSTM — all final logit softcapping), 2 cannot have a default config built (Musicgen), and 24 wrappers cannot be run by the probe. Those 24 are encoder, seq2seq and legacy families (BERT variants, Whisper, TrOCR, XLNet, ...) whose head is not `lm_head`, plus Llama4/Mllama; none of them declares a logit scale, and they behave exactly as they do today. ## Model evaluation Prompt logprobs for a 55-token paragraph, `ibm-granite/granite-swash-2b`, bf16, against HF eager as the reference (mean logprob -1.5943). Measured on top of #52156, since that model also needs the sink fix to produce sane output at all: | build | mean logprob | mean abs delta | max abs delta | | --- | --- | --- | --- | | #52156 alone | -7.8748 | 6.8942 | 81.0792 | | #52156 + this PR | **-1.5990** | **0.0165** | **0.0863** | The remaining 0.0165 is bf16/kernel-level noise. ## Notes - Not a duplicate: no open PR touches logits scaling in the Transformers backend. - Final logit softcapping is still dropped by this backend, as it was before. It is not a scale, so it does not belong in the logits processor; `Attention(..., soft_cap=)` handles the attention-logit variant but the final one needs handling in `compute_logits`. Left for a separate PR, and the probe now warns when it is hit. - HyperCLOVAX's muP direction is covered by the unit test only. The smallest registered checkpoint is `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B`, too big for an e2e test here. - AI assistance was used for this change (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/52158", "createdAt": "2026-08-13T11:15:14Z", "updatedAt": "2026-08-13T11:57:20Z", "timestamp": "2026-08-13T11:57:20Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "tdoublep", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f566121b88115700a27c", "signalId": "github:vllm-project/vllm:pull_request:51989", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51989", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix Cosmos3-Edge processor after transformers 5.15 release", "text": "## Purpose This PR fixes Cosmos3-Edge processor which is broken when `transformers==5.15` is used, due to refactoring of underlying Qwen3-VL processor. With the fixes preprocessor will work correctly for both transformers==5.14 and 5.15. Also as model was released removed `is_available_online=False` from registry. ## Test Plan ``` python -m pytest tests/models/multimodal/processing/test_cosmos3_edge.py ``` ## Test Result ``` 3 passed, 14 warnings in 10.34s ``` ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51989", "createdAt": "2026-08-12T12:28:23Z", "updatedAt": "2026-08-13T11:50:56Z", "timestamp": "2026-08-13T11:50:56Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "ready" ], "author": "bastefaniak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4d5f67f398aa90a1cb24", "signalId": "github:vllm-project/vllm:pull_request:52032", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52032", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add opt-in dynamic NVFP4 MoE GEMM2 quantization", "text": "### Summary This adds a default-off runtime option for ordinary per-routed-token NVFP4 quantization of the post-activation GEMM2 input in the FlashInfer TRTLLM MoE backend: ```bash VLLM_FLASHINFER_MOE_NVFP4_DYNAMIC_GEMM2=1 \\ vllm serve MODEL --moe-backend flashinfer_trtllm ... ``` The checkpoint's GEMM1 input remains quantized with its existing static outer scale and group-16 E4M3 block scales to allow working together with EP / EPLB. GEMM1 emits BF16 post-activation rows, FlashInfer computes one live FP32 outer scale per routed row, requantizes those rows to group-16 NVFP4, and runs GEMM2 with the existing W4A4 cubin. When the option is enabled, the checkpoint GEMM2 activation scale is ignored. The default path is unchanged. No FlashInfer or checkpoint change is required. ### Parallelism and state handling - Monolithic and modular TRTLLM expert wrappers share the same scale handling. - TP replicates the GEMM1 outer scale and constructs a rank-local row vector; no scale slicing or collective is needed. - Modular EP keeps `expects_unquantized_inputs=False`, so the established static NVFP4 GEMM1 input and block scales are dispatched normally. The row vector is created from the post-dispatch token count on each rank. - EPLB-visible values remain expert-major registered parameters. The shared GEMM1 outer decode scalar is a non-persistent buffer, so EPLB neither treats it as an expert tensor nor tries to permute it. - Existing online `nvfp4_per_token` remains supported; it already supplies live GEMM1 row scales and automatically uses dynamic GEMM2. However, it does not support EP or EPLB. ### Performance and quality evidence The exact production implementation was exercised on four B200 GPUs using the released Mistral Large 3 NVFP4, TP4, FlashInfer TRTLLM MoE, and concurrency one on an internal long context dataset. The results are as follows: | Checkpoint | Backend | Accuracy | | ------------- | ------------- | ------------- | | [Mistral Large 3 FP8](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512/commit/383ffea2c7d60dfd44ca960e8e691709d4fdb9cd) | Flashinfer TRTLLM | 88% | | [Mistral Large 3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4/commit/c8fe8a02a4b2fb6af0cdeb4ee71fcf147742e59c) | Flashinfer TRTLLM | 66% | | [Mistral Large 3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4/commit/c8fe8a02a4b2fb6af0cdeb4ee71fcf147742e59c) | Flashinfer TRTLLM (DYNAMIC_GEMM2=1) | 87% | ### Tests Successfully ran the following tests: ``` tests/quantization/test_trtllm_nvfp4_dynamic_gemm2.py tests/quantization/test_trtllm_nvfp4_hidden_dim_padding.py tests/test_envs.py tests/distributed/test_eplb_quant_scale_consistency.py tests/kernels/moe/test_trtllm_nvfp4_moe.py tests/quantization/test_trtllm_nvfp4_dynamic_gemm2.py ``` ### Duplicate check Merged PR #48538 is adjacent but different: it adds online weight quantization plus dynamic GEMM1/GEMM2 and restricts that mode to the monolithic path. This change targets already-quantized W4A4 checkpoints, keeps GEMM1 checkpoint-static, and supports the modular EP path. ### AI disclosure AI assistance was used to investigate, implement, test, and draft this change. The human submitter must review every changed line and be prepared to explain and maintain it. [Edited for conciseness] ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52032", "createdAt": "2026-08-12T18:29:36Z", "updatedAt": "2026-08-13T11:49:53Z", "timestamp": "2026-08-13T11:49:53Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia", "quantization" ], "author": "dbari", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9d43a1c9afc0f6b71620", "signalId": "github:vllm-project/vllm:pull_request:52026", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52026", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Frontend] Import parser plugins under the vllm.* namespace so their logs survive", "text": "Fixes #52027 ## Purpose `ToolParserManager.import_tool_parser` and `ReasoningParserManager.import_reasoning_parser` import a plugin under a module name taken from its file name, so a plugin doing what every in-tree parser does - `logger = init_logger(__name__)` - ends up with a logger named after its file. Only the `vllm` logger is configured in `DEFAULT_LOGGING_CONFIG`, so that logger has no handler anywhere up its chain: * `logger.info` and `logger.debug` are dropped entirely - Python's `logging.lastResort` fallback is WARNING-level; * `logger.warning` and above leave *outside* vLLM's configuration: unformatted, and unaffected by `VLLM_LOGGING_LEVEL` or `VLLM_LOGGING_CONFIG_PATH`. Importing the plugin as `vllm.tool_parsers.plugins.` (and `vllm.reasoning.plugins.`) puts `__name__` inside the configured namespace, so plugin authors need to do nothing. Two notes on safety: * `import_from_path` uses `spec_from_file_location` and assigns `sys.modules` directly, so the dotted name needs no parent package. * Relative imports inside a plugin were already impossible under a bare module name, so nothing regresses there. ## Test Plan `tests/tool_use/test_plugin_logging.py`, parametrized over both managers: it writes a plugin that records `__name__`, imports it through the manager, and walks the logger chain from that name. ## Test Result The test asserts that a record from the plugin's own logger reaches a handler, and that the handler belongs to the `vllm` logger. Before this change the walk ends at the root with no handler; after it, it stops at `vllm`. Verified in isolation (stdlib only, vLLM's logging config shape): dummy_parser_plugin -> handler: None vllm.tool_parsers.plugins.dummy_parser_plugin -> handler: vllm Found while running a tool parser as a plugin: three parser bugs stayed invisible for a day because every warning the parser emitted was dropped.", "url": "https://github.com/vllm-project/vllm/pull/52026", "createdAt": "2026-08-12T17:19:19Z", "updatedAt": "2026-08-13T11:47:40Z", "timestamp": "2026-08-13T11:47:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "tool-calling" ], "author": "honziik", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4a3c8ef3dd78c5a0213b", "signalId": "github:vllm-project/vllm:pull_request:51655", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51655", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add Muse Glimmer model support", "text": "Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context. Adds the model, its config and processor, channel-scoped reasoning and ATEM tool-call parsers, and DFlash speculative decoding support for its draft head. The model does not emit JSON tool calls and does not wrap reasoning in tags. Every turn is a sequence of channel-scoped messages, and both parsers key off that framing, so --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer are used together. The reasoning parser forces skip_special_tokens=False; without it the markers are stripped before parsing and both channels collapse into content. tool_choice=\"required\" and named tool_choice set supports_required_and_named=False so vLLM does not apply JSON guided decoding to them -- that path assumes JSON tool calls, and forcing it here either trapped the call in the reasoning channel or leaked the raw framing into content. The DFlash draft head (MuseGlimmerAssistantModel) reuses the existing qwen3_dflash implementation: same architecture, same tensors. It reads the target's residual stream at layers [1, 13, 25, 37, 49] and predicts a 16-slot block per forward. Squashed from the onyx-support integration branch (30 commits). ## Purpose ## Test Plan ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51655", "createdAt": "2026-08-10T10:20:30Z", "updatedAt": "2026-08-13T11:46:25Z", "timestamp": "2026-08-13T11:46:25Z", "metrics": { "reactions": 15, "comments": 34 }, "labels": [ "documentation", "new-model", "speculative-decoding", "ready", "tool-calling", "qwen", "mrv2" ], "author": "xianbaoqian", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7d9b384b9882cb1813b9", "signalId": "github:vllm-project/vllm:pull_request:51924", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51924", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Refine FlashInfer one-sided All2All integration", "text": "## Purpose Refine the FlashInfer NVLink one-sided All2All integration for DeepSeek Blockwise FP8 MoE and sequence parallelism. - Describe one-sided activation payloads explicitly in bytes for BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8. - Dispatch E4M3 activations with FP32 1x128 scales and feed the received layout directly to the FlashInfer TRT-LLM DeepSeekFp8/BlockMajorK kernel. - Validate activation and scale shapes before converting scales from `[M, K/128]` to the TRT-LLM `[K/128, M]` layout. - Enable MoE sequence parallelism for `flashinfer_nvlink_one_sided`, avoiding routing and compute on TP-replicated post-attention tokens. - Document and exercise DeepSeek Blockwise FP8 support in the existing MoE and MNNVL test matrices. No kernel-warmup changes are included because current upstream `main` already provides generalized distributed FlashInfer autotuning and persistent-cache synchronization. ### Duplicate-work check No issue number was supplied, so an issue-specific lookup was not applicable. I checked open PRs with: ```bash gh pr list --repo vllm-project/vllm --state open --search 'FlashInfer one-sided All2All' gh pr list --repo vllm-project/vllm --state open --search 'DeepSeek Blockwise FP8 FlashInfer' gh pr list --repo vllm-project/vllm --state open --search 'sequence parallel FlashInfer one sided' ``` The related open PRs are distinct: - #47733 handles per-tensor FP8 scaling, invalid expert IDs, and communicator cleanup; it does not add E4M3 activation dispatch with FP32 1x128 scales, TRT-LLM BlockMajorK integration, or sequence parallelism. - #42034 only changes the one-sided padding sentinel for local expert maps. - #42133 handles delayed MXFP8 scale swizzling for FlashInfer CUTLASS. ## Test Plan - Run all pre-commit hooks on the changed files. - Compile every changed Python file. - Check the patch for whitespace errors. - Validate DeepSeek Blockwise FP8 payload sizing and TRT-LLM scale layout with focused assertions. - Run a reproducible model evaluation on a public checkpoint. ## Test Result Passed all configured pre-commit hooks on the changed files, including ruff check/format, typos, markdownlint, mypy, SPDX, forbidden-import, configuration, and API checks: ```bash .venv/bin/pre-commit run --files \\ docs/design/moe_kernel_features.md \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py ``` Passed: ```bash .venv/bin/python -m py_compile \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py git diff --check origin/main...HEAD ``` A focused Python assertion script covering BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8 dispatch byte layouts plus the TRT-LLM `[M, K/128]` to `[K/128, M]` scale conversion passed. The repository-wide collection command below could not run in this CPU development environment because the installed `xgrammar` lacks `normalize_tool_choice` and the CUDA flash-attention extension is unavailable: ```bash .venv/bin/python -m pytest -q --collect-only \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py ``` ### Model evaluation Full 1,319-example GSM8K on public `Qwen/Qwen3.5-35B-A3B-FP8`, using 8 B300 GPUs with TP2 x DP4 / EP8. The runtime used this PR plus the Qwen3Next SP correctness fix from #50685; that fix is not part of this PR. Common settings: `flashinfer_trtllm`, FlashInfer autotune enabled, `max_num_batched_tokens=4096`, `max_num_seqs=64`, 5-shot greedy decoding, `max_gen_toks=16384`, and 256 concurrent requests. Only the All2All backend changed. The task used `\"\\nQuestion:\"` instead of stock lm-eval's broad `\"Question:\"` stop to avoid terminating inside model reasoning. | All-to-all backend | Flexible exact match | Strict exact match | |---|---:|---:| | `allgather_reducescatter` | 95.75% (1263/1319) | 92.49% (1220/1319) | | `flashinfer_nvlink_one_sided` | 95.00% (1253/1319) | 90.98% (1200/1319) | Both runs completed without retries or engine errors. The flexible-score gap is 10 samples and is not statistically significant (`p=0.143`). Per-sample review and a repeated AG/RS run found trajectory and formatting variation, but no systematic one-sided correctness failure. ## AI assistance AI assistance was used for code exploration, cherry-pick conflict resolution, compatibility review, validation, and PR drafting. The human submitter must review every changed line, understand the change end-to-end, and be able to defend it before merging. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] The current test results and model-evaluation status. - [x] The necessary documentation update.
    ", "url": "https://github.com/vllm-project/vllm/pull/51924", "createdAt": "2026-08-12T04:41:12Z", "updatedAt": "2026-08-13T11:42:16Z", "timestamp": "2026-08-13T11:42:16Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "documentation", "ready", "nvidia", "verified" ], "author": "bobboli", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1516305df2b0173d4e12", "signalId": "github:vllm-project/vllm:pull_request:45802", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45802", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine", "text": "## Purpose Add token-aware reasoning token counting for the Streaming Parser Engine and surface the count through OpenAI-compatible usage fields. - Adds `completion_tokens_details.reasoning_tokens` to usage responses. - Propagates token counts through the parser engine pipeline: `TokenIDScanner -> IncrementalLexer -> StreamingParserEngine -> SemanticEvent`. - Counts only `REASONING_CHUNK` tokens, excluding reasoning boundary terminals such as `` / ``. - Reuses the engine-based streaming parser path for non-streaming parsing when output token IDs are available, so streaming and non-streaming reasoning token accounting use the same token-aware boundaries. - Ensures parser instances are scoped per output choice in chat completion handling. - Adds coverage for parser-engine token counting, chat completion usage, and Responses API reasoning token accounting. ## Test Plan ## Test Result ``` curl http://localhost:8001/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -H \"Authorization: Bearer EMPTY\" \\ -d '{ \"model\": \"\", \"messages\": [ { \"role\": \"user\", \"content\": \"北京什么天气？\" } ],\"chat_template_kwargs\":{\"enable_thinking\": true}, \"tools\": [ { \"type\": \"function\",\"strict\":true, \"function\": { \"name\": \"get_weather\", \"description\": \"Get current weather for a city\", \"parameters\": { \"type\": \"object\", \"properties\": { \"city\": { \"type\": \"string\" } }, \"required\": [\"city\"] } } } ], \"tool_choice\": \"auto\", \"stream\": false }' {\"id\":\"chatcmpl-80b2a0ffa82be55a\",\"object\":\"chat.completion\",\"created\":1782098609,\"model\":\"/mnt/data4/models/Qwen/Qwen3.5-35B-A3B-FP8\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":null,\"refusal\":null,\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":[{\"id\":\"chatcmpl-tool-ac88f083f78ccffb\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\": \\\"北京\\\"}\"}}],\"reasoning\":\"用户询问北京的天气情况。我需要使用get_weather工具来获取北京的当前天气信息。\\n\"},\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"stop_reason\":null,\"token_ids\":null,\"routed_experts\":null}],\"service_tier\":null,\"system_fingerprint\":\"vllm-0.23.1rc1.dev243+gb529bfd6c-tp2-4b4f7693\",\"usage\":{\"prompt_tokens\":277,\"total_tokens\":323,\"completion_tokens\":46,\"prompt_tokens_details\":null,\"completion_tokens_details\":{\"reasoning_tokens\":18}},\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null} ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/45802", "createdAt": "2026-06-16T09:31:21Z", "updatedAt": "2026-08-13T11:34:01Z", "timestamp": "2026-08-13T11:34:01Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "frontend", "ready", "tool-calling" ], "author": "chaunceyjiang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2411092fdf78e14b6128", "signalId": "github:vllm-project/vllm:pull_request:51332", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51332", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization][Humming] Support MXFP4 weight + block-FP8 activation for MoE", "text": "## Purpose Enable Humming MoE GEMM with **MXFP4 weights + block-FP8 (group-128) activations**, and — on the expert-parallel (DeepEP) path — quantize activations to block FP8 **before** the all-to-all dispatch so FP8 rather than BF16 crosses the interconnect. The Humming kernel already implements MXFP4 weights (group-32 `e8m0`) with a grouped FP8 activation via WGMMA software dequant, and runs on SM90 (Hopper/H200) — the `grouped-fp8` cases in Humming's `tests/kernels/humming/test_mxfp4.py` cover it. Only the vLLM plumbing to select it was missing. Two fixes: ## EP path: pre-dispatch block-FP8 quantization By default Humming quantizes activations internally, so EP dispatches **BF16**. For block-FP8 (group-128) we now quantize **before** the DeepEP dispatch, so only FP8 + fp32 block scales cross the interconnect (~half the dispatch traffic) and Humming consumes the pre-quantized input directly: 1. `get_humming_moe_quant_config` surfaces the block-FP8 dtype + `[1, 128]` group shape so the config reports `is_block_quantized`. 2. `expects_unquantized_inputs` returns `False` for block-FP8, so prepare/finalize does not defer quantization and emits the `(fp8, scales)` tuple. 3. `apply()` forwards the dispatch scale (`a1q_scale`) into `may_quant_input` (a no-op when a scale is supplied), so Humming skips the redundant w13 quantization; the scale is permuted/reshaped alongside tokens. No DeepEP-side change is needed: the dispatch byte count is data-driven off the quant config (`use_fp8_dispatch`), the same block-FP8 v2 path other backends already use. Every other scheme leaves `activation_group_shape=None` → unchanged BF16-dispatch behavior. ## Cap MoE tuning K-block at 128 Small-M tiles can select a K-block of 256 (`block_shape[2]`), whose TMA descriptor the driver rejects at `cuLaunchKernelEx` (`CUDA_ERROR_MISALIGNED_ADDRESS`). It fires in `profile_run`, so `--enforce-eager` does not avoid it. `_cap_tuning_config_k_block` caps `block_shape[2]` at 128 — what Humming already uses for larger M — so the W4A8 MoE runs under CUDA graphs. ### Usage On an MXFP4-weight checkpoint (e.g. `nm-testing/Qwen3-30B-A3B-MXFP4A16`): ``` VLLM_HUMMING_INPUT_QUANT_CONFIG='{\"dtype\":\"float8e4m3\",\"group_size\":128}' ``` The per-token variant `{\"dtype\":\"float8e4m3\"}` and BF16 dispatch for non-block schemes are unchanged. ## Test Plan - `tests/quantization/test_humming_mxfp4_block_fp8.py`: schema→`QuantKey` mapping, `(kMxfp4Static, kFp8Dynamic128Sym)` now allowed, block-FP8 config is `is_block_quantized` with a `[1, 128]` block, and default (no block shape) stays per-token/deferred. Schema-mapping cases skip when Humming isn't installed. - `tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8-block.yaml` — GSM8K config, added to `config-act-fp8.txt`. ## Test Result QuantKey-level verification passing locally: ``` grouped-128 float32 == kFp8Dynamic128Sym : True grouped-128 uint8 == kFp8Dynamic128Sym : False (old behavior) per-token float32 == kFp8DynamicTokenSym: True group-32 uint8 == kMxfp8Dynamic : True ``` > [!NOTE] > Draft: end-to-end GSM8K accuracy on H200 still to be run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51332", "createdAt": "2026-08-07T00:33:30Z", "updatedAt": "2026-08-13T11:33:03Z", "timestamp": "2026-08-13T11:33:03Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "elvircrn", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1f227ac7c3fee54b2207", "signalId": "github:vllm-project/vllm:pull_request:52152", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52152", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Add shared-disk weight transfer backend", "text": "## Purpose Add an inference-side `disk` weight-transfer backend for deployments where a trainer writes a complete safetensors checkpoint to shared storage and an orchestrator reloads it into a running vLLM engine through the existing start/update/finish control plane. The backend: - accepts one absolute local checkpoint directory per update session, visible at the same path on every inference worker; - loads only the requested primary safetensors checkpoint (not retained secondary weight sources); - preserves the standard model loader's weight mapping, expert-parallel filtering, shard-index validation, and strict missing-weight tracking; - registers lazily in the existing weight-transfer factory and documents the HTTP client flow; - always clears worker/session state when finishing raises, while preserving the original exception. Related use-case context: #48644. This PR does not close that broader request: it is inference-side only and intentionally does not add a trainer data plane, Hub downloads, fastsafetensors/GDS, or transactional rollback. > AI assistance disclosure: this change was researched and implemented with OpenAI Codex assistance. It is opened as a Draft and must receive final human-owner review before being marked ready. ## Test Plan ```bash uv run --no-project --active python -m pytest \\ tests/distributed/test_weight_transfer.py -k disk -q uv run --no-project --active python -m pytest \\ tests/v1/worker/test_gpu_worker_weight_transfer.py \\ -k finish_error_resets_target_and_allows_next_session -q uv run --no-project --active pre-commit run \\ --from-ref origin/main --to-ref HEAD ``` ## Test Result - Shared-disk backend tests: **11 passed** - Worker finish-error cleanup (normal and draft sessions): **2 passed** - All applicable pre-commit hooks passed, including Ruff, formatting, typos, Markdown, mypy, SPDX, lazy-import, and configuration validation. - Re-run after rebasing onto the then-current upstream `main`. ---
    Essential Elements of an Effective PR Description Checklist - [x] Purpose and related context are documented. - [x] Reproducible test commands are included. - [x] Test results are included. - [x] Backend documentation and an end-to-end client example are included.
    ", "url": "https://github.com/vllm-project/vllm/pull/52152", "createdAt": "2026-08-13T10:44:21Z", "updatedAt": "2026-08-13T11:32:29Z", "timestamp": "2026-08-13T11:32:29Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "documentation" ], "author": "Sunbeam23333", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:065fbab01a23191dff0f", "signalId": "github:vllm-project/vllm:pull_request:52160", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52160", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Fix group numbering in Case 3 of hybrid_kv_cache_manager.md", "text": "## Purpose Fix a self-inconsistent group numbering in `docs/design/hybrid_kv_cache_manager.md` Case 3 (lines 111-124). The doc says Gemma-3-27b has 52 sliding-window layers, but lists: - Group 6: 10 sliding window attention layers (sw.40 - sw.49) - Group 7: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers With group_size=10, 52 sw layers end at Group 6 (sw.50-51 + 8 padding); there is no Group 7. sw.40-49 belongs to Group 5. Renumbered to: Group 5 = sw.40-49, Group 6 = sw.50-51 + 8 padding (7 groups total, Group 0-6). ## Why not duplicating an existing PR Searched `gh issue list` / `gh pr list` for \"hybrid_kv_cache_manager\": no open issue or PR. Pure documentation fix. ## Test Plan - pre-commit run markdownlint-cli2 --files docs/design/hybrid_kv_cache_manager.md ## Test Result Local environment has no pre-commit/Python; relying on CI check. --- ## AI Assistance This PR was drafted with AI assistance; the human submitter reviewed every changed line. Co-authored-by: deepseek-v4-flash", "url": "https://github.com/vllm-project/vllm/pull/52160", "createdAt": "2026-08-13T11:30:35Z", "updatedAt": "2026-08-13T11:31:11Z", "timestamp": "2026-08-13T11:31:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation" ], "author": "qwerqwerqwe8688-jpg", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:24c7b41ccac8883014c6", "signalId": "github:vllm-project/vllm:pull_request:50169", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50169", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Fix KV cache allocation for sliding-window drafters and local-attention pool sizing", "text": "## Purpose Three fixes around KV cache allocation for local-attention models and spec-decode drafters. Found via Laguna-S-2.1 + DFlash, but only one commit is model-specific. On current main (GB10, Laguna-S-2.1-NVFP4, DFlash, fp8 KV, max_model_len 131072, max_num_seqs 16, 23.1 GB KV pool) the series takes the KV pool from 415,517 to 871,430 tokens (2.1x), with draft acceptance unchanged. 1. Drafter layers of EAGLE-family speculators get scattered across KV cache groups by the strided group assignment in `_get_kv_cache_groups_uniform_page_size`. Fix: identify drafter layers (index >= target layer count, gated on `use_eagle()`) and give dedicated groups to the specs whose drafter layers would otherwise share a group with target layers. Sharing corrupts the drafter's absolute-slot context writes, which resolve through the target's block table: on Laguna-S-2.1 draft acceptance falls to 0.29% versus 26.67% with a dedicated group, at an identical pool size. On the V1 model runner the same layout instead trips the assertion in `SpecDecodeBaseProposer.validate_same_kv_cache_group` at startup. Separation is conditional because regrouping changes the group-size heuristic's inputs: separating regardless cut Qwen3.5-9B's pool from 863k to 185k tokens. Works for any PP size; the strided layout still applies to target layers. 2. `laguna_dflash.py` cleared the drafter's `attn.sliding_window`, forcing FullAttentionSpec on all 6 drafter layers (measured 38% of the pool for KV outside the window the drafter never reads). The `DFlashQwen3Model` base class already supports windowed draft layers; the clearing appears to be a workaround for the group-sharing corruption fixed in commit 1. Qwen3.5 DFlash avoids that corruption because its per-layer specs already give one group per layer, so its drafter never shares. Fix: keep the checkpoint-declared window. 3. `SlidingWindowSpec` and `ChunkedLocalAttentionSpec` charge the global `max_in_flight_tokens` bound to every request when sizing the pool. Since `sum_r (window - 1 + in_flight_r) <= R*(window - 1) + max_in_flight_tokens`, per-request sizing only needs the request's share of the allowance. Fix: amortize over `max_num_seqs`; the runtime admission gate is unchanged, and sizing is identical to today at `max_num_seqs = 1`. Without this, fixes 1 and 2 barely show (347k to 352k tokens), because sliding layers are charged almost like full attention. ## Scope - Commit 1: any EAGLE-family drafter (eagle/eagle3/mtp/dflash/dspark) over a hybrid target. All in-tree EAGLE-family implementations register drafter layers with indices continuing after the target's (`start_layer_id=target_layer_num` / `mtp_start_layer_idx=num_hidden_layers`), so identification cannot misclassify a target layer; unparseable names are skipped. No speculative config means grouping is unchanged. - Commit 2: Laguna only. Removes a divergence from its own base class. - Commit 3: every model using SlidingWindowSpec (Gemma 2/3/4, gpt-oss, Command R, EXAONE 4, and others), SlidingWindowMLASpec by inheritance, and ChunkedLocalAttentionSpec (Llama 4 style local attention). Broadest behavior change of the three; can be split into its own PR if preferred. The chunked local part is unit-tested only (no such model available for E2E here). ## Not duplicating existing PRs - #35062: same root cause as commit 1 (referenced in the commit message), stalled on rebase since June. Differences: per-spec drafter groups (a single merged group would mix specs for hybrid drafters), identification gated to `use_eagle()` (independent `draft_model` drafters number from 0), rebased with tests. Happy to defer to a revived #35062; will comment there. - #46032: relaxes the same-group invariant for `DraftModelProposer` only. EAGLE-family proposers still assert a single group. Complementary. - Nothing open covers commits 2 or 3. ## Test plan and results - `pytest tests/v1/core/test_kv_cache_utils.py tests/v1/core/test_swa_inflight_window_free.py tests/v1/core/test_single_type_kv_cache_manager.py tests/v1/test_kv_cache_spec_registry.py`: 131 passed, 0 failed (needs a GPU; some device-inference tests fail in a GPU-less container with or without this change). New tests: `test_drafter_layers_get_dedicated_kv_cache_group`, `test_drafter_layer_identification_requires_eagle_family`, `test_sliding_window_pool_sizing_amortizes_in_flight_tokens`, `test_chunked_local_pool_sizing_amortizes_in_flight_tokens`. Two existing tests that hardcoded the old sizing constants now pin `max_num_seqs=1`, where old and new sizing coincide. - Validated on a second real model, not simulated: `Intel/Qwen3.5-9B-int4-AutoRound` with `z-lab/Qwen3.5-9B-DFlash` at 30b0d54 (hybrid drafter, 5 sliding + 1 full; auto-selects the V2 runner). Pool 863,023 stock vs 864,749 with this PR, and identical spec-decode counters, i.e. exactly neutral. Synthetic `KVCacheSpec` fixtures were not predictive here, indicating no change for a configuration that in fact lost 79% of the pool, so grouping claims are traced inside the live engine. - `ruff check` and `ruff format --check` clean on all changed files. - Test env: `vllm/vllm-openai:nightly` (0.26.1rc1.dev18+gd223c900d), branch rebased onto that exact commit, changed files mounted over the installed package (aarch64 box, no source install). Should be re-run in a source install before merge. E2E on GB10, Laguna-S-2.1-NVFP4 + DFlash, fp8 KV, `max_model_len` 131072, `max_num_seqs` 16, 23.1 GB KV pool, V2 runner: | | pool tokens | max conc @128k | |---|---|---| | stock main | 415,517 | 3.17x | | + commits 1-2 | 466,211 | 3.56x | | + commit 3 | 871,430 | 6.65x | Acceptance, paired (identical seeded prompts in both arms, 15 speculative tokens): | context | stock | this PR | paired diff | |---|---:|---:|---:| | ~59 tokens (below the 512 window, no eviction) | 15.20% | 16.73% | +1.53pp (t=1.12) | | ~7,050 tokens (well above the window) | 17.82% | 23.48% | +5.65pp (t=0.84) | Both favour this PR and neither is significant, so I read this as no acceptance regression. Power caveat: the long-context paired sd is 13.5pp, so that row shows no evidence of a cost rather than proving neutrality. An earlier unpaired comparison suggested a ~2.8pp deficit; it did not survive pairing, because between-boot and between-prompt variance dominate on this checkpoint (greedy is non-deterministic from atomic-add reductions). Coherence (greedy known-answer) passes on every configuration. A controlled prefill/decode throughput A/B for commit 3 alone (on the prior 0.25 nightly, same box) showed no change and zero preemptions; throughput on this image was only sanity-checked. ## Disclosure AI assistance (Claude Opus 5 via Claude Code) was used to investigate, draft, and test this change. The submitter has reviewed every line and re-run the tests. _Edited: corrected the separation criterion to sharing-only after it was found to cost capacity, and replaced simulated and mismatched-length measurements with paired real-model ones._", "url": "https://github.com/vllm-project/vllm/pull/50169", "createdAt": "2026-07-28T18:02:11Z", "updatedAt": "2026-08-13T11:26:54Z", "timestamp": "2026-08-13T11:26:54Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "v1" ], "author": "iggerask", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4bdf282906ac108c7957", "signalId": "github:vllm-project/vllm:pull_request:52055", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52055", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle HarmonyError in process_chunk to fix gpt-oss streaming 500s", "text": "FIX #51977 ## Purpose gpt-oss requests fail with HTTP 500 (`openai_harmony.HarmonyError: unexpected tokens remaining in message header`) when the model samples a malformed Harmony message header. `HarmonyParser.flush()` already catches `HarmonyError` and recovers by returning the raw decoded text, but `process_chunk()` calls `self._harmony_parser.process(token_id)` with no handling, so a mid-stream parse failure escapes to the API server and kills the request. This gives `process_chunk()` the same recovery contract as `flush()`: on `HarmonyError`, decode the buffered message tokens plus the unconsumed tail of the chunk as raw final-channel text, emit it as a delta segment plus a completed message, reset parser state, and stop processing the chunk. The dual-segment shape matches what `flush()` already emits, so streaming, non-streaming `parse()`, and the Responses message loop all surface the text. `parse()` and `HarmonyContext.append_output` read only `completed_message`, so a delta-only segment would have turned the 500 into an empty 200 instead. Nothing the model generated is dropped, and the request completes. Note on the trigger: the issue points at a hyphen in the tool name, but openai_harmony 0.0.8 accepts hyphenated recipients. Reproducing directly against the library shows the real trigger is leftover tokens in the header. A duplicated `to=` clause reproduces the reporter's exact error string, which also explains why the failure was intermittent rather than reliable. Scope: vLLM cannot stop a model sampling a malformed header, so it has to tolerate `HarmonyError` from the library. `flush()` established that defensive contract already; this completes it for the mid-stream path. A leniency fix upstream in openai_harmony would be complementary, not a substitute. Not duplicating existing work: #51977 is unclaimed, and no open PR references it or touches `vllm/parser/harmony.py`. ## Test Plan New regression test `tests/parser/test_harmony.py::test_process_chunk_recovers_from_malformed_header` drives a duplicated `to=` header through `process_chunk`, asserts the generated text comes back as a final-channel delta segment plus a completed message instead of an exception, then asserts a normal message parses afterwards to prove the state reset. It fails without the source change. ```bash pytest tests/parser/test_harmony.py tests/reasoning/test_gptoss_reasoning_parser.py ``` ## Test Result ```text 66 passed ``` With `vllm/parser/harmony.py` reverted, the new test fails: ```text FAILED tests/parser/test_harmony.py::test_process_chunk_recovers_from_malformed_header ``` `ruff check` and `ruff format --check` clean on both files; mypy (CI configuration) reports no issues on either file. No model evaluation run: only the error-recovery path changes, and behaviour on well-formed output is untouched, covered by the 65 pre-existing tests in this file. --- This change was developed with AI assistance; I reviewed and tested every line.", "url": "https://github.com/vllm-project/vllm/pull/52055", "createdAt": "2026-08-12T21:45:49Z", "updatedAt": "2026-08-13T11:24:45Z", "timestamp": "2026-08-13T11:24:45Z", "metrics": { "reactions": 4, "comments": 4 }, "labels": [ "bug", "tool-calling", "gpt-oss" ], "author": "rajathpi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:653be1a7551ac6e8a1e1", "signalId": "github:vllm-project/vllm:pull_request:40337", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:40337", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring", "text": "## Summary Replaces the vanilla padded-bmm MaxSim path (introduced in #35330, re-enabled on GPU in #38620) with vendored flash-maxsim Triton kernels for ColBERT / ColPali document scoring. Addresses feature request #38282. ## Why this isn't a duplicate - Issue #38282 explicitly asks for this integration; no other open PR addresses it (searched `is:pr is:open flash maxsim`). - PR #38620 re-enabled GPU MaxSim infrastructure but explicitly deferred the kernel integration (\"only cleans up the API to make integration easier\") — this PR is the promised follow-up. - PR #35330 introduced the vanilla `compute_maxsim_score_batched` that this PR replaces. ## Approach Three scoring paths with automatic fallback: 1. **Zero-copy** (default for GPU, d ≥ 16, no chunked prefill, default pooling params): - Project the full `hidden_states` once via `TokenEmbeddingPoolerHead.project_batch()`. - Each doc request's pooler output is a `[L, d]` view into `projected_batch` — no copy, no extra allocation. - `flash_maxsim_rerank_direct` reads each doc at `(offset, length)` directly from the projected tensor. - The `[B, Lq, Ld]` score matrix is never materialized; the fused kernel tiles through SRAM. 2. **Flash-packed** (fallback when zerocopy disabled or pooling params use matryoshka / `use_activation=False`): - `torch.cat` all docs once, call `flash_maxsim_packed` with `cu_seqlens`. - No per-doc padding; kernel skips padding tokens via `cu_seqlens`. 3. **Vanilla** (fallback for CPU tensors, embedding dim < 16 (Triton `tl.dot` minimum), or `VLLM_FORCE_VANILLA_MAXSIM=1`): - Original sub-batched padded-bmm — unchanged. Triton autotune keys are bucketed (`{32, 64, 128, 256, 512, 1024, 2048, 4096}` for Lq/Ld, `next_pow2(d)` for embed dim). `LateInteractionRunner.__init__` pre-compiles 72 bucket combinations so no autotune fires on first request; failures during warmup are logged and cleanup is in `finally`. ## Files **New** (vendored Triton kernels — `vllm/v1/pool/flash_maxsim/`): - `flash_maxsim.py` — main forward kernel (hardware-specific autotune configs) - `flash_maxsim_rerank.py` — zero-copy rerank kernel (reads scattered offsets) - `flash_maxsim_varlen.py` — packed cu_seqlens kernel - `flash_maxsim_advanced.py` — q-reuse / split-K variants - `__init__.py` **Modified**: - `vllm/v1/pool/late_interaction.py` — `compute_maxsim_score_batched` dispatches to flash when available - `vllm/v1/worker/gpu/pool/late_interaction_runner.py` — kernel warmup, `_score_zerocopy`, `has_pending_docs` - `vllm/v1/worker/gpu_model_runner.py` — zerocopy branch in `_pool()` with fallback on chunked prefill / matryoshka / `use_activation=False` - `vllm/model_executor/layers/pooler/tokwise/heads.py` — `project_batch()` method (fp16 matmul, fp32 cast of small output) **Tests / demos**: - `tests/v1/worker/test_late_interaction_runner.py` — updated to d=32 (Triton requires K≥16) - `tests/v1/worker/bench_flash_maxsim.py` — detailed kernel benchmark - `tests/v1/worker/demo_flash_maxsim.py` — copy-paste kernel/memory comparison - `tests/v1/worker/demo_oom_resilience.py` — live server OOM demo ## Environment toggles - `VLLM_DISABLE_ZEROCOPY=1` — disable zero-copy, use flash-packed - `VLLM_FORCE_VANILLA_MAXSIM=1` — disable flash entirely, use vanilla bmm - `VLLM_FLASH_MAXSIM_WARMUP_D=` — extend warmup to an extra embedding dim ## Results (A100 80GB, ColBERT) ### Kernel-level on variable-length docs | Workload | Vanilla | Flash-Packed | Zero-Copy | |----------|---------|-------------|-----------| | B=1K, Ld=10-100 | 42.6ms | 0.58ms | **0.05ms** (~788×) | | B=5K, Ld=10-100 | 220.3ms | 2.16ms | **0.08ms** (~2,830×) | | B=10K, Ld=10-100 | 437.6ms | 4.04ms | **0.14ms** (~3,034×) | | B=10K, Ld=50-500 | 472.9ms | 4.22ms | **0.53ms** (~892×) | | ColPali B=1K, Ld≈1030 | 40.4ms | 0.72ms | **0.22ms** (~184×) | Most of vanilla's time (>90%) on varlen workloads is Python-side padding / fp32 cast / allocation, not bmm. The fused kernel eliminates that bookkeeping. ### E2E `/v1/score` (real embeddings, concurrent clients, best-of-3) | Docs/req | Conc | Flash TPS | Vanilla TPS | ΔTput | ΔP95 | |----------|------|-----------|-------------|-------|------| | 500 | 8 | 6.2/s | 5.2/s | **+19%** | **+19%** | | 500 | 16 | 6.0/s | 5.2/s | **+17%** | +7% | | 1000 | 4 | 3.0/s | 2.6/s | **+16%** | +13% | | 2000 | 4 | 1.6/s | 1.3/s | **+23%** | **+24%** | | 5000 | 1 | 0.5/s | 0.4/s | **+22%** | +18% | | 5000 | 2 | 0.6/s | 0.5/s | **+15%** | +19% | ### Correctness - Kernel vs fp32 bmm: `max_abs_err = 4e-6` - E2E 5K real docs: `max_abs_diff < 0.001`, zero pairs > 0.001 off, **top-3 rankings identical** Small tail-ranking noise at positions 4-5 reflects fp16 tensor-core nondeterminism on scores within 5e-4 of each other. ## Test plan - [x] `pytest tests/v1/worker/test_late_interaction_runner.py -v` → 4 passed - [x] `python tests/v1/worker/demo_flash_maxsim.py` → kernel speedups & memory savings displayed - [x] `python tests/v1/worker/bench_flash_maxsim.py` → full kernel benchmark - [x] 5K real ColBERT docs via `/v1/score` with `VLLM_FORCE_VANILLA_MAXSIM` vs flash → score parity verified - [x] `pre-commit run` ruff clean on all changed files - [ ] CI on `main` ## Notes for reviewers - The zerocopy path falls back to the normal pooler when `pooling_params.dimensions` is set (matryoshka) or `use_activation=False`, because `project_batch` normalizes before truncation — matryoshka would produce non-unit vectors. Falls back on chunked prefill for the same reason (partial cache). - `flash_maxsim_advanced.py` (q-reuse / split-K variants) is vendored but not dispatched from the default path. Kept for potential follow-ups. - Warmup takes ~160s cold (first launch on a host, no Triton disk cache) and ~15s warm; model load is ~80s either way. ## AI assistance Development of this PR used AI assistance (Claude). Every changed line was reviewed by the submitter; benchmarks were run end-to-end by the submitter against both paths on real A100 hardware with real ColBERT embeddings.", "url": "https://github.com/vllm-project/vllm/pull/40337", "createdAt": "2026-04-20T08:50:30Z", "updatedAt": "2026-08-13T11:15:45Z", "timestamp": "2026-08-13T11:15:45Z", "metrics": { "reactions": 3, "comments": 29 }, "labels": [ "frontend", "v1", "mrv2", "verified" ], "author": "roipony", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:34b9dc5aafee0b0cf573", "signalId": "github:vllm-project/vllm:pull_request:49233", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49233", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][V1] Reserve CUDA graph memory in V2 GPU model runner", "text": "## Purpose Fixes #49224. Model Runner V2 became the default for **dense** models in 0.25.1, but its `profile_cudagraph_memory()` was a placeholder returning `0`: ```python def profile_cudagraph_memory(self) -> int: # NOTE(woosuk): It is TBD whether we keep this API or not. return 0 ``` `Worker.determine_available_memory()` subtracts this estimate before sizing the KV cache (gated by `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS`, default on), so with V2 no headroom was reserved for CUDA graph capture: the KV cache claimed the whole `gpu_memory_utilization` budget and `capture_model()` OOMed at startup (e.g. `Llama-3.1-70B-Instruct` FP8, TP=8 on 8× L40S). The legacy runner is unaffected because it implements the method for real, and before 0.25.1 dense FP8 models used the legacy runner anyway. This PR implements `profile_cudagraph_memory()` for the V2 runner, reusing its own `initialize_kv_cache()` / `capture_model()`: 1. Return `0` early when `cudagraph_mode == NONE`. 2. Bootstrap a **minimal** KV cache (`_init_minimal_kv_cache_for_profiling`) — just enough blocks to capture every graph, matching the legacy runner. 3. Capture graphs into a **throwaway** graph pool (`current_platform.graph_pool_handle()`) so their memory is reclaimed on teardown and does not pollute the persistent global pool the real capture reuses. 4. Measure the free-memory delta via the existing `capture_model()`. 5. Release the profiling KV cache / attention groups / captured graphs while keeping model weights (`_teardown_profiling_state`), so the real `initialize_kv_cache()` / `capture_model()` path starts from a clean slate. **Note for reviewers:** the bulk (PIECEWISE) graphs are effectively captured once — in the e2e run below the real capture pass added only 0.07 GiB after profiling — so the extra startup cost is small. Happy to add legacy-style sampling (capture ~2 graphs per mode and extrapolate) if preferred, since the original `# NOTE(woosuk)` flagged this API as \"TBD\". ## Test Plan - **Unit** (`tests/v1/worker/test_gpu_model_runner_v2_cudagraph_profiling.py`, CPU-only): covers the orchestration — disabled-cudagraph short-circuit, the no-graphs path, the capture path (asserting a throwaway pool is used and the measured value is returned), and teardown-on-error. ```bash pytest -q tests/v1/worker/test_gpu_model_runner_v2_cudagraph_profiling.py ``` - **End-to-end:** `Llama-3.1-70B-Instruct` FP8, `tensor_parallel_size=8`, `gpu_memory_utilization=0.90` on 8× L40S (no NVLink), with CUDA graphs enabled and the V2 runner as default (i.e. **without** `VLLM_USE_V2_MODEL_RUNNER=0` or `enforce_eager`) — the exact config that OOMs on 0.25.1. ## Test Result **Before (0.25.1, unpatched):** OOM during `capture_model()` at startup; server never becomes ready. Users on released 0.25.1 can work around this by forcing the legacy runner with `VLLM_USE_V2_MODEL_RUNNER=0` (keeps CUDA graphs), or via `enforce_eager` / a lower `--gpu-memory-utilization`. **After (this PR):** startup succeeds with CUDA graphs enabled and V2 as default. Key startup log: ``` Graph capturing finished in 39 secs, took 6.05 GiB # profiling capture (throwaway pool) [gpu_worker.py] Available KV cache memory: 19.46 GiB # headroom now reserved Graph capturing finished in 31 secs, took 0.07 GiB # real capture, no OOM INFO: Application startup complete. ``` Config confirms graphs were active (not silently disabled): `enforce_eager=False`, `cudagraph_mode=FULL_AND_PIECEWISE`, `tensor_parallel_size=8`, `quantization=modelopt`. Peak memory ~36.9/46 GiB per L40S (fits with headroom). Live checks: - `GET /v1/health/ready` → `200` - base model: `\"The capital of France is\"` → `\"a city of love, art, fashion, and cuisine.\"` - LoRA adapter: `\"Where is the Eiffel Tower?\"` → `\"Paris\"` Unit tests pass. --- *AI assistance disclosure (per the contributing guidelines): this change was developed with AI assistance (Cursor). All lines were reviewed and validated by the submitter, including the e2e run above.*
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/49233", "createdAt": "2026-07-20T19:14:21Z", "updatedAt": "2026-08-13T11:15:23Z", "timestamp": "2026-08-13T11:15:23Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "v1", "nvidia", "mrv2" ], "author": "anhtra3889", "state": "open", "assignees": [ "njhill" ], "change": "updated" } }, { "id": "event:2ce148898b3ec1058786", "signalId": "github:vllm-project/vllm:issue:52155", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52155", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: VLLM_BATCH_INVARIANT does not cover convolution, used in VAE blocks", "text": "### Your current environment OS : Ubuntu 22.04.5 LTS (x86_64) Clang version : 22.0.0git (roc-7.2.3 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) Libc version : glibc-2.35 PyTorch version : 2.11.0+gitd0c8b1f ROCM used to build PyTorch : 7.2.53211 Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.15.0-131-generic-x86_64-with-glibc2.35 Is CUDA available : True GPU models and configuration : (gfx942:sramecc+:xnack-) # 8x AMD Instinct MI300X HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 CPU : Intel(R) Xeon(R) Platinum 8480C, 224 CPU(s), 2 sockets, 2 NUMA nodes ### 🐛 Describe the bug `init_batch_invariance()` does not address convolution, so a model whose forward contains one keeps a nondeterministic reduction while `VLLM_BATCH_INVARIANT=1` is on. The `torch.library` override set covers `mm`, `addmm`, `matmul`, `linear`, `softmax`, `mean.dim`, `bmm` and `rms_norm`. The only line touching conv is ```python # vllm/model_executor/layers/batch_invariant.py, init_batch_invariance() torch.backends.cudnn.conv.fp32_precision = \"ieee\" ``` which constrains precision, not reduction order — and `torch.backends.cudnn.deterministic` is never set. Affected models are the ones with a convolution in the forward path: multimodal vision patch-embeds (Qwen2.5-VL's is a `conv3d`) and diffusion VAEs. **This is not ROCm-specific**: that flag exists because cuDNN also has nondeterministic convolution algorithms, and it is unset on both backends. Possibly what [#27059](https://github.com/vllm-project/vllm/issues/27059) hit before it was closed as stale, though I have not verified that on CUDA. **Suggested fix.** Set it alongside the existing `fp32_precision` lines — already inside `if envs.VLLM_BATCH_INVARIANT:`, so the default path is untouched, and it has no effect on models without convolutions. On ROCm the same flag maps to MIOpen, no backend branch. **Scope.** The missing conv coverage is a code fact and platform-independent. The measurement is run-to-run at fixed shape on ROCm, not batch invariance, and not a claim about SM80/SM90. `cudnn.deterministic` also does not pin *which* algorithm is selected, and the heuristics read batch size as part of the shape — deterministic conv and batch-invariant conv stay different claims. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52155", "createdAt": "2026-08-13T10:57:25Z", "updatedAt": "2026-08-13T11:13:10Z", "timestamp": "2026-08-13T11:13:10Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "ZJLi2013", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:645cb4ecbb596de97b75", "signalId": "github:vllm-project/vllm:pull_request:45423", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45423", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Correct prompt lengths for timed_traces benchmark", "text": "## Purpose #39795 introduced `timed_traces` support for `vllm bench serve`. The traces look as follows: ``` {\"timestamp\": 0, \"input_length\": 6758, \"output_length\": 500, \"hash_ids\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]} {\"timestamp\": 0, \"input_length\": 7322, \"output_length\": 490, \"hash_ids\": [0, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]} {\"timestamp\": 0, \"input_length\": 7236, \"output_length\": 794, \"hash_ids\": [0, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]} {\"timestamp\": 0, \"input_length\": 2290, \"output_length\": 316, \"hash_ids\": [0, 42, 43, 44, 45]} ``` Current implementation creates prompts with these input_lengths, and then runs: ``` prompt = tokenizer.decode(prompt_ids) [...] samples.append( SampleRequest( prompt=prompt, prompt_len=prompt_len, ``` vLLM engine on the server side runs: ``` prompt_idx = tokenizer.encode(prompt) ``` Unfortunately, tokenizer logic is not idempotent, so whereas the client generates requests of length 6758, 7322, 7236, 2290, etc., the server receives requests of different length 7253, 7844, 7679, 2445, etc. **Solution:** Avoid calling `.decode` and `.encode` The OpenAI API and vLLM engine support direct tokens id passing, see: `vllm/entrypoints/openai/completion/protocol.py` ``` class CompletionRequest(OpenAIBaseModel): # Ordered by official OpenAI API documentation # https://platform.openai.com/docs/api-reference/completions/create model: str | None = None prompt: ( list[Annotated[int, Field(ge=0)]] | list[list[Annotated[int, Field(ge=0)]]] | str | list[str] | None ) = None ``` This PR avoids the `.encode()` to directly pass token ids, maintaining correct prompt lengths on the server side. ## Test Plan Just run any of the traces **sequentially** (**--max-concurrency 1**) : ``` vllm bench serve \\ --model Qwen/Qwen3.5-0.8B \\ --dataset-name timed_trace \\ --num-prompts 50 \\ --dataset-path conversation_trace_optimal_store_and_prefix_match.jsonl \\ --base-url http://localhost:12305 \\ --ignore-eos \\ --self-timed \\ --timed-trace-label-timestamp timestamp \\ --timed-trace-chunk-hash-size 512 \\ --timed-trace-label-hash-ids hash_ids \\ --timed-trace-label-input-length input_length \\ --timed-trace-label-output-length output_length \\ --timed-trace-sec-multiplier 0.001 --max-concurrency 1 ``` + vLLM serve in debug mode: ``` vllm serve --port 12305 --enable-prefix-caching --model Qwen/Qwen3.5-0.8B ``` and observe `request.num_tokens` in `vllm/v1/core/sched/scheduler.py::Scheduler.schedule` ## Test Result Json file input_lengths: 6758, 7322, 7236, 2290, ... **Main:** For consecutive scheduling rounds, `request.num_tokens=7253, 7844, 7679, 2445,...` **This PR:** For consecutive scheduling rounds, `request.num_tokens=6758, 7322, 7236, 2290, ...` -- as expected @animeshtrivedi @tdoublep", "url": "https://github.com/vllm-project/vllm/pull/45423", "createdAt": "2026-06-12T15:44:03Z", "updatedAt": "2026-08-13T11:06:38Z", "timestamp": "2026-08-13T11:06:38Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "bug", "performance", "ready" ], "author": "s3woz", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2828c2ee0ff47d240b52", "signalId": "github:vllm-project/vllm:issue:51914", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51914", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark", "text": "## Summary With **DeepSeek-V4-Flash-0731** on **vLLM v0.27.1** with **DSpark enabled**, we have intermittently observed malformed DSML tool-call output where the opening wrapper is corrupted from: ```text <｜DSML｜tool_calls> ``` to something like: ```text <｜DSML｜toolcalls> ``` while the rest of the DSML block remains structurally close to valid and the closing wrapper is still emitted as: ```text ``` This causes the tool-call parser to miss the call and surface the DSML as assistant content instead of producing `tool_calls`. This report intentionally uses a **synthetic example** and does not include the original production prompt, tool names, arguments, paths, or other private information. ## Environment - vLLM: **v0.27.1** - Model: **deepseek-ai/DeepSeek-V4-Flash-0731** - Speculative decoding: **DSpark enabled** - Tool-call parser: DeepSeek V4 / DSML path - Observed intermittently rather than deterministically ## Synthetic example Expected model output: ```text <｜DSML｜tool_calls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` Malformed output observed in the same class of failure: ```text <｜DSML｜toolcalls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` The difference is only the opening wrapper (`tool_calls` -> `toolcalls`), but that is enough for strict DSML parsing to fail. ## Expected behavior If the model emits an otherwise complete DSML invoke block for a tool that is actually declared in the request, vLLM should ideally either: 1. recover the tool call safely, or 2. at minimum detect the malformed DSML as a protocol error instead of silently returning the whole block as ordinary assistant content. Any recovery should remain conservative and validate the recovered tool name against the tools declared in the request. ## Relation to existing issues This appears related to #48931 and #49117, but it is a **different malformed-output variant**. - #48931 covers the case where the `<｜DSML｜tool_calls>` opening wrapper is **completely omitted**. - This case has an opening wrapper, but it is **corrupted/misspelled** (`tool_calls` -> `toolcalls`). - Therefore, recovery logic that only handles a fully missing wrapper may not cover this case as-is. It would be useful to add parser regression coverage for malformed opening wrappers in both streaming and non-streaming paths, especially with DeepSeek-V4-Flash-0731. ## DSpark note This was observed with DSpark enabled. I do **not** have enough evidence yet to claim that DSpark is the cause. A DSpark ON/OFF comparison would be useful because speculative decoding should not materially change protocol correctness if rejected draft tokens are handled correctly. The main issue here is that a small model-side DSML formatting error currently turns into a complete tool-call failure at the serving layer.", "url": "https://github.com/vllm-project/vllm/issues/51914", "createdAt": "2026-08-12T02:54:35Z", "updatedAt": "2026-08-13T11:06:17Z", "timestamp": "2026-08-13T11:06:17Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [], "author": "jinbagi", "state": "open", "assignees": [ "chaunceyjiang" ], "change": "updated" } }, { "id": "event:1072d01cd7401b1e918a", "signalId": "github:vllm-project/vllm:pull_request:51876", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51876", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Shard the MoE experts across NUMA nodes", "text": "## Purpose On a rank whose OpenMP threads span several NUMA nodes, the CPU MoE experts read their weights from every node. `parallel_2d` splits the work by thread index and has no notion of where a page lives, so each thread ends up pulling roughly `1 - 1/nodes` of its bytes across the interconnect. The kernel is already shaped for the fix. Both of its GEMM stages parallelise over the **output rows of the weight they read** — stage 1 over the `2N` rows of w1, stage 2 over the `K` rows of w2 — so handing a node a contiguous slice of that axis makes every thread read only pages placed on its own node. The outputs stay disjoint, so there is **no reduction, no barrier and no extra buffer**: the change is which thread gets which block, plus an `mbind` at load time. ### When this helps, and when it does nothing With the default `VLLM_CPU_OMP_THREADS_BIND=auto`, `_get_autobind_cpu_ids()` gives each rank the CPUs of exactly one NUMA node and asserts there are enough nodes to go around, so CPUs and memory already agree and this change is a no-op — the policy detects that and declines. It helps in one configuration: **a single rank whose threads span several NUMA nodes**, which is what a manual `VLLM_CPU_OMP_THREADS_BIND=0-127` on a multi-socket box gives you. ### How the policy decides The kernel maps OpenMP thread `ith` to shard `ith / (nth / shards)`, so sharding is only correct where that map is true of the actual thread binding. The policy therefore reads the binding itself — the ordered CPU list in `KMP_AFFINITY`'s `proclist` or `GOMP_CPU_AFFINITY` — and **verifies** the map rather than assuming it: the first `nth` entries have to form equal contiguous runs, one node each. The run boundaries are then the shard boundaries, and each run's node is where that shard's pages go. It declines, leaving the existing code path byte for byte, whenever that does not hold: threads on a single node (the auto-bind default), `OMP_NUM_THREADS` shorter than the CPU list so the first `nth` threads sit on a subset of the nodes, a list interleaved across nodes rather than grouped by them, a thread count that does not divide, fewer blocks than nodes, or a binding that does not name a CPU per thread at all — which includes the `OMP_PLACES={0,1,...}` + `OMP_PROC_BIND=true` fallback, a single place holding every CPU. Shard index and node id are kept distinct throughout: a rank confined to the second socket of an SNC box is nodes 2 and 3, not 0 and 1. `VLLM_CPU_MOE_NUMA_NODES` overrides the decision with an explicit list of node ids. ## Test Plan ```bash pytest tests/kernels/moe/test_cpu_moe_numa_shard.py pytest tests/kernels/moe/test_cpu_quant_fused_moe.py pytest tests/kernels/moe/test_cpu_fused_moe.py ``` Sharding only changes *which thread* computes which block; the arithmetic of each block is untouched. So the bar is `torch.equal` against the unsharded run, not a tolerance — anything else means a block was computed twice, or not at all, or against the wrong slice of the weights, and those are the three ways this can be wrong. ## Test Result | | | |---|---| | `test_cpu_moe_numa_shard.py` (new) | **19 passed.** 16 geometries × {2, 4} shards, all bit-identical | | `test_cpu_quant_fused_moe.py` | **154 passed** with the default; **58 passed** with sharding forced to 2 and to 4 | | `test_cpu_fused_moe.py` | **220 passed** with 1, 2 and 4 shards | The geometries deliberately include ones that do **not** divide evenly: `N=160` gives 10 blocks of `BLOCK_N`, which across 4 nodes is 3/3/2/2. Slices are whole blocks but need not be equal — requiring equality would have excluded `intermediate_size=2880`, which is the default in `benchmarks/kernels/cpu/benchmark_cpu_fused_moe.py`. ## Performance 2× Xeon Platinum 8592V (Emerald Rapids), SNC2, 4 NUMA domains, gcc 14.2. Geometry taken from this repo's own CPU MoE benchmark (hidden 2880, intermediate 2880, so stage 2 splits 90 blocks as 23/23/22/22). 64 threads bound in node order with `GOMP_CPU_AFFINITY`, one process per row, best of 30 iterations, three runs: | | GB/s of expert weights, six runs | median | spread | |---|---|---|---| | unsharded | 214 / 224 / 223 / 216 / 225 / 228 | 224 | 6.2% | | **4 shards** | **381 / 384 / 387 / 388 / 388 / 388** | **387** | **1.9%** | Paired ratios: 1.78 / 1.71 / 1.73 / 1.79 / 1.73 / 1.70 — **1.70× to 1.79×, median 1.73×**. Measured on an otherwise idle machine (load average 0.50 at the start of the run; an earlier set taken while the box was shared is what the 6.2% on the unsharded row is *not* — that spread is reproducible and is discussed below). Every row carries its thread placement, verified from `/proc/self/task/*/stat` rather than assumed: `64 active (64 parked), {n0=16, n1=16, n2=16, n3=16}`, no collisions. Reading that from `sched_getaffinity` instead would have been useless here — with `GOMP_CPU_AFFINITY` the calling thread is thread 0, pinned to one CPU, and reports `{node0: 1}` no matter how the other 63 landed. The unsharded row is three times more variable than the sharded one (6.2% against 1.9%), consistently. That is what you would expect if it is the one whose performance depends on where pages happen to have landed: with the shards placed, each thread reads its own node every time. Two notes on the measurement, both of which changed the number: * **Experts are rotated between iterations.** Repeating the same 8 experts keeps a 398 MB working set inside the 640 MB of L3 across the two sockets and reports **726 GB/s** — above this machine's 415 GB/s aggregate STREAM ceiling, which is the tell. Rotating puts both rows back under the ceiling and moves the ratio from 1.40× to 1.7×. * **One process per row**, because undoing a pinning restores the affinity mask but does not move threads that are already running, so a later row inherits the placement of an earlier one. ## Scope, and what this does not do * Weights are placed, scales are not. They are a sixteenth of the bytes for MXFP4, and on the FP8 block-quantized path their row axis does not line up with the block split, so a second rule would buy a few percent. * This does **not** change the process memory policy. `init_cpu_memory_env` still binds a rank's allocations to one node, so a model larger than a single node still fails to allocate; that is a separate problem, diagnosed in #51584. * Placement is driven from `CPUWorker.load_model` rather than from each quantization method's `process_weights_after_loading`, because the MXFP4 and FP8 CPU paths do not call the experts' post-load hook at all — hooking per-method would silently miss exactly the backends this helps most. ## Limitations **Measured on a single machine**: 2× Xeon Platinum 8592V, SNC2, 4 NUMA domains. The policy is topology-driven rather than hardcoded, but I have no access to an EPYC box to confirm the gain carries over to NPS4 or to per-CCD domains, and I would rather say so than have a reviewer find out. A 2P NPS4 box brings two changes at once — more domains *and* more shards — and I can only reason about the second: the policy never emits more shards than there are effective nodes, so raising the count past 4 here would mean breaking an invariant the patch itself enforces, and I would rather not quote a number I had to disable a check to get. If it would help the review, I am happy to measure the cost of the sharded structure alone at higher shard counts behind a temporary override. ## Notes for review `csrc/cpu/sgl-kernels/` is vendored from SGLang and re-synced wholesale, so the primitive lives outside it in `csrc/cpu/moe_numa_shard.hpp` and `moe_numa_parallel.hpp`; **the diff inside the vendored directory is three lines per file** — one include and the call names. `BLOCK_N` necessarily exists in three places (kernel, loader, policy). A `static_assert` in `moe_numa_parallel.hpp` ties the C++ copy to `block_size_n()`, so a change there breaks the build rather than leaving the loader placing pages on boundaries the kernel does not split on — which would be slower with no sign of it.", "url": "https://github.com/vllm-project/vllm/pull/51876", "createdAt": "2026-08-11T20:41:09Z", "updatedAt": "2026-08-13T11:03:01Z", "timestamp": "2026-08-13T11:03:01Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "cpu" ], "author": "ccaadaro", "state": "open", "assignees": [ "bigPYJ1151" ], "change": "updated" } }, { "id": "event:81f4e8fadaac2b1a892e", "signalId": "github:vllm-project/vllm:issue:51609", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51609", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Doc]: Stale rejection_sample_method and synthetic_acceptance_rate in speculative decoding README", "text": "### 📚 The doc issue The `--speculative-config` table in `docs/features/speculative_decoding/README.md` is out of sync with the current code: 1. `rejection_sample_method` (line 87): documented as `strict, probabilistic, synthetic` (default `strict`), but the current code is `Literal[\"standard\", \"synthetic\", \"block\"]` (default `standard`) (`vllm/config/speculative.py:79, 218`). History: #40651 (2026-04-26) renamed `strict` to `standard` and moved `probabilistic` into the `draft_sample_method` dimension (`speculative.py:80`). 2. `synthetic_acceptance_rate` (line 88, singular): the current code uses `synthetic_acceptance_rates` (list) or `synthetic_acceptance_length`, mutually exclusive (`vllm/config/speculative.py:226, 234, 258-262`). History: #40662 (2026-04-23). Impact: users following the README and passing `rejection_sample_method=\"strict\"` or `synthetic_acceptance_rate=...` hit a pydantic `ValidationError` at startup (the Literal validation in `vllm/config/speculative.py`). Related PRs (searched before filing): - #37383 (open, needs-rebase since 2026-05, no activity since 2026-03): adds a comprehensive section to the same README but documents the same stale values and a non-existent `speculative_token_tree` key - it does not fix this. - #51500 (open, typo-only): touches adjacent lines (89, 200, 220) of the same README but does not change the values on lines 87-88. ### Suggest a potential alternative/fix Sync lines 87-88 with the current `SpeculativeConfig` fields and values: - `rejection_sample_method`: `strict, probabilistic, synthetic` (default `strict`) -> `standard, synthetic, block` (default `standard`) - `synthetic_acceptance_rate`: renamed to `synthetic_acceptance_rates` (list), with `synthetic_acceptance_length` noted as mutually exclusive ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions. Update: fix PR submitted - https://github.com/vllm-project/vllm/pull/51611", "url": "https://github.com/vllm-project/vllm/issues/51609", "createdAt": "2026-08-10T03:45:16Z", "updatedAt": "2026-08-13T10:56:29Z", "timestamp": "2026-08-13T10:56:29Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "documentation" ], "author": "qwerqwerqwe8688-jpg", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:84a10a148b575012c843", "signalId": "github:vllm-project/vllm:pull_request:51611", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51611", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Fix stale rejection_sample_method and synthetic_acceptance_rate", "text": "## Purpose Fixes #51609 Sync the `--speculative-config` table in `docs/features/speculative_decoding/README.md` with the current code: 1. `rejection_sample_method` (line 87): `strict, probabilistic, synthetic` (default `strict`) -> `standard, synthetic, block` (default `standard`); `probabilistic` now belongs to `draft_sample_method`. See #40651. 2. `synthetic_acceptance_rate` (line 88): split into `synthetic_acceptance_rates` (list) and `synthetic_acceptance_length` (scalar, mutually exclusive). See #40662. ## Why not duplicating an existing PR - #37383: open but stale (needs-rebase since 2026-05, no activity since 2026-03); its new section documents the same outdated values plus a non-existent `speculative_token_tree` key. This PR takes the minimal fix of the existing table. - #51500: typo-only fixes in the same README (touches adjacent lines 89, 200, 220). It does not change the values on lines 87-88. If it merges first, I will rebase manually to resolve any adjacent-line conflicts. ## Test Plan - pre-commit run markdownlint-cli2 --files docs/features/speculative_decoding/README.md ## Test Result Markdown-only change; local environment has no Python, so markdownlint was not run locally — relying on CI check. --- ## AI Assistance This PR was drafted with AI assistance; the human submitter reviewed every changed line. Co-authored-by: deepseek-v4-flash", "url": "https://github.com/vllm-project/vllm/pull/51611", "createdAt": "2026-08-10T04:19:58Z", "updatedAt": "2026-08-13T10:56:28Z", "timestamp": "2026-08-13T10:56:28Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "ready", "build-docs" ], "author": "qwerqwerqwe8688-jpg", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a77c13a1f6b2f1924305", "signalId": "github:vllm-project/vllm:pull_request:51075", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51075", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Rust] MiniMax-M3 tool parser: accept elided first-parameter opening tag", "text": "## Purpose Fixes #51073 Deployed MiniMax-M3 checkpoints (`MiniMaxAI/MiniMax-M3-MXFP8`; chat template byte-identical on `olka-fi/MiniMax-M3-MXFP4`) deterministically emit tool-call invokes whose **first parameter is missing its opening tag**: the value sits directly between the namespace sentinel and a closing tag carrying the parameter name. Real capture (no whitespace around values; line broken here for readability only): ```text ]<]minimax[>[ ]<]minimax[>[]<]minimax[>[offerta.docx]<]minimax[>[]<]minimax[>[Hello]<]minimax[>[]<]minimax[>[]<]minimax[>[ ``` The Rust grammar in `rust/src/parser/src/tool/minimax_m3.rs` hard-fails at that parameter boundary: `parse_invoke_params` treats a namespace marker not followed by `<` as malformed (`ErrMode::Cut` -> `ToolParserError::ParsingFailed`), and the same rejection exists in nested element bodies. The whole message then degrades to raw markup content instead of a native tool call. Full root-cause analysis is in #51073. This PR closes the emission-vs-grammar mismatch with a tolerant rule (`elided_parameter_element`): - At a parameter boundary — top level of an invoke body, and inside nested element bodies — a namespace marker **not** followed by `<` starts an elided-opening-tag parameter. - Its value is the text up to the next namespace marker, which must begin the closing tag that names the parameter. - The rule is positional: the invoke body is already delimited by `]<]minimax[>[`, so every closing tag inside it unambiguously ends one parameter. Any parameter may therefore be elided, not just the first. - In nested element bodies, recovery additionally requires the discovered closing tag to differ from the parent's, which keeps the parent element properly terminated. - Recovered values flow through the existing schema-coercion path, identical to tagged parameters. Every other previously-malformed shape stays malformed, and well-formed input is unaffected. Deliberate non-goal, left as follow-up as suggested in #51073: stripping namespace sentinels from the raw-content fallback when parsing still fails unrecoverably. ## Test Plan New unit fixtures in `rust/src/parser/src/tool/minimax_m3.rs`, built from the real captures: - `minimax_m3_parse_complete_accepts_elided_first_param_opening_tag` — capture 1 reconstruction (elided first parameter + fully tagged second) - `minimax_m3_parse_complete_accepts_elided_first_param_second_capture` — capture 2 (truncated in flight at `]<]minimax[>[ integer) - `minimax_m3_parse_complete_accepts_elided_first_child_in_nested_element` — first child of a nested object parameter elided - `minimax_m3_elided_close_tag_matching_parent_stays_malformed` — parent-close guard: preserved strictness for the ambiguous shape - `minimax_m3_parse_complete_extracts_mixed_elided_and_tagged_invokes` — multi-invoke message mixing an elided-parameter invoke with a fully tagged one, plus prefix text - `minimax_m3_streaming_handles_elided_param_split_across_chunks` — 3-char chunking so every marker is split across chunk boundaries Full crate suite: ```bash cd rust/src/parser && cargo test --lib ``` ## Test Result ```text test result: ok. 423 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ``` 416 pre-existing tests stay green, plus the 7 fixtures above. `cargo fmt --check` is clean on the touched file. An equivalent tolerant rule, ported as a Python `--tool-parser-plugin`, is serving `MiniMaxAI/MiniMax-M3-MXFP8` in production on a 6-node GB10 (sm_121) cluster. Verified end-to-end after deployment: OpenAI-compatible non-streaming (`finish_reason: \"tool_calls\"`, complete JSON arguments), OpenAI-compatible streaming (tool-call deltas with full name + arguments), and Anthropic-compatible `/v1/messages` (`stop_reason: \"tool_use\"` with populated `input`) — all with clean `content` (no leaked sentinels) and zero parser errors in the server log, including under live agent traffic (multi-turn tool loops from an AnythingLLM client).", "url": "https://github.com/vllm-project/vllm/pull/51075", "createdAt": "2026-08-04T20:31:43Z", "updatedAt": "2026-08-13T10:55:51Z", "timestamp": "2026-08-13T10:55:51Z", "metrics": { "reactions": 3, "comments": 4 }, "labels": [ "bug", "rust" ], "author": "Stupidoodle", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6b9019f3eef1d6e0a1c1", "signalId": "github:vllm-project/vllm:pull_request:52142", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52142", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix standalone torch.compile cache loading after relocation", "text": "## Purpose The standalone Inductor compile cache persists artifact handles as `(key, absolute_path)`. vLLM supports reusing its `torch.compile` cache after copying it to another machine or baking it into a container image. If the cache directory changes location in that process, the artifact exists under the new cache directory, but `InductorStandaloneAdaptor.load()` still uses the persisted absolute path. If the original location is unavailable, loading fails with `FileNotFoundError`. This change reconstructs the artifact path from the adaptor's current `self.cache_dir` and the persisted artifact key (`handle[0]`) instead of loading from the stale path in `handle[1]`. The persisted handle format remains unchanged, so this fix does not require changing existing cache metadata. Fixes #52154. ## Test Plan Added a regression test covering both standalone artifact save formats: - `binary` - `unpacked` The test initializes the adaptor with a relocated cache directory while using a handle that contains the old absolute path, and verifies that `CompiledArtifact.load()` resolves the artifact from the current cache directory. ```bash python -m pytest -q \\ --confcutdir=tests/compile \\ tests/compile/test_compiler_interface.py ``` ## Test Result Before this change: ```text 2 failed Expected: .../new/artifact_shape_None_subgraph_0 Actual: .../old/artifact_shape_None_subgraph_0 ``` With this change: ```text 2 passed ``` Also verified with `git diff --check` and pre-commit on the modified files.", "url": "https://github.com/vllm-project/vllm/pull/52142", "createdAt": "2026-08-13T09:55:15Z", "updatedAt": "2026-08-13T10:54:12Z", "timestamp": "2026-08-13T10:54:12Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "jungjiyu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:42d3e31b1ab192aa2c59", "signalId": "github:vllm-project/vllm:pull_request:50665", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50665", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fuse Kimi-K3 FP8 pre-route projections", "text": "## Purpose Integrate the Kimi-K3 gfx950 pre-route primitives through one AMD-only, fail-closed ownership boundary: - prepack routed-down and shared-expert weights after checkpoint loading; - produce routed inputs, shared-expert output, and FP32 router logits in the AITER mixed-precision grid; and - pass the precomputed shared output through a scoped seam so the generic shared-expert lifecycle does not repeat the work. FP8 is opt-in with `VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8=1`; the exact BF16 path is separately opt-in, and FP8 takes precedence when both are enabled. Unsupported shapes, dtypes, devices, architectures, LoRA configurations, or AITER installations use the existing implementation. NVIDIA and other models are unchanged. Review map: - `amd/ops/moe_preroute.py`: checked AITER adapter and weight packing; - `amd/linear.py`: Kimi-K3 lifecycle and dispatch; - `shared_experts.py`: scoped precomputed-output ownership seam; - tests: operator contract, fallback, graph replay, and equal-logit behavior. Depends on ROCm/aiter#4498 and ROCm/aiter#4504. ## Test plan Tested on 8x MI355X (`gfx950`) with the public Kimi-K3 image and `moonshotai/Kimi-K3@9f62e4e9`. Kernel heads: AITER #4498 `0d755fe1` and #4504 `3d1208d4`; the command below pins the vLLM base and PR revisions. Fetch and verify the vLLM source: ```bash git clone https://github.com/vllm-project/vllm.git vllm git -C vllm fetch origin refs/pull/50665/head:pr-50665 test \"$(git -C vllm rev-parse pr-50665)\" = \\ 84a9b03c803421804eb3062de43022edce6a8992 git -C vllm checkout --detach 6c91de36897932ba9b5adb11992235a2a789e009 git -C vllm diff 6c91de36897932ba9b5adb11992235a2a789e009..pr-50665 \\ --binary | git -C vllm apply - git -C vllm diff --check ``` Construct both arms from the public image using the complete verified vLLM and AITER trees above, including AITER `csrc/` and `hsa/`; do not copy individual files. Verify `import vllm, vllm._C, aiter` in the GPU container before testing to prevent a stale or missing overlay. Focused tests and companion microbenchmarks: ```bash python -m pytest -q \\ ../aiter/op_tests/flydsl_tests/test_kimi_k3_moe_preroute_bf16.py \\ ../aiter/op_tests/flydsl_tests/test_kimi_k3_moe_preroute_fp8.py \\ tests/model_executor/layers/fused_moe/test_shared_experts.py \\ tests/models/kimi_k3/test_amd_moe_preroute.py ``` Serve each arm with the same exact model and flags; set both pre-route variables to `0` for control: ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FP4BMM=1 \\ AITER_SITUV2_A8W4=1 AITER_BF16_FP8_MOE_BOUND=0 \\ VLLM_ROCM_USE_KIMI_K3_PREROUTE_BF16=1 \\ VLLM_ROCM_USE_KIMI_K3_PREROUTE_FP8=1 \\ vllm serve /model --served-model-name moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 --trust-remote-code --moe-backend auto \\ --gpu-memory-utilization 0.95 --max-num-seqs 128 \\ --max-num-batched-tokens 4096 --max-model-len 1048576 \\ --enable-prefix-caching --kv-cache-dtype fp8 --reasoning-parser kimi_k3 ``` After one 8K/128 warmup, run three 8K/1K batch-one trials with seeds 1--3, temperature zero, and `--ignore-eos`. Run full `lm-eval==0.4.12` GSM8K on both arms: 1,319 questions, 5-shot, greedy completion, 2,048 generated tokens, concurrency 128, and seed 42. ## Test results - Combined focused suite: **21/21 passed**. - Changed-file pre-commit, `git diff --check`, and DCO passed. | MI355X microbenchmark | Control | Candidate | Speedup | | --- | ---: | ---: | ---: | | BF16 pre-route | 21.9358 us | 17.0914 us | **1.2834x** | | FP8 routed/shared output | 17.6833 us | 13.8263 us | **1.2790x** | | FP8 tri-projection | 20.3035 us | 10.6829 us | **1.9006x** | | TP8 8K/1K median | Control | Candidate | Change | | --- | ---: | ---: | ---: | | Decode throughput | 36.0596 tok/s/GPU | 38.2324 tok/s/GPU | **+6.026%** | | TPOT | 27.7319 ms | 26.1558 ms | **-1.5760 ms** | Full GSM8K: control **1270/1319**, candidate **1272/1319**, zero invalid or transport failures; 14 wins/12 losses (`p=0.8450`). This paired run found no statistically detectable accuracy difference. The public suite also exposed two exactly tied router logits (experts 111 and 184 at 189.0). The test requires the exact selected top-16 set and a strict 16/17 boundary (174.0 versus 167.0), without imposing undefined tie order. ## Overlap and limits - Operator tests establish the stated BF16/FP8 numerical contracts; GSM8K is the model-quality gate. - ROCm/aiter#4498 and ROCm/aiter#4504 own the kernels; this PR only owns vLLM lifecycle and dispatch. The two paths share one weight lifecycle, so splitting them would duplicate the model integration and shared-output ownership seam. ## Tool assistance OpenAI Codex assisted with implementation, tests, benchmarking, and drafting. ## Long-prefix agentic serving gate (2026-08-02) The exact user-requested serving sweep was run with official `aiperf==0.11.0` on 8x MI355X (TP8), FP8 KV cache, prefix caching, and no speculative decoding: ```bash aiperf profile --model moonshotai/Kimi-K3 --tokenizer moonshotai/Kimi-K3 \\ --tokenizer-trust-remote-code --url http://127.0.0.1:8000 --api-key EMPTY \\ --endpoint-type chat --streaming --use-server-token-count \\ --num-prefix-prompts 8 --prompt-prefix-length 63240 \\ --synthetic-input-tokens-mean 4760 --synthetic-input-tokens-stddev 0 \\ --output-tokens-mean 350 --output-tokens-stddev 0 \\ --extra-inputs ignore_eos:true --extra-inputs min_tokens:350 \\ --extra-inputs max_tokens:350 --warmup-request-count 3 --sweep-type zip \\ --concurrency 1,8,16,24,48 --request-count 5,40,80,120,240 \\ --random-seed 42 --ui simple ``` The parent, BF16 intermediate, and FP8 candidate each completed **485/485** requests (**1,455/1,455 total**) with byte-identical generated inputs, exactly 350 output tokens per request, and zero errors, cancellations, output-size mismatches, or endpoint fatal signatures. | Concurrency | Requests/arm | Output tok/s, parent -> BF16 -> FP8 | FP8 vs parent | Mean ITL, ms, parent -> FP8 | Mean TTFT, ms, parent -> FP8 | Mean E2E, ms, parent -> FP8 | |---:|---:|---:|---:|---:|---:|---:| | 1 | 5 | 14.12 -> 14.27 -> 14.41 | **+2.07%** | 51.70 -> 50.13 (**-3.02%**) | 6,743 -> 6,785 (+0.62%) | 24,785 -> 24,281 (**-2.03%**) | | 8 | 40 | 84.02 -> 83.23 -> 84.38 | +0.43% | 78.43 -> 78.02 (-0.53%) | 5,701 -> 5,714 (+0.23%) | 33,074 -> 32,941 (-0.40%) | | 16 | 80 | 145.22 -> 145.03 -> 145.36 | +0.09% | 85.87 -> 87.01 (+1.33%) | 7,776 -> 7,632 (-1.85%) | 37,744 -> 37,999 (+0.68%) | | 24 | 120 | 191.53 -> 191.62 -> 191.44 | -0.05% | 97.16 -> 94.91 (-2.32%) | 9,078 -> 10,069 (+10.92%) | 42,988 -> 43,191 (+0.47%) | | 48 | 240 | 296.30 -> 296.46 -> 296.51 | +0.07% | 124.75 -> 125.15 (+0.32%) | 12,306 -> 12,154 (-1.24%) | 55,845 -> 55,830 (-0.03%) | The FP8 path provides a clear concurrency-one benefit; c8-c48 is neutral. Both optimized support predicates require activation shape `(1, 7168)`, so larger scheduler batches are expected to use the existing fallback. This dispatch explanation is inferred from the source contract; no dispatch counter was collected. TTFT is queueing-sensitive, so the mixed TTFT deltas—especially c24—are reported without an improvement claim. Each cell is one profile trial and therefore a point estimate. Provenance: public image `vllm/vllm-openai-rocm:kimi-k3@sha256:5aa7e626ff73672f5ca7aae46754570488c23d33ca1ac90756a1d2d1a3fe099b`, model `moonshotai/Kimi-K3@9f62e4e9`, and random seed 42. The four runtime files at current PR head `84a9b03c803421804eb3062de43022edce6a8992` are SHA-256-identical to the measured candidate. The seven runtime files at AITER #4498 head `0d755fe1` and #4504 head `3d1208d4` are likewise byte-identical to the measured AITER stack.", "url": "https://github.com/vllm-project/vllm/pull/50665", "createdAt": "2026-08-01T08:37:37Z", "updatedAt": "2026-08-13T10:52:25Z", "timestamp": "2026-08-13T10:52:25Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "kimi", "k3" ], "author": "JohnQinAMD", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:92ca88157f2ca37e59e1", "signalId": "github:vllm-project/vllm:pull_request:47692", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47692", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix `--data-parallel-start-rank 0` being treated as unset in `create_engine_config`", "text": "## Summary `EngineArgs.create_engine_config` uses Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None` to detect whether `--data-parallel-start-rank` was explicitly set. Since `0` is a valid, meaningful starting rank (the node owning the first slice of global DP ranks), an explicit `--data-parallel-start-rank 0` is silently treated identically to \"not specified.\" This causes `data_parallel_hybrid_lb` to never be inferred `True` for that node, even though the rest of the codebase (including the sibling field `data_parallel_rank`, and the other two call sites of this same field) already use `is not None` correctly for exactly this class of field. The corrupted flag then propagates into `ParallelConfig.local_engines_only`, `wait_for_engine_startup`'s `remote_should_be_headless` computation, and the Rust-frontend engine-indexing logic added in #45805 — misclassifying the rank-0 node's role in a hybrid-LB deployment. The fix switches both truthiness checks to `is not None`, matching the established convention already used elsewhere in the same function. ## Duplicate-work check Searched for existing/overlapping work before starting: - `gh issue list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh issue list --repo vllm-project/vllm --state open --search \"data parallel rank\"` — 30 hits, none related (skimmed all titles: Ray deadlocks, MTP, PP tool-calling, RPC port robustness, etc.) - `gh issue list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank\"` — 0 relevant hits - `gh pr list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state open --search \"DP rank 0 hybrid\"` — 26 hits, none touch this logic - `gh pr list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank closed:>=2026-06-05\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state closed --search \"hybrid_lb closed:>=2026-06-05\"` — surfaced PR #45805 (merged 2026-06-17), which _consumes_ the affected `data_parallel_hybrid_lb`/`local_engines_only` flag in `serve.py` for Rust-frontend engine indexing but does not touch or fix `arg_utils.py:1997`/`2016` Zero overlapping in-flight work found across open issues, open PRs, and PRs closed in the last 30 days. ## Changes ### `vllm/engine/arg_utils.py` Line 1997, inside `create_engine_config`: ```python # Before if self.data_parallel_start_rank and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True # After if self.data_parallel_start_rank is not None and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True ``` Line 2016: ```python # Before self.data_parallel_rank = ( self.data_parallel_start_rank or inferred_data_parallel_rank ) # After self.data_parallel_rank = ( self.data_parallel_start_rank if self.data_parallel_start_rank is not None else inferred_data_parallel_rank ) ``` ### `tests/v1/engine/test_engine_args.py` Added `test_data_parallel_start_rank_zero_infers_hybrid_lb`, asserting that `data_parallel_start_rank=0` with `data_parallel_size_local=2`, `data_parallel_size=4` (headless defaults to `False`) yields `parallel_config.data_parallel_hybrid_lb is True` and `parallel_config.data_parallel_rank == 0`. Confirmed this test **fails** against the pre-fix code (verified via `git stash` on `vllm/engine/arg_utils.py` and re-running just this test) and **passes** with the fix applied — it is a genuine regression test, not a tautology. ## Test commands run and results ```bash python -m pytest tests/v1/engine/test_engine_args.py -v # 5 passed (including the new test) python -m pytest tests/engine/test_arg_utils.py tests/entrypoints/openai/test_dp_supervisor.py -v # 111 passed, no regressions pre-commit run --files vllm/engine/arg_utils.py tests/v1/engine/test_engine_args.py # ruff check: Passed # ruff format: Passed # typos: Passed # mypy (3.10): Passed # Check SPDX headers: Passed # Check root lazy imports: Passed # Check for forbidden imports: Passed # Validate configuration has default values / docstrings: Passed # (all other applicable hooks: Passed; unrelated hooks skipped — no matching files) ``` ## AI assistance disclosure This PR includes AI-assisted contributions. Claude Code was used to trace the control flow of `create_engine_config`, identify the truthiness-vs-`is not None` inconsistency, draft the fix, and draft the regression test. The human contributor independently: - Read the full `create_engine_config` function end-to-end (not just the two changed lines) to confirm no earlier code path coerces or transforms `data_parallel_start_rank` before it reaches the affected checks - Verified the sibling-field convention (`data_parallel_rank`, handled correctly with `is not None`) and the two other call sites of `data_parallel_start_rank` elsewhere in the codebase (`serve.py`, `dp_supervisor.py`), confirming both already use the correct pattern - Traced the full downstream consumer chain (`ParallelConfig.local_engines_only` → `wait_for_engine_startup`'s `remote_should_be_headless` → the Rust-frontend engine-indexing logic added in #45805) to confirm the bug has live, current consequences - Confirmed the added regression test fails on the pre-fix code and passes on the fix by manually stashing/restoring the source change and re-running the test - Ran the full test suite and pre-commit hooks locally and reviewed all output above Closes #47691", "url": "https://github.com/vllm-project/vllm/pull/47692", "createdAt": "2026-07-06T05:45:20Z", "updatedAt": "2026-08-13T10:51:53Z", "timestamp": "2026-08-13T10:51:53Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "ready", "v1" ], "author": "syedalijaseem", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:04fd08610697f31468a9", "signalId": "github:vllm-project/vllm:issue:52154", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52154", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Standalone torch.compile cache uses stale artifact path after relocation", "text": "### Your current environment
    The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 4.4.2 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cpu Is debug build : False CUDA used to build PyTorch : Could not collect ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.4.239-1.el7.elrepo.x86_64-x86_64-with-glibc2.39 ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 2 Stepping: 6 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 3 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 80 MiB (64 instances) L3 cache: 96 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-31,64-95 NUMA node1 CPU(s): 32-63,96-127 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Vulnerable Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Vulnerable Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] numpy==2.3.5 [pip3] pyzmq==27.1.0 [pip3] torch==2.13.0+cpu [pip3] torchaudio==2.11.0+cpu [pip3] torchcodec==0.15.0 [pip3] torchvision==0.28.0+cpu [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.2rc1.dev29+g50ba4bc6b (git sha: 50ba4bc6b) vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 SYS SYS PXB PXB SYS SYS SYS SYS SYS 0-31,64-95 0 N/A GPU1 NV12 X SYS SYS SYS SYS SYS PXB PXB SYS SYS 32-63,96-127 1 N/A NIC0 SYS SYS X PIX SYS SYS SYS SYS SYS SYS SYS NIC1 SYS SYS PIX X SYS SYS SYS SYS SYS SYS SYS NIC2 PXB SYS SYS SYS X PIX SYS SYS SYS SYS SYS NIC3 PXB SYS SYS SYS PIX X SYS SYS SYS SYS SYS NIC4 SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS NIC5 SYS PXB SYS SYS SYS SYS SYS X PIX SYS SYS NIC6 SYS PXB SYS SYS SYS SYS SYS PIX X SYS SYS NIC7 SYS SYS SYS SYS SYS SYS SYS SYS SYS X PXB NIC8 SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=, CUBLAS_VERSION=13.1.0.3 NVIDIA_REQUIRE_CUDA=cuda>=9.0 TORCHINDUCTOR_LOOP_ORDERING_AFTER_FUSION=0 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0+PTX NCCL_VERSION=2.28.8 NVIDIA_DRIVER_CAPABILITIES=compute,utility,video TORCH_NCCL_USE_COMM_NONBLOCKING=0 CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0 NVIDIA_PRODUCT_NAME=PyTorch CUDA_VERSION=13.0.2.006 PYTORCH_VERSION=2.10.0a0+b558c98 PYTORCH_BUILD_NUMBER=0 CUBLASMP_VERSION=0.6.0.84 CUDNN_FRONTEND_VERSION=1.15.0 CUDA_COMPONENT_LIST=cccl crt nvrtc driver-dev culibos-dev cudart cudart-dev nvcc CUDNN_VERSION=9.15.0.58 PYTORCH_HOME=/opt/pytorch/pytorch NVIDIA_BUILD_ID=231036167 CUDA_DRIVER_VERSION=580.95.05 PYTORCH_BUILD_VERSION=2.10.0a0+b558c98 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda CUDA_MODULE_LOADING=LAZY NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS= NVIDIA_PYTORCH_VERSION=25.11 TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/mnt/tmp/solar-serving-study/tmp/torchinductor_nsml ```
    ### 🐛 Describe the bug vLLM's [`torch.compile` compilation cache documentation](https://docs.vllm.ai/en/stable/design/torch_compile/#compilation-cache) documents that the `torch_compile_cache` directory can be copied in deployment scenarios to reuse compilation artifacts. For standalone Inductor compilation, an artifact handle is persisted as: ```text (artifact_key, absolute_path) ``` `InductorStandaloneAdaptor.load()` currently resolves the artifact directly from the persisted absolute path: ```python path = handle[1] ``` If the compile cache is relocated to a different cache root, the artifact moves with the cache, but the persisted handle still points to its original location. Once that location is unavailable, loading through the persisted handle attempts to access the stale path. ### Minimal reproducer The following reproduces the path-resolution behavior on vLLM main at commit `50ba4bc6b` without requiring a model or an existing compiled artifact: ```python from unittest.mock import MagicMock, patch from vllm.compilation.compiler_interface import InductorStandaloneAdaptor old_cache_dir = \"/tmp/vllm-cache-old\" new_cache_dir = \"/tmp/vllm-cache-new\" artifact_key = \"artifact_shape_None_subgraph_0\" adaptor = InductorStandaloneAdaptor(save_format=\"binary\") adaptor.initialize_cache(new_cache_dir) handle = ( artifact_key, f\"{old_cache_dir}/{artifact_key}\", ) with ( patch( \"torch._inductor.CompiledArtifact.load\", return_value=MagicMock(), ) as load_mock, patch( \"torch._inductor.compile_fx.graph_returns_tuple\", return_value=True, ), ): adaptor.load( handle=handle, graph=MagicMock(), example_inputs=[], graph_index=0, compile_range=MagicMock(), ) print(\"current cache dir:\", new_cache_dir) print(\"persisted path: \", handle[1]) print(\"path used by load:\", load_mock.call_args.kwargs[\"path\"]) ``` Observed output: ```text current cache dir: /tmp/vllm-cache-new persisted path: /tmp/vllm-cache-old/artifact_shape_None_subgraph_0 path used by load: /tmp/vllm-cache-old/artifact_shape_None_subgraph_0 ``` Although the adaptor is initialized with the relocated cache directory, `CompiledArtifact.load()` is still called with the absolute path from the original cache location. I also verified the failure with an existing standalone compiled artifact by relocating the same artifact from cache root A to cache root B: ```text ORIGINAL_PATH_LOAD = PASS STALE_PATH_LOAD = FAIL (FileNotFoundError) REBASED_PATH_LOAD = PASS ``` The artifact itself remains loadable from B after relocation. Loading through the persisted handle fails when it still references A, which isolates the stale absolute path as the failure condition. ### Expected behavior A standalone compile cache should remain loadable when copied to a different cache root for reuse and the cached artifacts are otherwise unchanged. Artifact loading should resolve the corresponding artifact from the current cache directory rather than depend on the cache's original absolute path. A fix with regression coverage for both `binary` and `unpacked` standalone artifact formats is available in #52142. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52154", "createdAt": "2026-08-13T10:51:36Z", "updatedAt": "2026-08-13T10:51:36Z", "timestamp": "2026-08-13T10:51:36Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "jungjiyu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:00029a2b66f1b2b48238", "signalId": "github:vllm-project/vllm:pull_request:51937", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51937", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Parser] Migrate Hermes tool parser to the new streaming Parser Engine", "text": "# [Parser] Migrate Hermes tool parser to the streaming Parser Engine ## Summary Migrates the Hermes tool parser from the legacy regex/diff implementation (~275 lines) to the declarative streaming Parser Engine, following the pattern already used by Qwen3, MiniMax M2, Gemma4, Kimi K2 and Inkling. - New `vllm/parser/hermes.py`: `HermesParser(ParserEngine)` + a declarative `ParserEngineConfig`. - `vllm/tool_parsers/hermes_tool_parser.py` becomes a thin bridge. The registry entry and the `Hermes2ProToolParser` class name are unchanged, so `--tool-call-parser hermes` is unaffected. - `structural_tag_model = \"hermes\"` is preserved, keeping guided decoding (and the `supports_required_and_named=False` it implies) identical. This follows the suggestion on #49535 to migrate tool parsers to the Parser Engine rather than patch the legacy implementations further. ### Why Hermes first, and not `hunyuan_a13b` / `xlam` Those two parsers emit a single undelimited JSON array (`[{...},{...}]`), where the boundary between one tool call and the next is a bare comma at the array's top nesting level — there is no terminal token. The engine's state machine only transitions on explicit terminal tokens; brace/bracket depth tracking in the engine exists only to decide which characters are safe to stream, not to drive a transition. Migrating those two needs new engine-level capability, not just a per-model config, so it's a separate design question rather than something to fold into this PR. Hermes uses the same per-call delimiter shape as the already-migrated models (``…`` repeated per call), which is proven engine capability, so it's independently actionable now. ## Root cause / design notes **Why an `arg_converter` is mandatory.** Hermes's tool-call body is a `{\"name\":..., \"arguments\":{...}}` wrapper, so the whole wrapper lands in `TOOL_ARGS`. `ParserEngine._compute_arg_delta` returns the raw delta and never calls `_fix_arg_types` when `arg_converter is None`, while non-streaming extraction always coerces via `_build_extracted_result`. Without a converter, streaming and non-streaming silently disagree on argument types. **Why a verbatim scanner rather than re-serialising.** The engine diffs successive converter outputs and requires each to extend the previous one (`startswith`); a violation silently drops argument deltas. Re-serialising through `json.dumps` changes whitespace and closes unterminated structures differently across ticks, so the only prefix-stable output is a verbatim substring of the input. `_hermes_arg_converter` therefore carves the `arguments` value out with a string/escape-aware scanner, mirroring `_inkling_arg_converter`, which has the same wrapper shape. The scanner is also what makes an argument literally named `\"name\"` safe — the bug class that started this work (#49535). **Why `token_id_terminals` is empty.** `StreamingParserEngine._process_lex_tokens` enters a strict token-id mode once any real token IDs have been seen, and refuses to accept a text-matched terminal that is listed in `token_id_terminals` unless it was also pre-lexed from an actual matching token ID. Hermes delimiters reliably arrive as literal text (`ParserEngine.adjust_request` forces `skip_special_tokens=False`), so token-ID pre-lexing buys nothing and would starve text matching for tokenizers where the delimiter is not a single true token. Inkling opts out the same way. **Non-streaming strictness is preserved explicitly.** The legacy parser required the *whole* tool-call body to parse as valid JSON and reported no tool calls at all otherwise. The engine's generic EOS path is best-effort and would happily emit a tool call from a truncated wrapper, so `HermesParser.extract_tool_calls_from_content` keeps the single-shot `json.loads` check. Streaming keeps the engine's normal best-effort behavior. **Longcat.** `LongcatFlashToolParser` subclassed `Hermes2ProToolParser` and customised the delimiters by assigning `tool_call_start_token` / `tool_call_end_token` / `tool_call_regex`, none of which the engine consults. Left as-is, the migration would have silently broken it (13 of its 17 tests). It now has its own `HermesParser` subclass overriding the delimiters as class attributes, which is why `TOOL_CALL_START`/`TOOL_CALL_END` are class attributes on `HermesParser` at all. ## Test-suite change, and why `tests/parser/test_streaming.py` used Hermes as its reference **non-engine** tool parser (a section titled \"Engine-based reasoning + non-engine tool parser\", asserting `engine_based_streaming is False`). This migration invalidates that premise by design. Those tests now use `Granite4ToolParser`, which is still non-engine, uses the identical ``/`` delimiters so `MODEL_OUTPUT` is unchanged, and has `supports_required_and_named=True` natively — which also removes the need for the `autouse` fixture that was force-setting that flag on Hermes. `test_parse_delta_finished_appends_remaining_args` still uses Hermes explicitly, since its subject is that parser's unstreamed-args behavior. No test was weakened or deleted; only the reference model changed. ## Out of scope (pre-existing, not introduced here) While validating the above I found that `tool_choice=\"required\"` crashes for **any** engine-based parser whose `supports_required_and_named` is `True`: `StreamState.advance()` passes only the latest delta for engine-based parsers, but `extract_required_tool_call_streaming()` expects the full accumulated text, so the final chunk reaches `partial_json_parser` as a bare `]` and raises `IndexError: pop from empty list`. This reproduces on the already-merged `KimiK2ToolParser` with no Hermes involvement, so it is not caused by this PR. It is normally unreachable because every engine-based parser sets `supports_required_and_named=False` via `structural_tag_model` + `VLLM_ENFORCE_STRICT_TOOL_CALLING` (default on), which Hermes continues to do here. Happy to file it separately. ## Not a duplicate - `gh pr list --repo vllm-project/vllm --search \"hermes parser engine\" --state all` and `--search \"migrate parser engine in:title is:open\"` — no PR migrates Hermes. In-flight migrations cover other models (Kimi K3 #50229, llama3_json/llama4_json #51577, Granite #49648, Olmo3 #48160, poolside_v1 #47167). - `vllm/parser/hermes.py` does not exist on `main`; the registry still points at the legacy `hermes_tool_parser` / `Hermes2ProToolParser`. - Three open PRs touch `vllm/tool_parsers/hermes_tool_parser.py` (#48864, #45310, #48353). This PR replaces that file wholesale, so it will conflict with them textually. Flagging explicitly so maintainers can sequence them — #48353 in particular changes tool-call boundary semantics, and if it is preferred first I am happy to rebase on top of it. - `gh pr list --search \"longcat\"` — no open PR touches `vllm/tool_parsers/longcat_tool_parser.py` (#45773 is the Rust frontend). ## Test plan Environment note: run offline against a torch-stubbed environment, so the CUDA/engine-dependent top-level `tests/conftest.py` is skipped via `--confcutdir=tests/tool_parsers`. ``` pytest tests/tool_parsers/test_hermes_tool_parser.py \\ tests/tool_parsers/test_longcat_tool_parser.py \\ tests/parser/test_parse.py \\ tests/parser/test_streaming.py \\ tests/parser/test_include_reasoning.py \\ -q --confcutdir=tests/tool_parsers ``` Result: **105 passed, 1 skipped, 1 xfailed** (the skip and xfail are pre-existing Granite4-specific cases, unchanged by this PR). Per-suite, before vs after this change: | Suite | Before | After | | --- | --- | --- | | `tests/tool_parsers/test_hermes_tool_parser.py` | 30 passed, 1 skipped | 34 passed, 1 skipped | | `tests/tool_parsers/test_longcat_tool_parser.py` | 16 passed, 1 xfailed | 16 passed, 1 xfailed | | `tests/parser/test_parse.py`, `test_streaming.py`, `test_include_reasoning.py` | 55 passed | 55 passed | The 4 added Hermes tests are one new case (`test_hermes_streaming_tool_call_with_name_key_argument`) across the existing `stream_interval` parametrisation (2/3/5/8), covering an argument literally named `\"name\"` — the bug class from #49535 — at multiple chunk granularities. Lint: `pre-commit` hooks ran on commit — `ruff check`, `ruff format`, and `mypy` for Python 3.10 all passed. ### End-to-end, against a real server and model Run separately on a T4 GPU, since the suite above is offline-only: ``` pytest tests/entrypoints/tool_parsers/test_hermes_tool_parser.py -k llama -v -s ``` Result: **4 passed, 4 deselected** (the deselected cases are the `granite4` config). This boots a real `vllm serve` with `--tool-call-parser hermes` against `meta-llama/Llama-3.2-1B-Instruct` plus the `minpeter/LoRA-Llama-3.2-1B-tool-vllm-ci` adapter, and drives it through the `openai` client — covering non-streaming, streaming, and parallel tool calls. The streaming case confirms argument type coercion survives the migration end to end, reconstructing `{'inserted': True, 'product_id': 7355608}` — a real boolean and a real integer rather than strings, which is the divergence the `arg_converter` exists to prevent. Longcat has no e2e suite upstream, so its coverage here is the 17 unit tests only — worth noting since it was changed as a consequence of the Hermes migration rather than being its target.", "url": "https://github.com/vllm-project/vllm/pull/51937", "createdAt": "2026-08-12T07:27:24Z", "updatedAt": "2026-08-13T10:48:19Z", "timestamp": "2026-08-13T10:48:19Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "tool-calling" ], "author": "muhammadfawaz1", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:06a34c8c1e28cc7a504f", "signalId": "github:vllm-project/vllm:pull_request:52153", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52153", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Pass model EOS ids to xgrammar tokenizer info", "text": "## Purpose Fixes #52146. `XgrammarBackend.__post_init__` built xgrammar `TokenizerInfo` from the tokenizer's single `eos_token_id`. Models can declare multiple EOS ids in `generation_config.json`, and vLLM's sampling path can stop on those ids. If xgrammar does not know about the full EOS set at tokenizer-info construction time, one of those model EOS tokens can remain samplable inside a structured-output string and truncate otherwise valid JSON. This PR unions the tokenizer EOS id with `model_config.try_get_generation_config()[\"eos_token_id\"]` and passes that set into both xgrammar tokenizer-info construction paths: - `xgr.TokenizerInfo(...)` for Mistral/Tekken tokenizers. - `xgr.TokenizerInfo.from_huggingface(...)` for Hugging Face tokenizers. The helper tolerates absent, duplicate, and malformed generation-config EOS values and preserves the tokenizer EOS fallback. Duplicate-work / coordination checks run before opening this PR: - `gh issue view 52146 --repo vllm-project/vllm --comments`: issue open; I posted a coordination comment because the issue body mentions a patch branch. - `gh pr list --repo vllm-project/vllm --state open --search \"52146 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"xgrammar generation_config eos_token_id stop_token_ids TokenizerInfo\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched Python files. - Focused helper assertions for multi-id, single-id, duplicate, absent, and malformed generation-config EOS values. - Focused pytest selection for the new helper tests. - Ruff format/check on touched files. - Whitespace diff check. No model evals were run; this changes stop-token metadata used by structured-output grammar masking, not model quality or accuracy. ## Test Result - `./.venv/bin/python -m py_compile vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `./.venv/bin/python -c \"... _model_stop_token_ids helper assertions ...\"`: passed. - `./.venv/bin/pre-commit run ruff-format --files vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files vllm/v1/structured_output/backend_xgrammar.py tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py`: passed. - `git diff --check`: passed. - `./.venv/bin/python -m pytest tests/v1/structured_output/test_backend_xgrammar_stop_tokens.py -k model_stop_token_ids -v`: first selected case reported `PASSED`, then local macOS test process exited with code 139 during global teardown in `torch.accelerator.memory.empty_host_cache()` via `tests/conftest.py::cleanup_fixture`. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52153", "createdAt": "2026-08-13T10:46:42Z", "updatedAt": "2026-08-13T10:47:20Z", "timestamp": "2026-08-13T10:47:20Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "structured-output" ], "author": "danielpolimac", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5af0a5c17fa52d79c4ac", "signalId": "github:vllm-project/vllm:pull_request:50903", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50903", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kernel] Fix divergent warp collectives in partial NeoX QK-Norm+RoPE", "text": "## Summary ## Author: Drew Jin, SJTU & Qwen Team, Solving Issue: https://github.com/vllm-project/vllm/issues/51049 This PR fixes silent Q/K corruption in the partial-NeoX path of `fused_qk_norm_rope`. When `rotary_dim < head_dim`, the multi-head-per-warp kernel produces incorrect Q/K values for every head after the first in each warp chunk. On H200, `token_heads_per_warp=8` can additionally produce NaNs. Verified against `upstream/main` at commit `005fa017566177e1a66d75eba7908d326c5acd60`. ## Root cause In the NeoX branch, only the rotary lanes entered the RoPE section, while the branch invoked warp collectives using the full-warp mask: ```cpp if (laneId < rotary_lanes) { __syncwarp(); rotated = __shfl_xor_sync(FINAL_MASK, value, pairOffset); __syncwarp(); } ``` For partial RoPE, `rotary_lanes < warpSize`. Therefore, `FINAL_MASK` names lanes that do not execute the collective, which is undefined behavior. `warpReduceSum` itself is not the issue. In the packed kernel, the invalid RoPE collective occurs before the next head’s RMSNorm reduction, causing the observed corruption in subsequent heads. The single-head kernel contains the same latent UB, but it has no following packed-head iteration that exposes the failure. ## Fix The NeoX shuffle is now executed by every lane named by `FINAL_MASK`. Only rotary lanes consume the shuffled value, access the cos/sin cache, and update the rotated elements. The fix is applied to both the single-head and multi-head-per-warp kernels. The implementation retains the platform-specific `FINAL_MASK` rather than constructing a narrowed 32-bit mask. This preserves the existing CUDA and ROCm wave64 mask semantics and avoids undefined expressions such as `1u << 32`. ## Tests Added regression coverage for the previously untested packing paths: - Force `token_heads_per_warp` to `1`, `2`, `4`, and `8`, and compare each variant directly against the existing unfused PyTorch reference. - Cover `head_dim ∈ {64, 128, 256}`, `rotary_ratio ∈ {1.0, 0.5, 0.25}`, NeoX and interleaved layouts, and fp16/bf16. - Verify that the V segment remains bitwise unchanged. - Verify that packed and auto-selected outputs are bitwise identical to the single-head implementation. - Exercise the SM90 auto-selected pack4 and pack8 paths with large-token inputs. Verified on NVIDIA H200 (SM90). `compute-sanitizer` racecheck and synccheck are clean. Because these tools do not report this divergent-mask UB, the regression is enforced by numerical tests. ## Impact The QK-Norm+RoPE Inductor fusion is currently opt-in and disabled by default, so this issue does not affect default vLLM inference. The observed packed-head corruption requires all of the following: 1. per-head Q/K RMSNorm routed through this fused operator; 2. NeoX-style RoPE; 3. partial RoPE (`rotary_dim < head_dim`); and 4. dispatch to the multi-head-per-warp kernel. On SM90 with `head_dim <= 128`, the dispatcher at the target commit uses: ```text U = num_tokens × (num_q_local + num_kv_local) U < 10240 → token_heads_per_warp = 1 10240 ≤ U < 40960 → token_heads_per_warp = 4 U ≥ 40960 → token_heads_per_warp = 8 ``` Large prefills can therefore enter the affected packed paths. Confirmed partial-NeoX configurations include GLM-4.5/4.6/4.7, Ring-1T, and the partial-RoPE attention layers in Step-3.5-Flash and Laguna-XS.2. Full RoPE, interleaved RoPE, and models not routed through this fused operator do not trigger this issue. ## Performance The fix also removes the slowdown observed in the affected packed path. Benchmark configuration: ```text GPU: NVIDIA H200 (SM90) PyTorch: 2.11.0+cu130 CUDA: 13.0 dtype: bf16 head_dim: 128 Q heads: 16 KV heads: 4 Timing: CUDA events Warmup iterations: 30 Measured iterations: 300 Statistic: latency per call ``` | Tokens | Pre-fix auto | Post-fix auto | Latency reduction | |---:|---:|---:|---:| | 2,048 | 22.5 µs | 15.9 µs | 29% | | 8,192 | 68.5 µs | 43.9 µs | 36% | | 16,384 | 125.6 µs | 80.1 µs | 36% | The single-head path is essentially unchanged. ## ROCm ROCm was not tested. The patch preserves the existing platform-specific `FINAL_MASK` and wave64-compatible code path; it does not introduce a CUDA-specific 32-bit active mask. ## AI assistance This PR was developed with AI assistance. I reviewed all changed lines, validated the behavior on H200, and understand the implementation and test coverage.", "url": "https://github.com/vllm-project/vllm/pull/50903", "createdAt": "2026-08-03T18:27:39Z", "updatedAt": "2026-08-13T10:45:29Z", "timestamp": "2026-08-13T10:45:29Z", "metrics": { "reactions": 2, "comments": 17 }, "labels": [ "bug", "ready" ], "author": "drewjin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0cef1d256109c2f9ea74", "signalId": "github:vllm-project/vllm:pull_request:51029", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51029", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Forward SwiGLU clamp/alpha/beta in compressed-tensors W4A4 MXFP4 MoE", "text": "## Purpose Fixes #51028 Models whose MoE uses the clamped SwiGLU-OAI activation (e.g. MiniMax-M3 in compressed-tensors MXFP4 W4A4 form, `olka-fi/MiniMax-M3-MXFP4`) crash at startup with: ``` RuntimeError: Worker failed with error 'SWIGLUOAI_UNINTERLEAVE requires clamp_limit' ``` `CompressedTensorsW4A4Mxfp4MoEMethod.get_fused_moe_quant_config` never forwards the layer's `swiglu_limit` / `swiglu_alpha` / `swiglu_beta` into the `FusedMoEQuantConfig`, so `gemm1_clamp_limit` / `gemm1_alpha` / `gemm1_beta` stay `None` and the clamp assert in `apply_moe_activation` (`vllm/model_executor/layers/fused_moe/activation.py`) trips. All four sibling compressed-tensors MoE methods (`w4a4_nvfp4`, `w8a8_fp8`, `w8a8_mxfp8`, `wna16`) already forward these via the `getattr(layer, \"swiglu_*\", None)` idiom; only the W4A4 MXFP4 method was missed. Branch-by-branch: 1. **Marlin / W4A16 branch**: `make_mxfp4_moe_quant_config` (`oracle/mxfp4.py`) already accepts `gemm1_alpha` / `gemm1_beta` / `swiglu_limit` and threads them into every backend config, and `MarlinExperts` already consumes them — the fix is to pass the three kwargs at the call site. 2. **CUTLASS W4A4 branch**: `mxfp4_moe_quant_config` (`fused_moe/config.py`) accepted only `w1_scale`/`w2_scale`. Extended its signature with `gemm1_alpha` / `gemm1_beta` / `gemm1_clamp_limit` (all defaulting to `None`, mirroring `nvfp4_w4a16_moe_quant_config` / `int4_w4a16_moe_quant_config` in the same file), passed through to `FusedMoEQuantConfig.make(...)`. Zero behavior change for the only other caller (`inc_mxfp4_moe.py`). Consumption needs no changes on current main: `ApplyMoEActivationConfig` already reads `gemm1_clamp_limit` / `gemm1_alpha` / `gemm1_beta` straight off the quant config in `from_configs`, it is built centrally in `FusedMoEModularKernel`, `run_cutlass_moe_mxfp4` already threads `activation_config` into `apply_moe_activation`, and `CutlassExpertsMxfp4._supports_activation` already delegates to `apply_moe_activation_supported`, which includes `SWIGLUOAI_UNINTERLEAVE`. So once the quant config carries the three values, the existing activation plumbing picks them up end to end. (An earlier revision of this PR wired those values through `cutlass_moe.py` by hand; the `ApplyMoEActivationConfig` refactor has since made that redundant, and this PR has been rebased down to construction-time plumbing only.) Notes: - Construction-time plumbing is the only reliable fix location: the config is consumed at kernel construction in `process_weights_after_loading`, so post-hoc mutation is not a viable workaround, and construction-time forwarding matches all sibling methods. - The fused `SILU` + `silu_and_mul_mxfp4_experts_quant` fast path in `run_cutlass_moe_mxfp4` is deliberately left untouched; a hypothetical `SILU`-with-clamp config on this path is out of scope here and is the class of problem the per-activation `supports_swiglu_clamp_limit` interface (#43589) is designed to surface. This PR does not overlap #43589's files. - Same fix family as #46845 (CT FP8, now merged — this PR's unit test sits directly alongside the one it added) and open PRs #47552 (int8), #49473 (ModelOpt FP8), #49854 (ModelOpt NVFP4), plus closed issues #48493 / #45624. ## Test Plan - New CPU unit test `tests/quantization/test_compressed_tensors.py::test_compressed_tensors_w4a4_mxfp4_moe_forwards_swiglu_params`, mirroring the pattern established by #46845: constructs the quant method directly and asserts both branches (CUTLASS W4A4 and Marlin W4A16) forward `gemm1_alpha` / `gemm1_beta` / `gemm1_clamp_limit` into the returned `FusedMoEQuantConfig`. - `ruff check` / `ruff format --check` (ruff 0.14.0, matching `.pre-commit-config.yaml`) pass on all touched files. - End-to-end: `vllm serve olka-fi/MiniMax-M3-MXFP4 --load-format instanttensor -tp 4 --distributed-executor-backend ray` on NVIDIA GB10 (`sm_121`, aarch64, DGX OS 7.5). Maintainers: which CI label should be applied for the CUTLASS MXFP4 MoE path (Blackwell runner)? Happy to adjust test placement if a kernels-level test is preferred. ## Test Result - Unit test asserts pass for both branches (CPU-only, no GPU required). - Startup crash is gone and generation quality is correct with the fix applied: a functionally equivalent patch (same three values populated on the quant config) has been serving MiniMax-M3-MXFP4 in production on a GB10 cluster via the CUTLASS W4A4 branch since 2026-08-04 against `0.26.1rc1.dev247+ge92dc7a9c`, with correct outputs. Without the fix, the server crashes at startup with `RuntimeError: Worker failed with error 'SWIGLUOAI_UNINTERLEAVE requires clamp_limit'`.", "url": "https://github.com/vllm-project/vllm/pull/51029", "createdAt": "2026-08-04T13:38:38Z", "updatedAt": "2026-08-13T10:44:07Z", "timestamp": "2026-08-13T10:44:07Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "nvidia", "quantization" ], "author": "Stupidoodle", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7ff8d4f3eb091b2f8c3d", "signalId": "github:vllm-project/vllm:issue:52109", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52109", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][ROCm/gfx942]: DeepSeek-V4-Flash silent retrieval corruption for prompts ≥ ~4-5k tokens (AITER sparse indexer)", "text": "## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Image: `vllm/vllm-openai-rocm:nightly` (2026-08-12, v0.26.1rc1.dev668+g3ee2df303) - plus local backports of #51821 (merged 08-13) and #52058 / #51252 (open) — reproduces identically without the two open-PR backports - Model: `deepseek-ai/DeepSeek-V4-Flash-0731`, TP=8, `--kv-cache-dtype fp8_ds_mla`, `--max-num-batched-tokens 16384`, attention backend `DEEPSEEK_SPARSE_SWA` ## 🐛 Describe the bug Needle-in-haystack retrieval works perfectly for short prompts and collapses to 0/3 somewhere between **3,611 tokens (3/3 retrieved)** and **5,294 tokens (0/3)**. Above the threshold the model claims the needles do not exist and output quality degrades broadly (often rambling to `max_tokens`). No crash, no error — **healthy server, silently wrong results**. GSM8K-style short-prompt correctness is unaffected (facts probes pass). Measured (3 needles at 10%/50%/90% depth, `temperature=0`): | prompt tokens | retrieved | |---|---| | 1,913 | 3/3 | | 3,611 | 3/3 | | 5,294 | 0/3 | | 7,003 / 10,411 / 12,061 / 27k / 84k / 253k / 506k | 0/3 (10,411 once gave 2/3 with a corrupted digit string) | ## Ruled out experimentally - `max_model_len` (fails identically at 131,072 and 1,048,576) - chunked-prefill boundary (`max_num_batched_tokens=16384`; 12k prompts are single-chunk and still fail) - #51252 (sparse-indexer prefill buffer budget) — backported, no change on this path - AITER `fp8_mqa_logits` split-KV heuristic (`seq_len_kv >= 4096` → splits>1): forced `num_splits=1`, no change Looks like the gfx942 sibling of #40018 (gfx950 `ROCM_AITER_MLA_SPARSE` garbage for prompt_len > ~20K), with a lower threshold (~4-5k). ## Repro Self-contained script (generates haystack, inserts 3 needles, checks retrieval): expand below. Run against an OpenAI-compatible endpoint: `python3 needle_test.py 6000` → RETRIEVED 0/3; `... 4000` → 3/3.
    needle_test.py ```python #!/usr/bin/env python3 \"\"\"長上下文 needle 測試:產生約 targetTok 的 haystack,插 3 根 needle(10%/50%/90% 深度),要求模型取回。\"\"\" import json, time, random, urllib.request, sys KEY = sys.argv[1] TARGET_TOK = int(sys.argv[2]) if len(sys.argv) > 2 else 600_000 MODEL = sys.argv[3] if len(sys.argv) > 3 else \"deepseek-v4-flash\" random.seed(42) subjects = [\"The quarterly report\", \"A municipal committee\", \"The research station\", \"An old lighthouse\", \"The logistics team\", \"A regional archive\", \"The observatory\", \"A harbor authority\", \"The botanical survey\", \"An engineering guild\"] verbs = [\"documented\", \"reviewed\", \"misplaced\", \"catalogued\", \"audited\", \"transferred\", \"digitized\", \"inspected\", \"renovated\", \"commissioned\"] objs = [\"seventeen ledgers\", \"a set of brass instruments\", \"the annual rainfall data\", \"three shipping manifests\", \"the訪客紀錄\", \"a collection of maps\", \"the maintenance schedule\", \"several personnel files\", \"the calibration records\", \"an inventory of spare parts\"] years = list(range(1951, 2026)) NEEDLES = { \"ZEPHYR-CODE\": \"738291\", \"MARLIN-KEY\": \"460517\", \"ONYX-TOKEN\": \"092384\", } def sentence(): return f\"{random.choice(subjects)} {random.choice(verbs)} {random.choice(objs)} in {random.choice(years)}. \" # 估 1 token ≈ 4 chars(英文混雜),目標字元數: target_chars = TARGET_TOK * 4 parts, n = [], 0 while n < target_chars: s = sentence() parts.append(s) n += len(s) keys = list(NEEDLES.items()) for frac, (k, v) in zip((0.10, 0.50, 0.90), keys): idx = int(len(parts) * frac) parts.insert(idx, f\"[IMPORTANT] The secret value of {k} is {v}. Remember it. \") haystack = \"\".join(parts) prompt = (haystack + \"\\n\\n---\\nFrom the document above, report the secret values of ZEPHYR-CODE, MARLIN-KEY, and ONYX-TOKEN. \" \"Answer in exactly this format:\\nZEPHYR-CODE=\\nMARLIN-KEY=\\nONYX-TOKEN=\") print(f\"haystack chars: {len(haystack):,}\", flush=True) body = {\"model\": MODEL, \"messages\": [{\"role\": \"user\", \"content\": prompt}], \"max_tokens\": 4000, \"temperature\": 0} req = urllib.request.Request(\"http://127.0.0.1:8000/v1/chat/completions\", json.dumps(body).encode(), {\"Authorization\": f\"Bearer {KEY}\", \"content-type\": \"application/json\"}) t0 = time.time() with urllib.request.urlopen(req, timeout=3600) as r: d = json.load(r) dt = time.time() - t0 u = d[\"usage\"] content = d[\"choices\"][0][\"message\"].get(\"content\") or \"\" print(f\"prompt_tokens={u['prompt_tokens']:,} completion_tokens={u['completion_tokens']} time={dt:.0f}s\", flush=True) print(\"--- model answer ---\") print(content[-500:]) ok = sum(1 for k, v in NEEDLES.items() if v in content) print(f\"RETRIEVED {ok}/3\") ```
    ", "url": "https://github.com/vllm-project/vllm/issues/52109", "createdAt": "2026-08-13T06:42:20Z", "updatedAt": "2026-08-13T10:42:48Z", "timestamp": "2026-08-13T10:42:48Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm" ], "author": "zzw09773", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:da06715c225024f7bca7", "signalId": "github:vllm-project/vllm:pull_request:52151", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52151", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix fused add RMSNorm residual-sum variance", "text": "## Purpose Fixes #52104. The CUDA `vllm_c` fused-add RMSNorm kernel was computing variance from the low-precision rounded `input + residual` value. The native IR implementation keeps `input.float() + residual.float()` in FP32 through variance calculation, then rounds only the published residual output. This PR aligns the vectorized and generic `fused_add_rms_norm` CUDA kernel paths with that native arithmetic contract: - First pass computes variance from the FP32 residual sum without mutating `residual`. - Second pass recomputes the FP32 sum, writes the rounded residual output, and normalizes from the FP32 sum. - Adds a focused `vllm_c` regression test that checks the output is closer to the FP32-sum contract than to the old rounded-sum contract. Duplicate-work checks run before opening this PR: - `gh issue view 52104 --repo vllm-project/vllm --comments`: issue open; one comment suggesting regression coverage. - `gh pr list --repo vllm-project/vllm --state open --search \"52104 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"fused_add_rms_norm residual sum variance BF16 FP32 rounding\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched Python test file. - Focused pytest selection for the new regression test. - Ruff format/check on the touched Python test file. - Clang-format on the touched CUDA file. - Whitespace diff check. No model evals were run; this is a kernel arithmetic consistency fix, not a model-output-quality change. Full CUDA validation should be run in CI because the local host has no GPGPU. ## Test Result - `./.venv/bin/python -m py_compile tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/python -m pytest tests/kernels/ir/test_layernorm.py -k fp32_sum_for_variance -v`: selected 2 tests; both skipped locally because this macOS host has no GPGPU. - `./.venv/bin/pre-commit run ruff-format --files tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files tests/kernels/ir/test_layernorm.py`: passed. - `./.venv/bin/pre-commit run clang-format --files csrc/libtorch_stable/layernorm_kernels.cu`: passed. - `git diff --check`: passed. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52151", "createdAt": "2026-08-13T10:42:29Z", "updatedAt": "2026-08-13T10:42:39Z", "timestamp": "2026-08-13T10:42:39Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "danielpolimac", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:af4600024f254eb17b8a", "signalId": "github:vllm-project/vllm:issue:52111", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52111", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeek-V4-Flash-0731 MTP speculative decoding fails to load: KeyError 'model.layers.43.mtp_block.main_norm.weight'", "text": "## Environment - 8× MI325X (gfx942), `vllm/vllm-openai-rocm:nightly` (2026-08-12, g3ee2df303) - `deepseek-ai/DeepSeek-V4-Flash-0731` (config has `num_nextn_predict_layers: 1`), TP=8 ## 🐛 Describe the bug Enabling self-speculative MTP: ``` --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' ``` Architecture resolves to `DeepSeekV4MTPModel`, then every worker dies during weight loading with: ``` KeyError: 'model.layers.43.mtp_block.main_norm.weight' ``` Looks like the MTP loader's expected parameter naming doesn't match this checkpoint's actual MTP tensor names. Serving without speculative config works fine (with #51821).", "url": "https://github.com/vllm-project/vllm/issues/52111", "createdAt": "2026-08-13T06:42:32Z", "updatedAt": "2026-08-13T10:42:23Z", "timestamp": "2026-08-13T10:42:23Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm" ], "author": "zzw09773", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9deb85d522209e01120c", "signalId": "github:vllm-project/vllm:issue:52150", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52150", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][ROCm/gfx942]: GLM-5.2-FP8 — first request after GPU idle emits garbage; piecewise CUDA graph cold replay corrupts the request's own prefill (workaround: cudagraph_mode=FULL_DECODE_ONLY)", "text": "## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Images: `vllm/vllm-openai-rocm:nightly` 2026-08-12 (v0.26.1rc1.dev668+g3ee2df303) **and** 2026-08-13 (`8f515d1d`) — both reproduce - Model: `zai-org/GLM-5.2-FP8`, TP=8, `--kv-cache-dtype auto`, `--max-num-batched-tokens 4096`, MoE backend AITER blockscale-FP8 (`fmoe_bf16_a16_blockscaleFp8...`) - Reproduces identically at `--max-model-len` 131072 and 1048576 ## 🐛 Describe the bug After the engine sits idle for ≥ ~20–50 s, the **next** request returns garbage from its very first sampled token, e.g. `'!0.1.5,0.1 and1.1.1.0 21.1.'` (token-0/digit noise). Requests sent immediately afterwards are correct. Reproduction probability rises with idle time (~50–100 % at ≥ 50 s idle; back-to-back requests never fail, dozens of samples). Key observations: 1. **The corrupt request never self-heals**: with `max_tokens=300` the output is noise from token 1 through token 300, while a request issued ~0.5 s later is clean — i.e. the corrupt request's *own prefill* wrote garbage into its KV/indexer cache, and every decode step then consumes that poisoned context. 2. `temperature=0`; prompts are ~6–10 tokens (below the 16-token block size, so prefix caching is not involved). 3. No errors in the server log, container healthy, RAS/ECC counters unchanged. 4. Not JIT/autotune related: Triton JIT warnings (`_cp_gather_indexer_quant_cache_kernel`, `_fp8_mqa_logits_kernel`) appear only on the very first request after startup; later corrupt events have no JIT activity. 5. A 25 s-interval keepalive (unique prompts forcing real prefills, all HTTP 200) does **not** reliably prevent it. ## Bisection | config | result | |---|---| | defaults (FULL_AND_PIECEWISE graphs), 131k & 1M, nightly 08-12 & 08-13 | ❌ corrupt | | `--enforce-eager` | ✅ clean 8/8 (startup burst + 60 s idle ×3 + 25 s idle ×2) | | `--compilation-config='{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}'` | ✅ clean 8/8 at 1M | → The corruption tracks **piecewise CUDA/HIP graph replay in the prefill path** after a cold window. With prefill graph-free (eager or `FULL_DECODE_ONLY`) the model is fully healthy: fact probes pass, tool calling OK, needle-in-haystack 3/3 at 2k/6k/12k/100k/900k tokens, and a 5-minute cold soak followed by a fresh request is correct. Other models on the same host/stack (Kimi-K3 with eager prefill + FULL decode graphs, DeepSeek-V4-Flash-0731) do not show this — consistent with the piecewise-prefill-graph attribution: decode-side graphs essentially never replay cold, because the preceding prefill has just warmed the GPU. This smells like a ROCm/HIP graph runtime issue (queue/doorbell state after idle → first graph replay misfires) rather than a model bug, but filing here since the practical impact is \"GlmMoeDsa default config silently corrupts on gfx942\" and the workaround is a vLLM flag. Related (same class — HIP graph replay silently wrong on AMD — but none match the idle-triggered gfx942 manifestation): - ROCm/hip#3887 — HIP graph replay reads stale device-pointer values (gfx1201/RDNA4, ~200+ nodes) - ROCm/aiter#3632 — W4A4 fused MoE kernel not HIP-graph-safe (gfx950) - pytorch/pytorch#155684 — HIP graph misuse yields silent wrong results where CUDA raises ## Workaround `--compilation-config='{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}'` — no measurable decode-speed loss (decode keeps FULL graphs; only prefill runs graph-free). Single-stream decode on this host: 44.5 tok/s (prose) / 61.9 tok/s (code) at 1M `max_model_len`. ## Repro 1. Serve `zai-org/GLM-5.2-FP8` on 8×gfx942 with the default compilation config. 2. `sleep 60` 3. ```bash curl -s localhost:8000/v1/completions -H 'content-type: application/json' \\ -d '{\"model\":\"glm\",\"prompt\":\"The capital of France is\",\"max_tokens\":16,\"temperature\":0}' ``` → garbage (`!0.1...`). Repeat immediately → ` Paris.` (correct). Vary the prompt each round to rule out prefix-cache effects; alternate idle/burst to see the pattern.", "url": "https://github.com/vllm-project/vllm/issues/52150", "createdAt": "2026-08-13T10:37:11Z", "updatedAt": "2026-08-13T10:42:04Z", "timestamp": "2026-08-13T10:42:04Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm", "kimi" ], "author": "zzw09773", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8abff4c0f1f147d9daf3", "signalId": "github:vllm-project/vllm:pull_request:52131", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52131", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Move api_server.py out openai folder", "text": "## Purpose Following https://github.com/vllm-project/vllm/pull/41907 > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. We'd better move api_server.py out openai folder ## Test Plan ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52131", "createdAt": "2026-08-13T08:39:15Z", "updatedAt": "2026-08-13T10:41:26Z", "timestamp": "2026-08-13T10:41:26Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "documentation", "performance", "frontend" ], "author": "noooop", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0817f686d917069ac6da", "signalId": "github:vllm-project/vllm:pull_request:52136", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52136", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add `pydocstyle` to the `ruff` rules", "text": "Enables the `D` ruleset in ruff and fixes the resulting violations across the codebase. ### Config `D100`–`D107` are ignored, so this does **not** require a docstring on anything that lacks one today. Six more rules are ignored because their remaining violations were either impossible to autofix or actively wrong to autofix: | Rule | Why skipped | |---|---| | `D205`, `D400`, `D415` | All treat a summary sentence that merely *wraps* as summary-plus-body. The autofix appends periods to list items. | | `D401`, `D404` | Flag idiomatic noun-phrase docstrings on properties and attributes. | | `D301` | Churn-only (`r\"\"\"` prefixes). | Plus `ignore-var-parameters = true` (no `*args`/`**kwargs` descriptions required) and a `tests/**` ignore for `D417`, since test params are mostly pytest fixtures. ### Changes The bulk is autofix output for the mechanical rules (`D209`, `D212`, `D200`, `D202`, `D413`, and friends). On top of that, by hand: - **~290 `Args:` entries** written across 88 files for `D417`, mostly in `fused_moe`, attention backends, and multimodal models. Several were format bugs rather than missing docs: numpy-style `Parameters`/`----------` blocks that ruff's Google parser can't read at all. - **807 trailing periods** for `D400`/`D415`, applied only where `D205` doesn't co-fire, so no list items were punctuated. - **37 `E501` regressions** fixed. These were introduced by the `D212` autofix collapsing wrapped summaries onto line 1 and pushing them past 88 chars. - A handful of one-offs: `D210`, `D402`, `D418`, `D419`. Incidental fixes picked up along the way: typos (`videoand`, `output_dytpe`, `the m odel`) and stale docs naming parameters that no longer exist.", "url": "https://github.com/vllm-project/vllm/pull/52136", "createdAt": "2026-08-13T09:16:21Z", "updatedAt": "2026-08-13T10:40:32Z", "timestamp": "2026-08-13T10:40:32Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "build-docs", "kimi", "k3" ], "author": "hmellor", "state": "open", "assignees": [ "sangstar" ], "change": "updated" } }, { "id": "event:66e6a9f3622b0069fe78", "signalId": "github:vllm-project/vllm:issue:51771", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51771", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "EAGLE/MTP block drop + prefix caching is untested for hybrid models with ≥3 attention groups (DeepSeek-V4-Flash + DSpark lands there)", "text": "## Summary #33524 fixed the \"0% prefix cache hits with hybrid attention + EAGLE\" bug for the 2-attention-group (GPT-OSS) shape, and its author noted in the PR body that more complex models were not addressed: > However, it is worth noting that for more complicated models with multiple attention groups, this PR does not fully address the EAGLE spiral block drop issue either. A general fix to this issue cannot directly cache the hit_blocks list returned by each attention type, because SWA attn and Mamba-style attn do not follow the downward-closed property (cache hit at token j does not indicate cache hit at i where i < j). So we need some more fundamental changes there. > > **Fortunately, we don't have such complex models yet, so this is not a huge issue for now.** There is such a model now, and it is one vLLM ships. **DeepSeek-V4-Flash with DSpark speculative decoding builds 4 attention groups** — 1 full MLA (`block_size=256`) plus 3 sliding-window-family groups (`block_size` 64 / 4 / 8) — and `SpeculativeConfig.use_eagle()` returns `True` for `method=\"dspark\"`, so the EAGLE block-drop path is active. `is_simple_hybrid` is `False`, so it takes the general convergence loop rather than #33524's early-break. This is a supported configuration on main: `_annotate_eagle_groups_deepseek_v4` lives in `vllm/v1/core/kv_cache_utils.py`. **Nothing in CI exercises that path.** Every EAGLE-enabled prefix-caching test in `tests/v1/core/test_prefix_caching.py` resolves to at most 2 attention groups, and the file says so itself: ```python # Test cases with eagle enabled: Only test a single simple case for now. # - 2 groups: 1 full + 1 other _EAGLE_HYBRID_MODEL_TEST_CASES = [ # 2 groups: 1 full + 1 other pytest.param([\"full\", \"sliding_window\"], 3, id=\"2g-full+sw\"), ] ``` ```python def test_prefill_hybrid_model_combinations_eagle(...): \"\"\" Test prefix caching with hybrid models (1 full attn + 1 other) with EAGLE. More complex hybrid models with EAGLE are not yet supported (see issue #32802). \"\"\" ``` (The tests that look like they have more groups do not: `make_kv_cache_config_hybrid_model` emits 3 KV groups but two of them share one spec, and `test_eagle_grouped_swa_siblings_use_same_cache_mask` likewise builds `swa_main` and `swa_mtp` from the same `swa_spec` object. `verify_and_split_kv_cache_groups` dedupes by spec, so both land at 2 attention groups and hit the `is_simple_hybrid` early break.) We hit a **total (0.0%) prefix-cache loss** on this configuration, on a fork whose base sits between #42258 and #44082. We think **main is fine for this model shape** — we worked the arithmetic and #44082's mask does admit the blocks the read gate needs, for all three of our sliding-window groups (table in §4). The reason for filing anyway: our base carries #42258's admission mask without #44082's fix, and on that combination this model class loses prefix caching entirely. Main carried the same combination between 2026-05-15 and 2026-06-02; we have not run main from that window, so we can only say this is what we would expect it to have done there. Either way nothing would have reported it — #44082 shipped without adding a test (its only change under `tests/` is the `use_eagle` → `drop_eagle_block` rename) — and the hole stays open for the next change to the SWA admission mask or to the convergence loop. So this is a test-coverage issue with a worked instance attached, not a bug report against main. We are happy to send the regression test as a PR — see §8. --- ## 1. Configuration Runtime dump from `HybridKVCacheCoordinator` construction (instrumented print, real engine): ``` num_attention_groups=4 lcm_block_size=256 grp0: spec=MLAAttentionSpec block_size=256 isinstance_FullAttentionSpec=True kv_group_ids=[0] manager=FullAttentionManager grp1: spec=SlidingWindowMLASpec block_size=64 isinstance_FullAttentionSpec=False kv_group_ids=[1, 2, 3] manager=SlidingWindowManager grp2: spec=SlidingWindowMLASpec block_size=4 isinstance_FullAttentionSpec=False kv_group_ids=[4] manager=SlidingWindowManager grp3: spec=SlidingWindowMLASpec block_size=8 isinstance_FullAttentionSpec=False kv_group_ids=[5] manager=SlidingWindowManager is_simple_hybrid=False ``` 6 KV cache groups collapse to 4 attention groups. Sliding windows: grp1 `sliding_window=128` (from `hf_config.sliding_window`); grp2/grp3 come from the compressor, where `coff = 1 + (compress_ratio == 4)` and `sliding_window = coff * compress_ratio` gives **w=8 for C4** and **w=128 for C128**. Model config: `sliding_window=128`, `num_hidden_layers=43`, `compress_ratios=[0,0,4,128,4,128,...]`, `dspark_target_layer_ids=[40,41,42]`, `num_speculative_tokens=7`. Two properties are what put this outside the tested envelope: 1. **`len(attention_groups) == 4`** — `is_simple_hybrid` requires exactly 2 with `attention_groups[0].spec` a `FullAttentionSpec`, so the convergence loop runs to fixpoint. 2. **Three *distinct* sliding-window specs**, with `block_size` 64/4/8 against `alignment_tokens = lcm = 256`. All three satisfy `block_size != alignment_tokens` and `tail < per_segment`, so both the SWA admission mask and the alignment gate in `find_longest_cache_hit` are live for each of them, at three different moduli. --- ## 2. Measured impact Two independent harnesses, each a controlled A/B on our base with only the EAGLE block drop toggled (env switch that empties `eagle_group_ids`; nothing else changes). Same server build, same `ctx=262144`, same `max-num-batched-tokens=2048`. **Harness A — 5-turn agentic coding session, identical prompts, sequential:** | | drop enabled (stock) | drop disabled | |---|---:|---:| | Prefix cache hit rate | **0.0%** | **72.7%** | | Wall clock, 5 turns | 64.27 s | 30.61 s | | Mean acceptance length | 4.80 | 4.76 | **Harness B — benchmark script:** | | drop enabled (stock) | drop disabled | |---|---:|---:| | Prefix cache hit rate | **0.0%** | **48.6%** | | TTFT, repeated shared prefix | 12355.6 ms | 995.1 ms | | TTFT, short request w/o shared prefix (control) | 3499.6 ms | 3509.5 ms | | Mean acceptance length | 2.38 | 2.26 | The no-shared-prefix control moving 0.3% confirms the effect is confined to the prefix-cache path. Hit rates are load-dependent — 72.7% and 48.6% come from different workloads and should not be compared to each other, only within a column pair. Acceptance-length cost of disabling the drop spans **−0.8% (A) to −5.0% (B)**, n=1 per arm on different workloads. We would not describe that as lossless; see §7. --- ## 3. Mechanism on our base (why it is 0% and not ~3%) Worth writing down because the failure mode differs from what #32802's title suggests. #32802 describes a spiral of progressive block drops. Here it is a **single-step collapse**: one group returns zero blocks on its very first query, `curr_hit_length` goes to 0, and the loop converges at 0 on the next iteration. The `eagle_verified` machinery from #32802 is present on our base and never gets a chance to engage, because the failure needs only one drop. The read side (identical on main modulo the `drop_eagle_block` rename): ```python # SlidingWindowManager.find_longest_cache_hit if num_contiguous_blocks == 0 and block_size != alignment_tokens: post_pop_blocks = i if use_eagle else i + 1 if (post_pop_blocks * block_size) % alignment_tokens != 0: continue ``` Under EAGLE the first accepted block of a run must sit at `i % per_segment == 0`. The write side on our base (`_cache_block_mask`, as introduced by #42258) admits only `i % per_segment >= per_segment - tail`: ```python per_segment = alignment_tokens // self.block_size tail = cdiv(self.sliding_window - 1, self.block_size) if tail >= per_segment: return None skip = per_segment - tail return [i % per_segment >= skip for i in range(num_cached_blocks, num_full_blocks)] ``` For every sliding-window group in this model the two sets are **disjoint**, so the right-to-left scan `continue`s past every cached block, `num_contiguous_blocks` never leaves 0, `match_found` stays `False`, and the group returns 0 blocks: | group | `block_size` | window | `per_segment` | `tail` | `skip` | mask admits `i % per_segment ∈` | gate needs run start at | |---|---:|---:|---:|---:|---:|---|---| | grp1 SWA MLA | 64 | 128 | 4 | 2 | 2 | {2, 3} | `i % 4 == 0` | | grp2 C4 | 4 | 8 | 64 | 2 | 62 | {62, 63} | `i % 64 == 0` | | grp3 C128 | 8 | 128 | 32 | 16 | 16 | {16, …, 31} | `i % 32 == 0` | This also explains why the GPT-OSS shape never showed this symptom even before #44082: the gate is guarded by `block_size != alignment_tokens`, so when all groups share a page size the whole branch is skipped. The two bugs are genuinely different, which is why carrying #32802's fix did not help us. --- ## 4. Is main affected? Our reading: no, for this shape #44082 changed the write side so that the admitted set contains what the gate asks for — `need = tail + 1` under EAGLE and the retained run shifted one block right: ```python need = cls._contiguous_blocks_for_hit(window_size=..., block_size=block_size, use_eagle=use_eagle) shift = 1 if use_eagle else 0 ... if i >= shift and (i - shift) % per_segment >= per_segment - need: mask[i - start_block] = True ``` plus `HybridKVCacheCoordinator.cache_blocks` making the lookahead block eligible: ```python if manager.use_eagle and aligned_num_computed_tokens > 0: num_tokens_to_cache = min(num_computed_tokens, aligned_num_computed_tokens + manager.block_size) ``` Worked out for our three groups (a hit needs blocks `i, i-1, …, i-(need-1)` with `i % per_segment == 0`): | group | `per_segment` | `need` | main admits `i % per_segment ∈` | required run | satisfied on main? | |---|---:|---:|---|---|---| | grp1 SWA MLA | 4 | 3 | {0, 2, 3} | 0, 3, 2 | **yes** | | grp2 C4 | 64 | 3 | {0, 62, 63} | 0, 63, 62 | **yes** | | grp3 C128 | 32 | 17 | {0, 16, …, 31} | 0, 31, …, 16 | **yes** | Versus our base, where the same three rows are all \"no\". So we are not claiming a live defect on main — we are claiming this arithmetic is currently load-bearing for a shipped model and is asserted by no test. Note also that this is only the SWA branch of `reachable_block_mask`; the `retention_interval` / `reachable_boundaries` branches added later interact with the same invariant and are likewise unexercised in combination with 4 groups. --- ## 5. Reproduction This reproduces on CPU with no model and no GPU, exercising only `vllm/v1/core/kv_cache_coordinator.py` and `single_type_kv_cache_manager.py`. We drove the real `HybridKVCacheCoordinator.find_longest_cache_hit`, the real `BlockPool`, the real `SlidingWindowManager.find_longest_cache_hit` and the real `cache_blocks`, hand-building four group specs to match the runtime dump in §1, caching an 8192-token prefix, then querying the same prefix. Which group carries the EAGLE flag was enumerated A/B: | EAGLE-flagged group | hit tokens / 8192 | hit rate | |---|---:|---:| | grp0 full MLA (bs=256) | 7936 | 96.9% | | grp1 SWA MLA (bs=64, w=128) | 0 | **0.0%** | | grp2 C4 (bs=4, w=8) | 0 | **0.0%** | | grp3 C128 (bs=8, w=128) | 0 | **0.0%** | | all four (coordinator fallback) | 0 | **0.0%** | | none (drop disabled) | 8192 | 100.0% | Prefix-length sweep, to show this is structural rather than a corner case: | prefix tokens | all flagged | grp0 only | grp1 only | no EAGLE | |---:|---:|---:|---:|---:| | 512 | 0 | 256 | 0 | 512 | | 1,024 | 0 | 768 | 0 | 1,024 | | 2,048 | 0 | 1,792 | 0 | 2,048 | | 8,192 | 0 | 7,936 | 0 | 8,192 | | 32,768 | 0 | 32,512 | 0 | 32,768 | | 131,072 | 0 | 130,816 | 0 | 131,072 | The `grp0 only` column is the ~3% single-block cost one would expect from an EAGLE drop on a full-attention group (`prefix − 256`). The 0% columns are the disjointness above. That contrast is the part worth turning into a test: **a correct EAGLE drop costs one block; a broken block-admission mask costs everything, and the two are not distinguishable from the aggregate hit-rate metric.** We never captured `eagle_group_ids` from a live engine in the failing arm, so which group actually carries the flag in production is inferred rather than observed: the measured 0.0% rules out \"grp0 only\" (that arm gives 96.9%), leaving \"at least one sliding-window group is flagged\" — by `_annotate_eagle_groups_deepseek_v4` or by the coordinator fallback in §6. Nothing above depends on resolving it, since every sliding-window placement and the fallback all give 0%. --- ## 6. Two secondary observations on main Both are pre-existing and neither is dangerous today given §4; flagging them because they are what turns \"annotation missed\" into \"prefix caching silently off\". **a) The all-groups fallback.** `kv_cache_coordinator.py`: ```python if use_eagle and not self.eagle_group_ids: self.eagle_group_ids = set(range(len(kv_cache_config.kv_cache_groups))) ``` For a model where no group gets annotated, this flags every group as EAGLE. On main that costs about one block per group; on a base without #44082 it is indistinguishable from the worst case (row 5 of the §5 table) — i.e. on a sliding-window model it is the most aggressive choice available, not the most conservative one. If a missing annotation is a bug, failing loudly seems better than silently enabling the drop everywhere. **b) The annotation is still keyed to one model string.** `_annotate_eagle_groups_deepseek_v4` returns early unless some spec has `model_version == \"deepseek_v4\"`, and picks its group via `next(reversed(kv_cache_spec))` with an in-source `FIXME(yifan): avoid/generalize this hacky check.` Every other multi-group architecture therefore reaches (a). #41640 proposes moving `is_eagle` onto `AttentionSpec`; it has been open since 2026-05-04. --- ## 7. Workaround we are running, and what it costs Disclosed for completeness, **not** recommended as a fix: we added an env switch that empties `eagle_group_ids`, i.e. disables the block drop entirely. That is the \"drop disabled\" column in §2 — it restores the hit rate because reader and writer agree again, at the cost of the drop's actual purpose. We cannot certify it as output-preserving. We ran a token-level equivalence gate and **it produced no signal**: at `temperature=0` our non-speculative reference arm failed to reproduce *itself* across runs on 3 of 4 prompts, so the harness is non-deterministic (batch-composition dependent numerics on our side — we are not making any claim about vLLM here) and the comparison against the patched arm is meaningless. We are reporting this so the acceptance-length deltas in §2 are not read as an equivalence argument. They are not one. If you are on a base between #42258 and #44082 with a >2-group hybrid model, the right move is to take #44082, not this switch. --- ## 8. What we would suggest The minimal ask is regression coverage for the general convergence path under EAGLE — i.e. **≥3 distinct attention-group specs so `is_simple_hybrid` is `False`** — asserting a nonzero hit rate. Two pieces, because they cover different things: **a) The convergence loop with more than one non-full group.** `_make_hybrid_kv_cache_config` already supports `\"sliding_window\"` (w=2·bs) and `\"sliding_window_large\"` (w=4·bs) as distinct specs, so this is a one-line addition to the table that currently reads \"Only test a single simple case for now\": ```python _EAGLE_HYBRID_MODEL_TEST_CASES = [ pytest.param([\"full\", \"sliding_window\"], 3, id=\"2g-full+sw\"), pytest.param([\"full\", \"sliding_window\", \"sliding_window_large\"], ..., id=\"3g-full+sw+sw_large\"), ] ``` **b) The block-admission mask at more than one modulus.** (a) alone does not reach the failure in §3. `make_kv_cache_manager` derives `scheduler_block_size` as the lcm of the group block sizes, and `_cache_hit_alignment_tokens` returns exactly that whenever `enable_partial_hash_hits` is `False` (no Mamba group). Since every group `_make_hybrid_kv_cache_config` builds shares one `block_size`, that lcm *is* the block size, so `alignment_tokens == block_size` and the gate's `block_size != alignment_tokens` guard skips the branch entirely. The mask/gate agreement is only exercised when per-group block sizes differ. `test_eagle_swa_alignment_caches_extra_block` is the right template — full at `4*block_size` against SWA at `block_size` gives alignment `4*block_size` — and what is missing is a **third** sliding-window spec at yet another block size, so that `is_simple_hybrid` is `False` *and* two distinct `per_segment` values are live at once. The DeepSeek-V4-Flash moduli are a good stress case for (b): alignment 256 against block sizes 64/4/8, giving `per_segment` 4/64/32 and `tail` 2/2/16 — including one group where `tail` is large relative to `per_segment` and one where it is tiny. (b) would have caught the mask/gate disagreement in §3 on its first CI run. **We are happy to open that PR if maintainers want it** — we have the harness and the shape already, and we can write it against main's API rather than our base. Beyond the test, we do not have a general-fix proposal we would defend. We explored moving the drop out of per-group lookup entirely (reduce with the drop disabled, then step back one `alignment_tokens` and re-query, which needs no downward-closed property from any manager), but it degenerates for prefixes shorter than `alignment_tokens` and we have not checked it against Mamba-style managers, so we mention it only as a direction rather than a recommendation. --- ## Environment - 8× V100-SXM2-32GB (SM70), TP8, DGX-1 - DeepSeek-V4-Flash, `fp8_ds_mla`, DSpark speculative decoding, `num_speculative_tokens=7`, `ctx=262144`, `max-num-batched-tokens=2048` - vLLM `1.2.3.dev3+g6f946b603a`, from the 1cat SM70 fork - Base predates #44082: `_cache_block_mask` present (#42258), while `reachable_block_mask`, `_contiguous_blocks_for_hit`, `drop_eagle_block`, `eagle_margin` and `enable_partial_hash_hits` are all absent from `single_type_kv_cache_manager.py` and `kv_cache_coordinator.py` The fork is our environment, not the subject. V100/SM70 specifics play no part in any of the above — the mechanism, the arithmetic and the coverage gap are all in architecture-independent KV-cache code, and the §4/§5 analysis is against main's sources. ## Limitations - Single machine, single checkpoint, single fork. No upstream-main run of DeepSeek-V4-Flash + DSpark + prefix caching; §4 is arithmetic against main's sources, not an execution of main. - §5 uses real vLLM classes but hand-built group specs matched to the §1 runtime dump; it is not a full engine run. - Which group carries the EAGLE flag in production is inferred by elimination, not observed (§5). - Output equivalence under the workaround is **not** established (§7). - Each A/B arm is n=1; acceptance-length deltas span −0.8% to −5.0% across the two workloads. --- cc @ivanium (authored #33524 / #42258 / #44082 and the caveat this answers), @heheda12345 (KV cache manager), @benchislett (spec decode)", "url": "https://github.com/vllm-project/vllm/issues/51771", "createdAt": "2026-08-11T06:22:08Z", "updatedAt": "2026-08-13T10:37:28Z", "timestamp": "2026-08-13T10:37:28Z", "metrics": { "reactions": 0, "comments": 5 }, "labels": [], "author": "kezboard233", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0769266f9f01a847573c", "signalId": "github:vllm-project/vllm:issue:52146", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52146", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] xgrammar backend ignores the model's other EOS ids, so guided decoding can stop mid-string", "text": "`XgrammarBackend.__post_init__` builds the `TokenizerInfo` from `tokenizer.eos_token_id` alone: ```python tokenizer_info = xgr.TokenizerInfo.from_huggingface( self.tokenizer, vocab_size=self.vocab_size, ) ``` `from_huggingface` accepts `stop_token_ids`; omitted, it derives a single id. A model whose `generation_config.json` declares several EOS ids ends up with a grammar that knows one of them. xgrammar holds a *stop* token back until the grammar may terminate — but an id it was never told about is ordinary vocabulary, legal wherever its text is legal, which inside a JSON string is essentially everywhere. The engine, which does know the full set, then stops on it: `finish_reason=\"stop\"` on a guided request whose JSON is cut inside a string. Ours declares `\"eos_token_id\": [200001, 200008]` while `tokenizer.eos_token_id` is `200001`. Measured at a prefix inside a property-name string: ``` from_huggingface(tok, vocab_size=len(tok)) -> 200008 allowed: True from_huggingface(tok, vocab_size=len(tok), stop_token_ids=[200001, 200008]) -> 200008 allowed: False ``` ### Fix Union the ids the engine can stop on and pass them. `model_config.try_get_generation_config()` already documents itself as the place where \"information about special tokens\" lives. The patch on this branch adds `model_stop_token_ids()` and wires it into **both** `TokenizerInfo` construction sites — the mistral branch made the same single-id assumption — with unit tests covering the multi-id, single-id, absent, duplicate and malformed cases.", "url": "https://github.com/vllm-project/vllm/issues/52146", "createdAt": "2026-08-13T10:06:35Z", "updatedAt": "2026-08-13T10:37:25Z", "timestamp": "2026-08-13T10:37:25Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "y-edy-os", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8be571aae3eb53d3590f", "signalId": "github:vllm-project/vllm:pull_request:52149", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52149", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle persistent top-k candidate overflow", "text": "## Purpose Fixes #51782. `persistent_topk` uses fixed-size candidate buffers in several optimized dispatch paths. When many scores map to the same coarse histogram bin, those buffers can overflow. The counters keep increasing while excess candidates are dropped, so the kernel returns valid, unique indices that are not the true top-k values. This change: - detects candidate-buffer overflow in the short, persistent-decode, persistent-medium, and filtered-large paths; - falls back to a bounded-memory exact FP32 radix rescan of the full row; - leaves the existing fast paths and the 1 MiB workspace contract unchanged; - adds a regression matrix covering all four dispatch paths and `k = 512, 1024, 2048`. The fallback performs four full-row radix passes and is intentionally entered only after an overflow is detected. ## Test Plan 1. Reproduce the bug with the vLLM v0.27.0 installed kernel on narrow score distributions across all persistent-top-k dispatch paths. 2. Compile the patched headers into a CUDA extension against the v0.27.0 image. 3. For each case, compare the selected-value multiset exactly with `torch.topk`, and also validate index range and uniqueness. 4. Run the 87-case matrix with 20 kernel repetitions per case on an NVIDIA B300 (SM 10.3, CUDA 13.0, PyTorch 2.13.0+cu130). ## Test Result | Implementation | Cases | Failing cases | Bad rows | Invalid indices | Duplicate-index rows | | --- | ---: | ---: | ---: | ---: | ---: | | vLLM v0.27.0 baseline | 87 | 65 | 2,339 | 0 | 0 | | Patched | 87 | 0 | 0 | 0 | 0 | The patched run produced zero value mismatches (`rtol=0`, `atol=0`) across all 87 cases. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, including the issue it resolves. - [x] The test plan and environment. - [x] Before/after correctness results. - [x] No documentation update is needed; this is an internal CUDA kernel fix.
    ", "url": "https://github.com/vllm-project/vllm/pull/52149", "createdAt": "2026-08-13T10:34:44Z", "updatedAt": "2026-08-13T10:35:20Z", "timestamp": "2026-08-13T10:35:20Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug" ], "author": "xijiaat", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3fac57575d4bfbc9fb27", "signalId": "github:vllm-project/vllm:pull_request:50221", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50221", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(security): enforce audio decode duration limit in NanoNemotronVL", "text": "The _extract_audio_from_videos method called load_audio_pyav without max_duration_s, allowing a small compressed video to decompress into gigabytes of PCM and crash the server via OOM. Pass VLLM_MAX_AUDIO_DECODE_DURATION_S to match the safeguard already used by AudioMediaIO. This should be merged only when https://github.com/vllm-project/vllm/pull/49948 branch) is merged", "url": "https://github.com/vllm-project/vllm/pull/50221", "createdAt": "2026-07-29T07:37:28Z", "updatedAt": "2026-08-13T10:31:56Z", "timestamp": "2026-08-13T10:31:56Z", "metrics": { "reactions": 2, "comments": 16 }, "labels": [ "ready", "multi-modality" ], "author": "jperezdealgaba", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:fac43e0f85244bd926c7", "signalId": "github:vllm-project/vllm:pull_request:51794", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51794", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4", "text": "## Purpose > Part of https://github.com/vllm-project/vllm/issues/41820. > Continue working on https://github.com/vllm-project/vllm/pull/43718. > Based on https://github.com/vllm-project/vllm/pull/50866. > Added https://github.com/vllm-project/vllm/pull/52093 and https://github.com/vllm-project/vllm/pull/52094. To be completed... ## Test Plan - Benchmark with SA InferenceX `8k1k` and `agentic` workload. - Acc test with `gsm8k` dataset. ## Test Result ### Benchmark (workload=8k1k dp8+ep) ### Benchmark (workload=agentx) To be completed... ### Acc Test To be completed... ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51794", "createdAt": "2026-08-11T09:48:40Z", "updatedAt": "2026-08-13T10:29:57Z", "timestamp": "2026-08-13T10:29:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm", "deepseek", "nvidia" ], "author": "shen-shanshan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4b7dbec326c95fc2c035", "signalId": "github:vllm-project/vllm:pull_request:48200", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48200", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Refactor]: StructuredOutputManager x Speculative Decoding Refactor", "text": "## Purpose See https://github.com/vllm-project/vllm/issues/48197. ~100 insertions, ~200 deletions of non-test changes. Then ~400 insertions, ~600 deletions for test consolidation. Moves `tests/v1/spec_decode/test_mtp_structured_output.py` and `tests/v1/structured_output/test_reasoning_structured_output.py` into `tests/v1/structured_output/test_structured_output_manager.py`. Tests are carefully picked to be more systematic while maintaining existing coverage. We can defer test consolidation to another PR to ensure that refactor pass current tests. ## Test Plan ``` venv/bin/python -m pytest \\ tests/v1/structured_output/test_structured_output_manager.py \\ tests/v1/core/test_scheduler.py \\ tests/v1/core/test_async_scheduler.py ``` Same repro as in https://github.com/vllm-project/vllm/pull/44297: https://gist.github.com/yzong-rh/b01fb276feb8f457aa68275cd44cae10 - Expected to pass both on main and on this branch Similar repro but with JSON constaints instead of STRUCTURAL_TAG constraints (see https://github.com/vllm-project/vllm/pull/44993): https://gist.github.com/yzong-rh/ed62c5f8a9fc85fb9633bb630dcc0244 - Failing on main - Should pass on this PR ## Test Result ``` 178 passed, 1 xfailed, 31 warnings in 109.18s (0:01:49) ``` Structural Tag constraints: ``` Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Successes: 50 (100.00%) Failures: 0 (0.00%) Successful tool calls: 50 ``` JSON Constraints: ``` Main: Using model: Qwen/Qwen3.6-35B-A3B Progress: S...SS..S.S.S.S.S......S..S.......S..S....SSS.S... Summary Total requests: 50 Valid schema-constrained responses: 34 (68.00%) Failures: 16 (32.00%) invalid_json: 16 This: Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Valid schema-constrained responses: 50 (100.00%) Failures: 0 (0.00%) ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/48200", "createdAt": "2026-07-10T02:24:24Z", "updatedAt": "2026-08-13T10:29:41Z", "timestamp": "2026-08-13T10:29:41Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "structured-output", "speculative-decoding", "v1" ], "author": "yzong-rh", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6fcba2c6404003bf3fae", "signalId": "github:vllm-project/vllm:pull_request:51349", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51349", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Render][1/n] Paged shared memory storage for mm tensor ipc.", "text": "## Purpose Paged shared memory storage for mm tensor ipc. 0. This compares two approaches for IPC of mm tensors: - **Paged shared memory (pshm):** mm tensor → shm → swap_blocks_batch → GPU - **ZMQ RPC:** mm tensor → ZMQ IPC → CPU Buffer → Pin Memory → GPU 1. CPU‑to‑GPU (H2D) bandwidth saturates when transferring 1 MiB blocks, so a page size of 1 MiB is a reasonable choice for shared memory storage. 2. A typical multimodal data payload is around 10 MiB, which is larger than page size of 1 MiB. Therefore, using paged shared memory for storing and transferring mm tensors is practically meaningful. 3. For payloads smaller than one page (e.g., 0.5 MiB), shared memory space is wasted, and the performance is nearly identical to that of ZMQ RPC. Hence, such small payloads offer no practical advantage. 4. Every copy operation along the mm data path introduces overhead, with dynamic memory allocation and pin memory being particularly expensive. Thus, removing the ZMQ IPC → CPU Buffer → Pin Memory steps in the ZMQ RPC path would bring significant performance gains. \"cpu_to_gpu_bandwidth\" Supplement↓
    ### 1. CPU‑to‑GPU (H2D) bandwidth saturates when transferring 1 MiB blocks PTAL: https://github.com/vllm-project/vllm/pull/48936 \"image\" ### 2. A typical multimodal data payload is around 10 MiB Qwen2VL outputs 10,764 patches when the input is a 1920×1080 image. Stored in bf16, this takes about 24 MiB; stored in uint8 (see #50411), it uses about 12 MiB. ### 3. For payloads smaller than one page (e.g., 0.5 MiB), the performance is nearly identical to that of ZMQ RPC. Hence, such small payloads offer no practical advantage. https://github.com/noooop/snippet/tree/main/benchmarks/pshm/h2d | Scenario | Data Path | ZMQ Usage | |----------|-----------|-----------| | **zmq** | CPU Source → ZMQ Send → ZMQ IPC → ZMQ Receive → CPU Buffer → Pin Memory → GPU | **Transfers full data payload** | | **zmq+shm (copy)** | Shared Memory → CPU Read → Clone → Pin Memory → GPU | **Control only** | | **zmq+shm (no-copy)** | Shared Memory → Zero-copy CPU Read → GPU | **Control only** | \"cpu_to_gpu_bandwidth\" ### 4. Removing the ZMQ IPC → CPU Buffer → Pin Memory steps in the ZMQ RPC path would bring significant performance gains. https://github.com/vllm-project/vllm/blob/5ac2684976ee22c04fe0d2f968c6cf6096b383f2/vllm/v1/serial_utils.py#L416-L425
    ## Let's do some calculations. https://github.com/noooop/snippet/tree/main/benchmarks/pshm/offline In my tests, for a typical Qwen2-VL-2B-Instruct model, the TTFT for one image is 558 ms. Transferring a 12 MiB image via ZMQ takes about 12 MiB / 2 GiB/s = 5.8 ms. Transferring a 12 MiB image via PSHM takes about 12 MiB / 5 GiB/s = 2.3 ms. 5.8 ms - 2.3 ms = 2.5 ms. Compared to 558 ms, this is completely negligible. Even if the speed were increased to 20 GiB/s, 12 MiB / 20 GiB/s = 0.58 ms — hm, that would only save 5 ms, still completely negligible. I feel like I'm tilting at windmills. (╯‵□′)╯︵┻━┻ ## Test Plan tests/models/multimodal/generation_ppl_test/test_qwen.py::test_pshm ## Test Result Model: Qwen/Qwen2-VL-2B-Instruct mm_device_do_normalize: True VLLM: dtype:torch.bfloat16 head_dtype:torch.bfloat16 41096328.0 Transformers: Constant 41081356.0 Difference (%): 0.03644475610785584 PASSED ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51349", "createdAt": "2026-08-07T03:14:17Z", "updatedAt": "2026-08-13T10:27:22Z", "timestamp": "2026-08-13T10:27:22Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "frontend", "multi-modality", "qwen", "nvidia", "mrv2" ], "author": "noooop", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bd0b304ba03f84714387", "signalId": "github:vllm-project/vllm:pull_request:52080", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52080", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 AMD MLA: fuse the q-a and kv-a RMSNorms", "text": "## Purpose On the Kimi-K3 AMD MLA front-end, every token runs **two** separate RMSNorm launches — `q_a_layernorm(q_c)` then `kv_a_layernorm(kv_c)` — once per MLA layer. This PR collapses them into a single `fused_q_kv_rmsnorm` call (`models/common/ops/fused_qk_rmsnorm.py`), a **portable Triton** kernel already in the tree — no CUDA-only gate, and its one PDL branch is `tl.constexpr`-guarded on `current_platform.is_arch_support_pdl()`, which returns `False` on ROCm — so it runs on ROCm as-is with no new kernel. The front-end norm fires on every token across all MLA layers, so collapsing two launches into one removes a per-token kernel launch and its intermediate write. This is a **small, low-risk micro-optimization**: verified at the kernel level (see Test Result), throughput-neutral end-to-end on the workloads measured (the norms are a small, non-critical-path share of GPU time, and the saving is launch overhead that cudagraph capture already elides on the served decode path). It is offered as a clean reduction in launch count and intermediate writes with no accuracy risk, not as a headline speedup. ### How it works - **`MLAModules` gains an optional `q_kv_norm` callback** (`layers/mla.py`). When `None` (every other model/backend), the wrapper keeps its two separate norms — that path is byte-identical to before. - **The MLA wrapper's `forward`**, on the q-LoRA path (`q_lora_rank is not None`), defers `q_a_layernorm` and, when `q_kv_norm` is set, normalizes `q_c` and `kv_c` together at the kv-split point. It rebinds `q_c` to the normed tensor so both the q-projection and any later indexer see the normalized value — matching the separate-norm semantics exactly. - **Kimi-K3 AMD `KimiMLAAttention`** installs the callback, wrapping `fused_q_kv_rmsnorm` with the `q_a`/`kv_a` layernorm weights and `config.rms_norm_eps` (both norms share the same eps). Only wired on the q-LoRA path; the uncompressed path has no q norm and is unchanged. This mirrors the pattern established by #50664 (optional `MLAModules` callback for the K3 output gate; generic default, AMD installs the fused variant). The uncompressed path (no q norm) and every other model that leaves `q_kv_norm` unset are untouched. Relates to #50682 (ROCm/AMD Kimi-K3 gap and roadmap tracking → Performance Optimization → Mixed Attention / MLA). ### Not a duplicate No AMD adapter or open PR wires `fused_q_kv_rmsnorm` into the AMD K3 path (checked against #50682's PR list and open `rocm`+`k3` PRs). The nearby fusions are different: #50664 (MLA output gate), #50637 (AttnRes + RMSNorm), #50634 (KDA decode gate), #41095 (AITER dual-RMSNorm for DeepSeek MLA, not K3). #51772 (chunked-context K/V packing) does not touch `common/ops` or this norm. ### Files changed - `vllm/model_executor/layers/mla.py` — optional `q_kv_norm` field on `MLAModules`; wrapper `forward` uses it on the q-LoRA path. No behavior change when unset. - `vllm/models/kimi_k3/amd/linear.py` — `KimiMLAAttention` builds and passes the fused callback. - `tests/models/kimi_k3/test_amd_mla_qkv_norm.py` — parity + empty-batch test. ## Test Plan Unit — the fused op must equal two independent RMSNorm modules (ROCm GPU): ```bash .venv/bin/python -m pytest -q tests/models/kimi_k3/test_amd_mla_qkv_norm.py ``` Covers token counts `{1, 7, 64, 512, 4096}` (decode- through prefill-sized) plus an empty batch. Accuracy — this changes an activation path, so a full eval is required: ```bash # serve K3 on the AMD box (gfx942), then: make lm-eval-gsm8k # gsm8k parity vs main (must match within tolerance) ``` End-to-end prefill perf (long input) — baseline `main` vs this PR, same seeds: ```bash make bench- # 8k/1k profile; compare TTFT (prefill) before/after ``` ## Test Result Environment: 8× MI325X (`gfx942`), `moonshotai/Kimi-K3`, `int4_per_group_32`, TP=8, EP off. This is a **small but fair micro-optimization**: the kernel-level effect is real and verified, and end-to-end it is throughput-neutral (within run-to-run noise) on this prefill-heavy workload — it neither helps nor harms serving numbers, and it carries no accuracy risk. **Kernel evidence (torch profiler traces)** — the fused kernel does exactly what it should. Each `fused_q_kv_rmsnorm` launch replaces the two separate `q_a` / `kv_a` RMSNorm launches, and nothing else in the trace changes: | | candidate (fused) | baseline (two norms) | |---|---:|---:| | `fused_q_kv_rmsnorm` launches | 384 | 0 | | plain q-a + kv-a RMSNorm launches | 0 | 768 | | front-end norm GPU time | 1.61 ms | 3.16 ms | The plain-RMSNorm count drops by exactly `768 = 2 × 384` — every fused launch removes the two norms it replaces — for **~1.55 ms less GPU time and 384 fewer kernel launches** across the profiling window. All other kernels are unchanged. **End-to-end serving** (`vllm bench serve`, 8024/1024, conc 16). Baseline = `main` (two norms); Candidate = this PR (fused): | Metric | Baseline | Candidate | Δ | |---|---:|---:|---:| | Mean TTFT (ms) | 4615.0 | 4617.2 | +0.0% | | Median TTFT (ms) | 2559.4 | 2559.3 | ~0% | | Mean TPOT (ms) | 52.70 | 52.49 | −0.4% | | Output tok/s | 279.35 | 280.34 | +0.4% | | Total tok/s | 2468.3 | 2477.1 | +0.4% | All deltas are within run-to-run noise (≤0.4%). Expected: the front-end norms are ~4.4% of GPU time and are **not on the critical path** (the workload is bound by the projection GEMMs and the SiTU activation), and the saving here is kernel-launch overhead — which full cudagraph capture already elides on the served decode path. So the fusion is a clean win in eager mode and at the kernel level, but does not move aggregate serving throughput at this operating point. It is included as a correct, low-risk reduction in launch count and intermediate writes, not as a headline speedup. **Correctness** - Unit (`test_amd_mla_qkv_norm.py`): _TODO PASS/FAIL_ (`N passed` — the fused op equals two independent RMSNorm modules across token counts `{1, 7, 64, 512, 4096}`). - gsm8k parity vs `main` (1319 samples): _TODO — baseline acc vs candidate acc; the fused op is fp32-accurate, so parity is expected._ ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/52080", "createdAt": "2026-08-13T01:43:05Z", "updatedAt": "2026-08-13T10:23:02Z", "timestamp": "2026-08-13T10:23:02Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "mpashkovskii", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:76fa76ee2cf41e07880c", "signalId": "github:vllm-project/vllm:pull_request:51705", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51705", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][DSpark][DCP] Support decode context parallelism for Kimi-K3 DSpark", "text": "## Purpose Enable decode context parallelism (DCP) for Kimi-K3 with the DSpark drafter on ROCm. Two things stood in the way: 1. **Hybrid KV cache grouping was not drafter-invariant.** The DSpark draft adds an MLA-only attention type with far fewer layers than the target. Since group size was `min(bucket_sizes)`, that small bucket shrank `group_size` and over-split the target's mamba layers. It also made an engine holding the drafter group its mamba layers differently from one without it, which breaks KV transfer in PD-disaggregated serving. 2. **The draft KV cache group cannot be DCP-sharded.** The DSpark draft attends over the whole sequence and discards its decode LSE, so it needs replicated KV while the target stays sharded. ### Changes **Grouping** — exclude draft-only buckets, identified by the existing `non_causal_multi_token_decode` spec flag, when choosing the group size. The draft bucket is then padded up to the target's group size instead of setting it; for K3 DSpark that is the 19 padding layers vLLM already logs, which is the deliberate trade for keeping the target's grouping stable. Models without a drafter are unaffected because no spec carries the flag. **Replicated draft group** — whole-sequence block tables, an unsharded slot mapping so every rank writes the full sequence, `dcp_world_size=1` metadata, and an undivided capacity estimate. `MultiHeadLatentAttention` accepts DCP for these groups and pins their impl to `dcp_world_size=1`; the config-time guard that rejected the combination is removed. **Target decode LSE** — single-token decode runs through aiter gluon with `return_lse=True` so per-rank partials can be merged. `AiterMLAImpl` declares `can_return_lse_for_decode` and keeps queries in bf16 on the gluon path. Multi-token decode (`q_len > 1`) under DCP has no gluon kernel yet and raises `NotImplementedError` rather than returning partials without an LSE. Also included: two K3 GDN decode-kernel fixes that surface under spec decode — a strided `decode_conv_indices` view needs compaction, and `state_indices` only needs unit stride when `numel > 1`. ## Test Plan Single node, 8×MI355X, `Kimi-K3` (mxfp4) + `Kimi-K3-DSpark` draft, TP=8, spec decode on (`num_speculative_tokens=7`, `method=dspark`, draft `attention_backend=TRITON_MLA`, `draft_sample_method=probabilistic`, `rejection_sample_method=block`). Baseline is TP=8 without DCP; the DCP runs add `--decode-context-parallel-size N` and are otherwise byte-identical. 1. **GSM8K**, 200 items from the `openai/gsm8k` test split, `temperature=0`, `max_tokens=1024`, concurrency 16, scoring the last integer in `content`. 2. **Configuration coverage**: `decode_context_parallel_size` 1, 2 and 8, each with spec decode on and off, at native (1M) and capped context. ## Test Result TP8 and DCP=8 below are a matched pair from the same build. ### Accuracy and draft acceptance — GSM8K, 200 items | Config | Score | Accuracy | Mean acceptance length | |---|---|---|---| | TP8 (dcp=1) | 192/200 | 96.0% | 4.88 | | DCP=8 | 193/200 | 96.5% | 4.55 | Repeat runs land at 192–195/200 for TP8 and 193–196/200 for DCP=8, so the overlapping ranges are run-to-run variation from concurrent batching (`temperature=0` fixes a single request, not the batch composition it lands in). Acceptance length varies over 4.55–4.96 across the same runs for both configurations. ### Throughput — GSM8K phase, 200 items @ concurrency 16 | Config | Wall time | Mean generation throughput | Peak | |---|---|---|---| | TP8 (dcp=1) | 107 s | 270.0 tok/s | 394.5 tok/s | | DCP=8 | 117 s | 261.2 tok/s | 395.3 tok/s | Within run-to-run spread (a repeat DCP=8 run finished in 90 s at 310.6 tok/s), so these are on par rather than a measured difference in either direction. ### KV cache capacity Replicating only the draft group preserves the target's sharding benefit: | Config | spec on | spec off | |---|---|---| | TP8 (dcp=1) | 956,229 tokens | 2,319,404 tokens | | DCP=8 | 1,506,430 tokens | 12,426,353 tokens | ### Grouping The reported KV cache sizes and the padding-layer warnings are identical to a build without the grouping change, for both TP8 and DCP=8, so grouping is unchanged for models this does not target. ### Configuration coverage `decode_context_parallel_size` 1, 2 and 8 all start and serve, with spec decode on and off, at both native 1M context and capped context. `dcp=2` also scores 8/8 on a short GSM8K gate at 128K and at native context. Before this PR, `decode_context_parallel_size > 1` with the DSpark drafter was rejected at config time.", "url": "https://github.com/vllm-project/vllm/pull/51705", "createdAt": "2026-08-10T16:40:18Z", "updatedAt": "2026-08-13T10:22:34Z", "timestamp": "2026-08-13T10:22:34Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "needs-rebase", "mrv2", "kimi", "k3" ], "author": "YukioZzz", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:36a057a8af1647270b42", "signalId": "github:vllm-project/vllm:pull_request:49458", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49458", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Hardware-agnostic model definition via HF transformer backend (1/N)", "text": "## Purpose This PR is an alternative approach to realize hardware-agnostic model definitions based on the HF transformer backend. In particular, the idea is that the modeling code of tail models resides in HF transformers and can be executed in vLLM through the help of the `transformers` backend, i.e., `--model-impl transformers`. The way this backend currently works is by replacing / patching particular layers from HF transformers with native vLLM implementations. This first PR introduces a lightweight infrastructure to overrides the native vLLM implementations with hw-agnostic definitions instead through an environment variable `VLLM_USE_HW_AGNOSTIC`. If set to VLLM_USE_HW_AGNOSTIC=1, the hw_agnostic layers/activations are tried to be imported first and only if failing, the default vLLM layers are imported. In addition to the basic infrastructure, this PR also introduces the SiluAndMul activation function as well as the RMSNorm layer, to sketch the usage of the new infrastructure (originally prepared as a separate [PR](https://github.com/bohnstingl/vllm/pull/5)). The basis for the hw-agnostic layer implementation is the DSv4 migration effort from #45470 ## Test Plan In the future, we might need to have dedicated tests for the hardware-agnostic models, see last point for discussion above. Currently we only performed rudimentary verification tests, i.e., serving the model via and then prompting the model via ``` VLLM_USE_HW_AGNOSTIC=1 vllm serve \\ ibm-granite/granite-3.3-8b-base \\ --model-impl transformers \\ --tensor-parallel-size 1 \\ --data-parallel-size 1 \\ --trust-remote-code \\ --max-model-len 4096 \\ --gpu-memory-utilization 0.85 \\ --enforce-eager ``` ``` curl -s http://localhost:8000/v1/completions -H \"Content-Type: application/json\" -d '{\"model\":\"ibm-granite/granite-3.3-8b-base\",\"prompt\":\"What is vLLM?\",\"max_tokens\":13,\"temperature\":0}' ``` yielding the response ``` {\"id\":\"cmpl-a3fffb08bcf8b243\",\"object\":\"text_completion\",\"created\":1784727174,\"model\":\"ibm-granite/granite-3.3-8b-base\",\"choices\":[{\"index\":0,\"text\":\"\\n\\nvLLM is a new way to learn languages.\",\"logprobs\":null,\"finish_reason\":\"length\",\"stop_reason\":null,\"token_ids\":null,\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"routed_experts\":null}],\"service_tier\":null,\"system_fingerprint\":\"vllm-0.23.1rc1.dev955+gc2a8e0b45.d20260707-nohash\",\"usage\":{\"prompt_tokens\":6,\"total_tokens\":19,\"completion_tokens\":13,\"prompt_tokens_details\":null},\"kv_transfer_params\":null,\"ec_transfer_params\":null,\"metrics\":null} ``` ## Test Result None yet cc @tdoublep ---
    Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/49458", "createdAt": "2026-07-22T14:15:34Z", "updatedAt": "2026-08-13T10:21:50Z", "timestamp": "2026-08-13T10:21:50Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "ready", "verified" ], "author": "bohnstingl", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:231ed0946b030ec86a59", "signalId": "github:vllm-project/vllm:pull_request:51707", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51707", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Humming] Fuse Kimi SiTU activation + per-token FP8 quant on w2 path", "text": "Draft. Stacked on #51332 (Humming MXFP4 w + block-FP8 activation) and #51673 (SiTU `valid_rows` early-exit) — only the top commit is this PR's contribution; the rest are those two open PRs. Do not merge until both land. Fuses the Kimi SiTU (SituGLU) activation and the following per-token dynamic FP8 (e4m3) quantization of the Humming down-projection (w2) input into one CUDA kernel (`situ_and_mul_quant`), replacing the separate `situ_and_mul` + `quant_input` passes and skipping the intermediate bf16 activation buffer. - Bit-for-bit identical to the unfused path (activation rounded through `scalar_t` before abs-max/quant; per-token scale = absmax/448, dequant = q*scale, matching humming `calc_scale`). - Honors the DeepEP v2 contiguous-layout `valid_rows` count from #51673: padding rows are skipped and get a benign scale. - Off by default; behind `VLLM_HUMMING_FUSE_ACT_QUANT`, gated on SiTU + per-token FP8/float32-scale w2. Two-step path preserved as fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51707", "createdAt": "2026-08-10T16:54:51Z", "updatedAt": "2026-08-13T10:20:12Z", "timestamp": "2026-08-13T10:20:12Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization", "kimi" ], "author": "elvircrn", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6677e98887a308eb8d2d", "signalId": "github:vllm-project/vllm:pull_request:51864", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51864", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Fix MultiConnector returning duplicate finished_recving signals", "text": "## Purpose Resolves #51846 Fixes a scheduler crash (`assert req.status == RequestStatus.WAITING_FOR_REMOTE_KVS`) that occurs in Disaggregated Prefill/Decode (PD) setups when multiple consumer KV connectors are configured (e.g., `NixlConnector` + `LMCacheMPConnector`). ### Root Cause When multiple KV connectors are configured as consumers, `MultiConnector.get_finished()` blindly aggregated the `finished_recving` signal from *all* connectors. Even though `MultiConnector` only assigns the actual load operation to a single chosen connector, the unassigned connectors are given `0` tokens to load in `update_state_after_alloc`. As a result, they immediately consider their load operation \"completed\" and return the `req_id` as finished receiving. This results in the `MultiConnector` yielding the same `req_id` in two separate scheduler steps. The first signal correctly transitions the request out of `WAITING_FOR_REMOTE_KVS` to `RUNNING`, but the second signal crashes the scheduler since the request is no longer waiting for KV. ### The Fix Updated `MultiConnector.get_finished()` to only trust the `finished_recving` signal if the connector reporting it was the one explicitly chosen to load the request's KV. ## Test Plan This is a pure logic fix to `MultiConnector` (fixing an oversight in the loop conditional). **Local linting:** - `pre-commit run ruff-check` - `pre-commit run mypy-3.12 --all-files --hook-stage manual` **Testing:** Due to local environment dependency extraction errors blocking `pytest`, I am relying on the automated GitHub Actions CI pipeline to verify the existing `tests/v1/kv_connector/unit/test_multi_connector.py` suite. ## Test Result - `pre-commit` passed cleanly on the modified file. - No model evaluation is included because this change is confined to the KV connector communication layer and cannot affect model output accuracy or logic. > **Note:** AI assistance was used to root cause. I have reviewed every changed line and verified the logic fix myself. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51864", "createdAt": "2026-08-11T19:59:19Z", "updatedAt": "2026-08-13T10:19:40Z", "timestamp": "2026-08-13T10:19:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "PrakshaaleJain", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f2791fd15ea75b191546", "signalId": "github:vllm-project/vllm:pull_request:47941", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47941", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] P2P NIXL + CPU EC Connector", "text": "## Purpose This PR extends ECCPUConnector to support P2P EC sharing based on NIXL. [The original PR](https://github.com/vllm-project/vllm/pull/42998) was split into two parts; [The first part](https://github.com/vllm-project/vllm/pull/47423) is a standalone CPU-based offloading EC connector, and this PR is an extension of it, allowing CPU-offloaded EC cache to be shared between vLLM instances via NIXL. The control plane is ZMQ (consumer DEALER → producer ROUTER) carrying msgpack XferReq / XferAck. **Dependencies:** - [EC Transfer Params](https://github.com/vllm-project/vllm/pull/42433) - [CPU OffloadingEC Connector](https://github.com/vllm-project/vllm/pull/47423) ## Logic - **Producer scheduler.** Binds a ZMQ ROUTER. A single router thread owns the socket, ingests `XferReq` frames, validates `mm_hash` presence in `_local_encodings` (NACK queued otherwise), pins the matching block indices in the CPU mmap region, and posts an async NIXL `WRITE` directly to the consumer's pre-registered destination blocks (In the consumer's CPU mmap region). Completions are polled on every router-loop iteration via `_sweep_completions`, which unpins blocks and emits `XferAck(ok=True/false)`. Two-phase bookkeeping (`_pending_save` → `_local_encodings`) prevents serving pre-`save_caches` garbage: a freshly allocated block range only becomes serveable on the next `build_connector_meta` step, by which point the worker's `save_caches` has finished and synchronized. - **Producer worker.** `save_caches` copies `encoder_cache[mm_hash]` → `mmap[block_indices]` on a dedicated copy stream and CPU-syncs before returning, so the bytes are coherent for the scheduler's NIXL reads. - **Consumer scheduler.** Lazy DEALER pool keyed by `(host, port)` with ZMTP heartbeat + per-DEALER monitor sockets subscribed to `EVENT_DISCONNECTED`. `ensure_cache_available(request)` allocates blocks for unseen `mm_hash`es from `request.ec_transfer_params`, sends `XferReq`, and returns `pending=True` until ack arrives. A `_loaded` mmap cache keeps completed transfer blocks alive for local re-copy on subsequent requests (`_pending_reload`), avoiding the producer round-trip when possible. - **Consumer worker.** `start_load_caches` copies `mmap[block_indices]` → `encoder_cache[mm_hash]` on a dedicated copy stream. - **Failure path.** Producer-side NACK (unknown `mm_hash`, compat hash mismatch, evicted) → consumer frees blocks and falls through to local encode. Producer disconnect → ZMTP heartbeat fires `EVENT_DISCONNECTED` within `HEARTBEAT_TIMEOUT` (4s); `on_peer_down` evicts the peer, removes the NIXL agent, and tombstones in-flight entries so stalled requests fall back cleanly. ## End-to-End Architecture ``` ┌──────────────────────────────────────┐ │ External caller (orchestrator + │ │ request router, out of scope here) │ │ │ └─┬──▲────────────────────────┬────────┘ │ │ │ HTTP │ │ HTTP response body │ HTTP req w/ req │ │ carries top-level │ ec_transfer_params (OpenAI│ │ ec_transfer_params │ in extra_args + b64 │ │ │ + │ │ │ max=1)│ │ │ ▼ │ ▼ ┌──────────────┬──┴──────────┐ ┌──────────────────────────┐ │ Producer vLLM │ │ Consumer vLLM │ │ ec_role=ec_producer │ │ ec_role=ec_consumer │ │ mm_encoder_only=true │ │ │ │ │ │ │ │ Scheduler │ │ Scheduler │ │ ├─ ECSharedRegion │ │ ├─ ECSharedRegion │ │ │ (mmap, NIXL-reg, │ │ │ (mmap, NIXL-reg, │ │ │ alloc/free/pin) │ │ │ alloc/free) │ │ ├─ _local_encodings: │ │ ├─ _remote_encodings: │ │ │ mm_hash → │ │ │ mm_hash → │ │ │ block_indices │ │ │ block_indices │ │ ├─ NIXL agent │ │ ├─ _ready: arrived │ │ ├─ ZMQ ROUTER │ │ │ mm_hashes │ │ │ (VLLM_EC_SIDE_CHANNEL) │ │ ├─ NIXL agent │ │ ├─ router thread │ │ ├─ peer pool: │ │ └─ in-flight xfers │ │ │ (host,port) → │ │ │ │ │ (DEALER,agent,md) │ │ Worker │ │ └─ ensure_cache_ │ │ └─ save_caches: │ │ available(request) │ │ GPU → mmap │ │ │ │ at block_indices │ │ Worker │ │ from metadata │ │ └─ start_load_caches: │ │ │ │ mmap → GPU │ │ request_finished() emits │ │ at block_indices │ │ ec_transfer_params │ │ from metadata │ │ in response body │ │ │ └──────────┬─────────────────┘ └──────────┬───────────────┘ │ │ │ ROUTER ← ZMQ DEALER │ │ XferReq / XferAck │ │ ◄───────────────────────────── │ │ │ │ NIXL WRITE │ │ mmap → remote mmap │ │ ──────────────────────────────►│ └────────────────────────────────►│ ``` ## Lifecycle (cold path) 1. Caller dispatches request to producer. Producer runs with `--mm-encoder-only`. 2. Producer scheduler `build_connector_meta` allocates blocks for the newly-scheduled `mm_hash`, parks them in `_pending_save`, emits `meta.saves[mm_hash] = block_indices`. 3. Producer worker's `save_caches` copies GPU → mmap at those indices, synchronizes the copy stream. 4. Next step's `build_connector_meta` promotes `_pending_save` → `_local_encodings`. Producer's `request_finished` returns `ec_transfer_params = {mm_hash: {peer_host, peer_port, size_bytes, nixl_agent_metadata_b64}}` in the response body. 5. Caller forwards the original request to a consumer with `sampling_params.extra_args[\"ec_transfer_params\"]` populated from (4). 6. Consumer scheduler `ensure_cache_available` allocates blocks, opens a ZMQ DEALER to the producer (cached in the peer pool), `add_remote_agent`s the producer's NIXL identity, and sends `XferReq` over the DEALER. 7. Producer router thread receives `XferReq` over the ZMQ ROUTER, pins blocks, posts NIXL `WRITE` to consumer's `dst_block_indices`. On `check_xfer_state == DONE`, sends `XferAck(ok=True)` and unpins. 8. Consumer `_drain_acks` consumes the ack, moves `mm_hash` to `_ready`. Next `build_connector_meta` adds it to `meta.loads` and moves blocks to `_loaded`. Worker copies mmap → GPU on its dedicated stream. 9. Subsequent requests for the same `mm_hash` re-serve via local mmap → GPU re-copy (`_pending_reload`), no producer round-trip (until evicted when the CPU mmap region is full - FIFO policy). ## Configuration All knobs in `ec_connector_extra_config`. | Field | Scope | Default | Purpose | |---|---|---|---| | `--mm-encoder-only` (existing) | producer, required | `False` | Skip the LM entirely; only the vision encoder runs. | | `ec_connector_extra_config.num_ec_blocks` | both | `80000` | Block count in the mmap region on each side. | | `VLLM_EC_SIDE_CHANNEL_HOST` (env) | producer | pod IP | ZMQ ROUTER bind host. | | `VLLM_EC_SIDE_CHANNEL_PORT` (env) | producer | — | ZMQ ROUTER bind port; advertised to caller in response's `peer_port`. | Consumer needs no static peer config — every value to reach a producer arrives per-request via `ec_transfer_params`. ## Testing plan - **Unit** (`tests/v1/ec_connector/unit/`): `test_metadata.py` (msgpack round-trip, compat hash factors); `test_scheduler.py` (~10 tests covering producer + consumer branches: alloc/send, idempotency, ack success/failure, peer-pool invalidation, router-thread WRITE/NACK, concurrent xfers per `mm_hash`, `request_finished` shape); `test_worker.py` (3 tests for save/load copy paths); `test_ec_shared_region.py`, `test_utils.py`, `test_common.py` - **Run command:** ```bash python -m pytest tests/v1/ec_connector/unit/ -v ```", "url": "https://github.com/vllm-project/vllm/pull/47941", "createdAt": "2026-07-08T01:43:53Z", "updatedAt": "2026-08-13T10:19:02Z", "timestamp": "2026-08-13T10:19:02Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "cpu", "kv-connector", "mrv2", "rust" ], "author": "omerpaz95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a84d4b7f6db7a8b2810f", "signalId": "github:vllm-project/vllm:pull_request:52147", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52147", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Standardise weight tying on `ParallelLMHead.tie_weights`", "text": "## Purpose vLLM expresses tied word embeddings in three different ways. Only one of them, `self.lm_head = self.lm_head.tie_weights(embed_tokens)`, dispatches through `quant_method.tie_weights`. This PR converts the other two so tying is expressed one way everywhere. - `self.lm_head.weight = embed_tokens.weight` (33 sites) bypassed the quant method entirely, so it was wrong for quant methods that repack. The `ParallelLMHead` already exists at that point, so this is a one-line swap with no structural change. - `self.lm_head = embed_tokens` (23 sites) built no `ParallelLMHead` at all in the tied branch. These now always build one and tie it. Two consequences of the second bullet are worth review: 1. A transient vocab x hidden allocation before the head is tied. 2. The quant config is now consulted for the `lm_head` prefix where it previously never was. Five models hardcode tying with `assert config.tie_word_embeddings` (`gemma`, `gemma2`, `commandr`, `cohere2_moe`, `mpt`) and have no real `lm_head` to build. They are left alone here. ## Relationship to #51665 This is not a duplicate. It is the tying-standardisation half of my own #51665, split out so it can be reviewed on its own. Everything else in that PR is deliberately excluded: - `ModelConfig.maybe_untie_word_embeddings` and the checkpoint-metadata plumbing. - The `AutoWeightsLoader` aliased-parameter skip and the ~50 hand-rolled `skip_prefixes=[\"lm_head.\"]` removals that depend on it. **Every existing skip is left untouched here**, so weight-loading behaviour is unchanged. - The `assert config.tie_word_embeddings` removals, which only exist because #51665 can untie at config time. I searched open PRs for `tie_weights`, `tie_word_embeddings` and `weight tying`; nothing else overlaps. ## Test Plan Structural change with no new behaviour, so this leans on construction and load-path checks plus existing suites. 1. Construct tied models on CPU and assert `lm_head` is a `ParallelLMHead` whose `weight` is the same `nn.Parameter` object as the embedding. Covers both conversion forms. 2. Load a real checkpoint through `load_weights` and confirm the tied head ends up populated. 3. `pre-commit run --all-files` on the changed files (ruff, ruff-format, mypy). ## Test Result Models constructed on CPU (Bloom, OPT and Mamba are all conversions of the second, structural form): ``` Qwen/Qwen3-0.6B: tie=True ParallelLMHead=True shared=True -> OK Qwen/Qwen2-0.5B: tie=True ParallelLMHead=True shared=True -> OK facebook/opt-125m: tie=True ParallelLMHead=True shared=True -> OK state-spaces/mamba-130m-hf: tie=True ParallelLMHead=True shared=True -> OK bigscience/bloom-560m: tie=True ParallelLMHead=True shared=True -> OK ``` End-to-end `load_weights` with the real `Qwen/Qwen3-0.6B` checkpoint: ``` checkpoint has lm_head.weight: True num loaded params: 226 lm_head.weight in loaded: False # still skipped by the existing skip_prefixes shared storage: True matches checkpoint: True head is nonzero: True ``` Existing suite touching the quantised tied-head path: ``` $ pytest tests/model_executor/test_qwen3_5_quantization.py -q 2 passed, 14 warnings in 0.65s ``` Linters: ``` $ pre-commit run --all-files # on the changed files ruff check ... Passed ruff format ... Passed Run mypy for Python 3.10 ... Passed ``` No model evals yet, they need a GPU box. Happy to run whatever the reviewers consider load-bearing here. ## Note for reviewers In `ernie45_moe.py` and `ernie45_vl_moe.py` the tie runs outside the `is_last_rank` check, so with PP > 1 `self.lm_head` is a `PPMissingLayer`, which has no `tie_weights`. That path already raised `AttributeError` on `PPMissingLayer.weight` before this change, so it is not a regression, but the failure mode changes. Worth fixing separately. --- AI assistance was used to produce this change. I have reviewed every changed line and stand behind it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52147", "createdAt": "2026-08-13T10:12:28Z", "updatedAt": "2026-08-13T10:17:42Z", "timestamp": "2026-08-13T10:17:42Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "qwen", "mistral" ], "author": "hmellor", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:84f0b4aaf3cbc7e9dfad", "signalId": "github:vllm-project/vllm:pull_request:51403", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51403", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Route non-divisor small-head MLA decode to Gluon on gfx950", "text": "Fixes #51232. Kimi-K3 has 96 MLA query heads; at TP8 each rank gets 12 local heads. 12 is not a divisor of 16, so `use_gluon_decode` returned `False` in \"auto\" mode, routing single-token decode to the padded ASM persistent path. That path pads 12 heads to 16 and asks AITER for a `{bf16, fp8, gqa=16, qseqlen=1}` kernel, which does not exist on gfx950 — every rank crashes with `cannot get heuristic kernel` before the server starts. The Gluon MLA decode kernel handles any head count natively (no padding), so the fix is to prefer Gluon for ALL small-head counts on gfx950 in \"auto\" mode, not just divisors of 16. On gfx942 (no Gluon build) the ASM path is still used for all counts — no behavior change. `VLLM_ROCM_AITER_MLA_ASM_PADDING=asm` still forces the padded ASM path if needed. ### Changes - `rocm_aiter_mla.py`: remove the `m % num_heads == 0` gate from the \"auto\" return in `use_gluon_decode` - `test_rocm_aiter_mla_head_padding.py`: update 3 assertions so non-divisor counts (12, 6) assert `is on_gfx950` instead of `not`", "url": "https://github.com/vllm-project/vllm/pull/51403", "createdAt": "2026-08-07T14:38:14Z", "updatedAt": "2026-08-13T10:16:43Z", "timestamp": "2026-08-13T10:16:43Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "zvigrinberg", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a9f9f1f9014f85a6f1ed", "signalId": "github:vllm-project/vllm:pull_request:46336", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46336", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Profiler] Add execution trace capture to torch profiler config", "text": "## Summary Adds a `torch_profiler_execution_trace` option to `ProfilerConfig` that captures a PyTorch **execution trace (ET)** alongside the existing Kineto trace, wired through the existing `--profiler-config` flow. Previously vLLM's profiler config could only collect the Kineto/Chrome trace via `tensorboard_trace_handler` — `TorchProfilerWrapper` built `torch.profiler.profile(...)` without an `ExecutionTraceObserver`, and there was no flag to enable one. Collecting an ET (needed for tools like Chakra/PARAM replay and graph-level analysis) required hand-editing `gpu_model_runner._model_forward()`. This makes ET a first-class config option, with no per-model code changes. ## Changes - **`vllm/config/profiler.py`**: new `torch_profiler_execution_trace: bool = False` field, validated to require `profiler=\"torch\"`. - **`vllm/profiler/wrapper.py`**: when enabled, `TorchProfilerWrapper` registers a `torch.profiler.ExecutionTraceObserver` writing `execution_trace_.json` under `torch_profiler_dir`, passes it to `torch.profiler.profile(execution_trace_observer=...)`, and calls `cleanup()` on stop (vLLM stops the profiler directly rather than via a context manager, so the observer is not auto-cleaned). Skipped with a warning for URI trace dirs (gs://, s3://, …) since ET writes only to local files. - **`docs/contributing/profiling.md`**: documents the new option. ## Usage ```bash vllm serve meta-llama/Llama-3.1-8B-Instruct \\ --profiler-config '{\"profiler\":\"torch\",\"torch_profiler_dir\":\"/tmp/prof\",\"torch_profiler_execution_trace\":true}' ``` Produces `execution_trace_.json` per worker, alongside the Kineto trace, and respects the existing `delay_iterations`/`max_iterations` controls. ## Testing Added to `tests/v1/worker/test_gpu_profiler.py`: - `torch_profiler_execution_trace` requires `profiler=\"torch\"` (validation). - Disabled by default / enabled when set. - No `ExecutionTraceObserver` registered when disabled, and skipped for URI dirs. - **CPU-only end-to-end** test: runs the wrapper, asserts a valid `execution_trace_*.json` containing `aten::` ops is produced (no GPU required, CI-friendly). ``` $ python -m pytest tests/v1/worker/test_gpu_profiler.py -q 31 passed ``` Also verified end-to-end on Intel XPU (`CPU+XPU` activities): the wrapper produced `execution_trace_rank0.json` (real `aten::matmul`/`aten::mm`/… nodes) together with the `.pt.trace.json.gz` Kineto trace, observer enabling/disabling cleanly. Ruff lint + format clean; lines ≤88 chars. ## Not a duplicate The only other open PR touching `ProfilerConfig`/`test_gpu_profiler.py` is #37524, which adds profiler capture *during CUDA graph capture* plus roofline/KV trace annotations — a different feature that does not add execution-trace capture. No open PR adds an `ExecutionTraceObserver` / `torch_profiler_execution_trace` option. ## Known limitation When a profiler schedule is active (`warmup_iterations`/`wait_iterations`), the single ET observer captures the whole start→stop window rather than following the per-cycle schedule. The common (unscheduled) path is unaffected. ## Notes - AI assistance (Claude) was used in preparing this change.", "url": "https://github.com/vllm-project/vllm/pull/46336", "createdAt": "2026-06-22T05:21:24Z", "updatedAt": "2026-08-13T10:16:34Z", "timestamp": "2026-08-13T10:16:34Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "documentation", "v1" ], "author": "sachinkademane", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:409284f33b0e24ec0f0a", "signalId": "github:vllm-project/vllm:pull_request:51540", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51540", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Avoid a per-prefill-step GPU->CPU sync in the KDA chunk kernels", "text": "`prepare_chunk_indices` derives per-sequence chunk counts with `.tolist()`, so passing it a device `cu_seqlens` blocks the caller. The count also fixes the Triton grid, so it has to reach the host either way; compute it from the host-side cu_seqlens we already have and copy the result across, as `gdn_attn` already does for its own prefill slice. Models from `vllm/models/kimi_k3` using FLA hit this on prefill batches, on NVIDIA and ROCm. The existing `chunk_indices` describes the decode-peeled `prefill_query_start_loc`, so add `non_spec_chunk_indices` alongside it and thread an optional `chunk_indices` through `chunk_kda_with_fused_gate`. It defaults to None and falls back to deriving on device, so other callers are unaffected. Verified host- and device-derived indices match in value, dtype and length. Not run on Kimi hardware. Note this applies to Kimi Linear but **not** Kimi K3 with cuda since the latter uses FlashKDA rather than FLA. Claude was used for this.", "url": "https://github.com/vllm-project/vllm/pull/51540", "createdAt": "2026-08-09T08:23:52Z", "updatedAt": "2026-08-13T10:15:44Z", "timestamp": "2026-08-13T10:15:44Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "ready", "needs-rebase", "kimi", "k3" ], "author": "njhill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7949bc1c5264d6e2d6bf", "signalId": "github:vllm-project/vllm:pull_request:52143", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52143", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Inkling structured output stop tokens", "text": " ## Purpose Fixes #51693. Inkling can emit structural response tokens after completing a JSON-schema structured output. The reported failure token `200028` is `<|begin_of_text|>` in the public `thinkingmachines/Inkling-Small-NVFP4` tokenizer metadata. Because these Inkling structural tokens are not declared through generation config, xgrammar did not treat them as stop-token overrides and could reject them after the JSON payload was complete. This PR resolves that path by: - Adding tokenizer-resolved Inkling structural terminal tokens to the structured-output grammar stop-token override set when the Inkling reasoning parser is active. - Leaving `SamplingParams.stop_token_ids` unchanged, so Inkling reasoning block terminators such as `<|end_message|>` do not prematurely stop the request before structured output begins. - Marking the request stopped once the structured-output grammar terminates after accepting one of its stop overrides. Duplicate-work checks run before opening this PR: - `gh issue view 51693 --repo vllm-project/vllm --comments`: issue open, unassigned, no comments. - `gh pr list --repo vllm-project/vllm --state open --search \"51693 in:body\"`: no open PRs. - `gh pr list --repo vllm-project/vllm --state open --search \"Inkling structured output stop tokens grammar rejected\"`: no open PRs. AI assistance was used to investigate and prepare this change. The submitting human should review and be able to defend every changed line before merge. ## Test Plan - Syntax check touched files. - Focused unit tests for Inkling structural stop-token injection and scheduler stop-on-grammar-termination behavior. - Ruff format and ruff check on touched files. No model evals were run; this change affects request stopping / structured-output grammar handling, not model quality or accuracy. ## Test Result - `./.venv/bin/python -m py_compile vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/python -c \"... StructuredOutputManager._create_grammar Inkling stop-token check ...\"`: passed; grammar compile received `{199999, 200006, 200010, 200028}` while `SamplingParams.all_stop_token_ids` and `SamplingParams.stop_token_ids` remained `[199999]`. - `./.venv/bin/pre-commit run ruff-format --files vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/pre-commit run ruff-check --files vllm/v1/structured_output/__init__.py vllm/v1/core/sched/scheduler.py tests/v1/structured_output/test_reasoning_structured_output.py tests/v1/core/test_scheduler.py`: passed. - `./.venv/bin/python -m pytest tests/v1/structured_output/test_reasoning_structured_output.py -k 'inkling_structural_tokens_are_stop_overrides' -v`: selected test reported `PASSED`, then local macOS test process exited with code 139 during global teardown in `torch.accelerator.memory.empty_host_cache()` via `tests/conftest.py::cleanup_fixture`. - `./.venv/bin/python -m pytest tests/v1/core/test_scheduler.py -k 'stop_request_when_structured_output_fsm_terminates' -v`: selected test reported `PASSED`, then local macOS test process exited with code 139 during the same global teardown path. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    **BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/52143", "createdAt": "2026-08-13T10:00:35Z", "updatedAt": "2026-08-13T10:15:40Z", "timestamp": "2026-08-13T10:15:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "structured-output" ], "author": "danielpolimac", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:080069ee850506c6a470", "signalId": "github:vllm-project/vllm:pull_request:52015", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52015", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][MRV2] Support pipeline parallelism for DiffusionGemma", "text": "## Purpose - Support pipeline parallelism for DiffusionGemma (follow-up to #45828) - Broadcast the sampler-owned per-step state to the other ranks; the initial canvas is seeded per request, so all ranks agree without a broadcast - Fix a PP deadlock in the need-sampled mask for models that roll back `num_computed_tokens` - Harden `PPHandler.broadcast` against in-place overwrite during the async send - Not a duplicate: no open PR or issue touches DiffusionGemma PP ## Test Plan - `pytest tests/v1/worker/test_diffusion_pp_state.py -v` - PP=2 vs PP=1 GSM8K A/B on Modal 2x L40S (FP8-dynamic 26B, canvas 256) - PP=1 regression: serve + KV cache size vs main ## Test Result - 2 passed, 0 failed | | PP=1 | PP=2 | |---|---|---| | GSM8K 50q (3 runs) | 88 / 94 / 96% | 90 / 96 / 94% | | Throughput (q/s) | 4.35 | 5.41 | - PP=1 unchanged: KV cache size identical to main, outputs coherent - State broadcast is ~22 MB/step at 8 requests, same order as inter-stage hidden traffic - Verified at PP=2; deeper pipelines untested --- AI assistance was used for this change (Claude); every line was reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/52015", "createdAt": "2026-08-12T15:59:25Z", "updatedAt": "2026-08-13T10:12:30Z", "timestamp": "2026-08-13T10:12:30Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "mrv2" ], "author": "guan404ming", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c565f29963f7e3646f15", "signalId": "github:vllm-project/vllm:pull_request:51653", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51653", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Enable V2 model runner for Kimi-K3 on ROCm", "text": "## Purpose Kimi-K3 on ROCm was gated from using V2 model runner. After validation using the up-to-date upstream. V2 model runner is working as expected. ## Test Plan Command to start Kimi-K3 on mi355x ``` export VLLM_ROCM_USE_AITER=1 export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # enable/disable V2 as A/B test export VLLM_USE_V2_MODEL_RUNNER=0/1 vllm serve moonshotai/Kimi-K3 \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --load-format auto \\ --gpu-memory-utilization 0.95 \\ --mm-encoder-tp-mode data \\ --max-num-batched-tokens 4096 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --max-num-seqs 128 ``` ## Test Result Tested with lm-eval: ``` lm_eval --model local-chat-completions --model_args model=moonshotai/Kimi-K3,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=256,max_retries=10,max_gen_toks=4096,max_length=16384,timeout=60000 --batch_size auto --tasks gsm8k --num_fewshot 20 --apply_chat_template ``` | runner | run 1 | run 2 | |--------|-------|-------| | V1 | 96.66% | 97.04% | | V2 | 97.57% | 96.89% | ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51653", "createdAt": "2026-08-10T10:03:11Z", "updatedAt": "2026-08-13T13:33:16Z", "timestamp": "2026-08-13T13:33:16Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "ready", "kimi", "k3" ], "author": "vllmellm", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a973049a127e018643f5", "signalId": "github:vllm-project/vllm:pull_request:41412", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:41412", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add MixtureOfExperts support for PhiMoE", "text": "This PR adds EPLB (Expert-Parallel Load Balancing) support for PhiMoE by implementing the `MixtureOfExperts` protocol. ## Changes - Add `PhiMoEMixtureOfExperts` mixin class with `extract_moe_parameters()` and `update_physical_experts_metadata()` - Thread `enable_eplb` from model config through `PhiMoEModel` → `PhiMoEDecoderLayer` → `PhiMoE` - Pass `enable_eplb` and `num_redundant_experts` to `FusedMoE` constructor - Add `set_moe_parameters()` to `PhiMoEForCausalLM` Follows the same mixin pattern used by DeepseekV2, DeepseekV4, and Glm4MoE. Fixes #40740", "url": "https://github.com/vllm-project/vllm/pull/41412", "createdAt": "2026-04-30T20:43:23Z", "updatedAt": "2026-08-13T10:01:55Z", "timestamp": "2026-08-13T10:01:55Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "needs-rebase" ], "author": "SandishKumarHN", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2aa5fb062ce0fef8afba", "signalId": "github:vllm-project/vllm:pull_request:52140", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52140", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Quantization] Add Ling 3.0 Flash FP4 support", "text": "## Purpose Add loading and serving support for [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4). Ling FP4 uses mixed FP8/MXFP4 quantization metadata: ```json { \"quant_method\": \"fp8\", \"routed_experts_quant_method\": \"mxfp4\" } ``` This PR: - adds a model-specific configuration hook that keeps `Fp8Config` while selecting MXFP4 storage for routed experts; - maps routed-expert `*_weight_scale_inv` checkpoint names to the internal MXFP4 `*_weight_scale` parameters; - applies the loading behavior to both the main and MTP models. The change is gated on Ling's FP4 metadata, leaving BF16, FP8, and INT4 checkpoints unchanged. No open PR currently adds Ling FP4 checkpoint loading. ## Test Plan Run the full in-tree GSM8K evaluation in no-thinking mode. The evaluator uses the raw completions endpoint without applying Ling's chat template or enabling native thinking: ```bash TMPDIR=/path/to/gsm8k PYTHONPATH=. .venv/bin/python -c ' from tests.evals.gsm8k.gsm8k_eval import evaluate_gsm8k print(evaluate_gsm8k( port=8001, num_questions=1319, num_shots=5, max_tokens=256, temperature=0, seed=42, use_chat_completions=False, max_concurrency=16, request_timeout_seconds=3600, )) ' ``` ## Test Result NVIDIA GB10, CUDA 13.0, PyTorch 2.13.0+cu130: | Checkpoint | Mode | Accuracy | Invalid responses | Output tokens/s | | --- | --- | ---: | ---: | ---: | | Ling 3.0 Flash FP4 | No thinking | 84.08% | 0.00% | 102.75 | | Ling 3.0 Flash INT4 | No thinking | 86.28% | 0.00% | 109.67 | Both runs used the same vLLM revision, 1,319 questions, 5-shot prompts, temperature 0, `max_tokens=256`, concurrency 16, and the raw completions endpoint. Ling's native thinking mode was not enabled. ## AI assistance disclosure OpenAI Codex assisted with investigation, implementation, deployment, testing, evaluation analysis, and drafting this description. The human submitter reviewed the changes and understands the implementation.", "url": "https://github.com/vllm-project/vllm/pull/52140", "createdAt": "2026-08-13T09:48:47Z", "updatedAt": "2026-08-13T09:58:58Z", "timestamp": "2026-08-13T09:58:58Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "FENP", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:bf18dd500184bbf7f0d2", "signalId": "github:vllm-project/vllm:pull_request:52141", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52141", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Observability] add model initializing duration log", "text": "## Purpose Add model initializing duration log ## Test Plan NA ## Test Result NA ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52141", "createdAt": "2026-08-13T09:54:03Z", "updatedAt": "2026-08-13T09:55:29Z", "timestamp": "2026-08-13T09:55:29Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "andyxning", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5a203978c5d1de16598c", "signalId": "github:vllm-project/vllm:pull_request:52139", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52139", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][CI] Give the AITER MLA decode metadata stub its MLA dims", "text": "# Purpose `tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py::test_persistent_decode_metadata_matches_fp8_golden` fails on main with `AttributeError: 'types.SimpleNamespace' object has no attribute 'q_lora_rank'`. Two jobs report it, the dedicated AITER MLA job and the sharded `kernels/attention` job, but it is the same test. \"[Model] Add native Dots3 NOTE multimodal support\" (#51255), changed `MLACommonMetadataBuilder.__init__` to read the MLA latent dimensions off the layer in `static_forward_context` instead of calling `get_mla_dims(self.model_config)`, so that hybrid MLA models can use different dimensions per KV cache group. The builder now also sizes its chunked-prefill workspace from `kv_lora_rank + qk_rope_head_dim` rather than from `model_config.get_head_size()`. This test stubs that layer with a `SimpleNamespace` holding only `prefill_backend`, so the builder raises on the first dimension it asks for. #51255 updated the same stub in the sibling test `test_rocm_aiter_mla_causal_verify_mask.py` and in the MLA backend tests, but missed this file, which uses the identical pattern. # Test Plan The change adds the five MLA dimensions to the stub. They are taken from `get_mla_dims(vllm_config.model_config)`, which is what the builder itself used before #51255, so the stub cannot drift from the `deepseek-ai/DeepSeek-R1` configuration the rest of the test is built around. The sibling test hardcodes the same values as module constants because it also builds tensors from them; this one does not need them anywhere else. On gfx950 (MI355X), with AITER: ``` pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py \\ tests/kernels/attention/test_rocm_aiter_mla_causal_verify_mask.py ``` The repository was also searched for any other test stubbing a layer the same way; these two are the only ones. # Test Result Before the change the decode metadata test fails with the `AttributeError` above. After it, both tests pass. The assertion the test exists for, that the builder's persistent decode metadata matches the golden recomputed with explicit fp8 dtypes, is reached and holds; nothing about the coverage changes. ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52139", "createdAt": "2026-08-13T09:47:40Z", "updatedAt": "2026-08-13T09:53:24Z", "timestamp": "2026-08-13T09:53:24Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm" ], "author": "stefankoncarevic", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:106fa290f384df4e84ea", "signalId": "github:vllm-project/vllm:pull_request:47596", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47596", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: aggressively zero speculator buffers to prevent CUDA illegal memory access on H100", "text": "Fixes #47561. Root Cause: When running vLLM speculative decoding on H100s with quantized models (e.g. w8a8_fp8), the system was crashing with a CUDA illegal memory access exclusively during the speculator's prefill CUDA graph capture. During vLLM's warmup phase prior to graph capture, a \"dummy run\" eagerly executes the speculator (propose()). This leaves the AutoRegressiveSpeculator's internal state buffers (hidden_states, draft_tokens, draft_logits, idx_mapping, current_draft_step, etc.) populated with values specific to the dummy run's final execution steps. Because the prefill/decode captures operated on this stale state, the dynamic shape calculations inside _prefill were thrown out of bounds. Furthermore, the TMA descriptors (used in Hopper FP8 kernels) are extremely sensitive to uninitialized/stale data (like NaN scale factors or unaligned shape inferences), causing hard segfaults when fed this stale state. Solution: This PR introduces a robust fix that deterministically zeroes out all of the AutoRegressiveSpeculator buffers inside capture() immediately before the CUDA graphs are recorded. By aggressively running .zero_() on last_token_indices, current_draft_step, hidden_states, inputs_embeds, idx_mapping, temperature, seeds, draft_tokens, and draft_logits, we guarantee that each CUDA graph capture step executes against a pristine, zero-initialized state. This definitively prevents the Hopper architecture from crashing due to stale bounds, NaNs, or out-of-bounds embedding lookups.", "url": "https://github.com/vllm-project/vllm/pull/47596", "createdAt": "2026-07-04T08:08:49Z", "updatedAt": "2026-08-13T09:50:13Z", "timestamp": "2026-08-13T09:50:13Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "v1", "nvidia", "mrv2" ], "author": "Saurav-Gupta-9741", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5a68ed68f96be1d01f52", "signalId": "github:vllm-project/vllm:pull_request:51778", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51778", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add prediction token usage details", "text": "## Summary - expose per-request speculative decoding acceptance and rejection counts through engine outputs - populate `completion_tokens_details.accepted_prediction_tokens` and `rejected_prediction_tokens` for streaming and non-streaming Chat Completions and Completions responses - preserve per-choice accounting for parallel sampling and add focused scheduler, output processor, and serving tests This PR intentionally does not implement `reasoning_tokens`. Existing PRs, including #35037, already cover that work. `CompletionTokenUsageInfo` includes the field for protocol completeness, but this change leaves it unset. ## Why this is not a duplicate I checked issue #50895 and searched open PRs by the issue number and by `accepted_prediction_tokens rejected_prediction_tokens`. No open PR implements these two fields. The existing related PRs focus on `reasoning_tokens`, while this PR handles the speculative-decoding fields listed as untracked in #50895. ## Tests Passed: ```text .venv/bin/ruff check --ignore ISC004 .venv/bin/ruff format --check .venv/bin/python -m compileall -q git diff --check pre-commit hooks run by `git commit` ``` Focused pytest execution: ```text .venv/bin/python -m pytest \\ tests/v1/core/test_scheduler.py::test_stop_via_update_from_output \\ tests/v1/engine/test_output_processor.py::test_speculative_token_counts_accumulate_per_request \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_prediction_token_details \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_omits_prediction_details_without_spec_decode \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_sums_prediction_details_across_choices \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_streaming_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_streaming_prediction_token_details -q ``` Result: `8 passed, 15 warnings in 2.89s`. ## Model evaluation Not applicable. This change reports existing speculative-decoding accounting and does not affect model output, accuracy, or serving decisions. ## AI assistance AI assistance was used to implement this change. I have reviewed every changed line and understand the implementation end to end. Fixes #50895", "url": "https://github.com/vllm-project/vllm/pull/51778", "createdAt": "2026-08-11T07:22:23Z", "updatedAt": "2026-08-13T09:49:49Z", "timestamp": "2026-08-13T09:49:49Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend", "needs-rebase" ], "author": "luyixiao95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:699b2f83a8e552851928", "signalId": "github:vllm-project/vllm:pull_request:52138", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52138", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU]bump up vllm_xpu_kernels to 0.1.13.1", "text": "## Purpose ## Test Plan ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52138", "createdAt": "2026-08-13T09:46:44Z", "updatedAt": "2026-08-13T09:47:29Z", "timestamp": "2026-08-13T09:47:29Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "intel-gpu", "ci/build" ], "author": "jikunshang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e2b43c228a594856a29b", "signalId": "github:vllm-project/vllm:pull_request:52118", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52118", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] [Bugfix] process ragged weights in xpu linear backend", "text": "python examples/basic/offline_inference/generate.py --model gaunernst/DeepSeek-V2-Lite-Chat-FP8 --enforce-eager --max-model-len 2048 --trust-remote-code Before: \"image\" After: \"image\"", "url": "https://github.com/vllm-project/vllm/pull/52118", "createdAt": "2026-08-13T07:29:01Z", "updatedAt": "2026-08-13T09:46:42Z", "timestamp": "2026-08-13T09:46:42Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "intel-gpu" ], "author": "zufangzhu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5c8580d20a84966a7b40", "signalId": "github:vllm-project/vllm:pull_request:52108", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52108", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][CI/Release][3/N] Add xpu wheel release to release pipeline", "text": "# Summary This PR adds XPU wheel building and publishing support to the Buildkite release pipeline, enabling pre-built XPU wheels to be distributed via wheels.vllm.ai. # Changes Release pipeline (release-pipeline.yaml) - Add a new `Build wheel - x86_64 - XPU` step that builds the XPU wheel using `Dockerfile.xpu` and uploads it to S3. - Rename the existing `Publish XPU Triton shim index` step to `Publish and stage XPU Triton shim`, and extend it to also stage the Triton shim wheel into the per-commit S3 directory so that the nightly index generator can pick it up. Nightly index generation (generate-and-upload-nightly-index.sh, generate-nightly-index.py) - Filter for `vllm-*` wheels when detecting the version from the commit directory, avoiding false matches on the Triton shim wheel. - Recognize xpu as a known variant suffix in wheel filename parsing. - Include `triton-*+xpu-*` wheels in release (non-nightly) index generation so the XPU Triton shim is co-published alongside the vLLM XPU wheel. Triton shim publish script (publish-triton-shim.sh) - Stage the Triton shim wheel into the per-commit S3 prefix (s3://vllm-wheels/$COMMIT/) so it is available alongside the vLLM XPU wheel. - Add BUILDKITE_COMMIT validation and fix --wheel-dir / --output-dir paths. Dockerfile.xpu (Dockerfile.xpu) - Add a vllm-build multi-stage target that produces a standalone dist/*.whl artifact for CI wheel builds. - Remove the manual triton/triton-xpu uninstall-reinstall workaround from the runtime stage (no longer needed with the Triton shim). Documentation (gpu.xpu.inc.md) - Replace \"no pre-built XPU wheels\" with installation instructions for nightly and per-commit wheels via wheels.vllm.ai. - Document the triton==3.7.2+xpu shim and remove the manual triton-xpu swap instructions. ## Test Plan trigger release pipeline for this branch, https://buildkite.com/vllm/release-v2/builds/5120 ## Test Result wheels can be found https://wheels.vllm.ai/9b7e36805bf42c0a23a58971fd72ba598fc1e984/xpu or install by uv pip install vllm --pre --extra-index-url=https://wheels.vllm.ai/9b7e36805bf42c0a23a58971fd72ba598fc1e984/xpu/ --extra-index-url=https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52108", "createdAt": "2026-08-13T06:41:24Z", "updatedAt": "2026-08-13T09:45:19Z", "timestamp": "2026-08-13T09:45:19Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation", "intel-gpu", "ci/build" ], "author": "jikunshang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:84c7ba0e9af398c8d241", "signalId": "github:vllm-project/vllm:pull_request:52045", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52045", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Workaround contradictory mamba validation after CPU MLA disables prefix caching", "text": "## Purpose Partial workaround for #52008. **Not the full fix.** #52008 asks for one of: - **A.** Keep prefix caching and chunked prefill enabled on CPU when KimiLinear requires them - **B.** Fail fast with a clear “KimiLinearForCausalLM is not supported on CPU” - **C.** Honor `--hf-overrides '{\"use_mla\": false}'` before the CPU platform check **This PR does not implement A, B, or C.** What it does: after the existing CPU MLA path turns prefix caching and chunked prefill **off**, also reset auto-applied mamba `align` / `mamba_block_size` so `VllmConfig` does not raise: ``` ValueError: --mamba-block-size can only be set with --enable-prefix-caching ``` That is only the contradictory validation. It does not leave prefix caching or chunked prefill on. It does not treat `use_mla=false` as a real opt-out (`use_mla` is still derived from `kv_lora_rank` / `kimi_linear`). It does not make `inference-optimization/Kimi-K3-0.40B` a supported CPU serving path. If the user passed `--mamba-block-size` themselves, we raise a clear error instead of resetting it. That is still not option B for the default serve command. ## Out of scope - CPU MLA prefix caching / chunked prefill (option A) - A hard “architecture not supported on CPU” on default flags (option B) - Honoring `hf_overrides[\"use_mla\"]` (option C) - End-to-end generation for Kimi-K3-0.40B on CPU (CPU MLA decode is still limited, e.g. `head_dim=576`) ## Test Plan ```bash python -m pytest --noconftest -v tests/config/test_cpu_mla_hybrid_kimi.py ``` Local `config.json` fixture only. No weight download. No serve/generate. ## Test Result On macOS arm64 / Python 3.12: ``` 3 passed ``` `EngineArgs.create_engine_config()` no longer raises the `mamba-block-size` validation error. Resulting config still has prefix caching and chunked prefill **disabled**, `use_mla=True`, mamba mode `none`. This PR includes AI-assisted edits (Grok). I reproduced the validation error, limited the change to that conflict, and ran the tests above. - [x] Purpose (workaround only; not A/B/C) - [x] Test plan - [x] Test results (config construction only) - [ ] Docs (none; this is not a supported CPU Kimi-K3 path)", "url": "https://github.com/vllm-project/vllm/pull/52045", "createdAt": "2026-08-12T20:54:58Z", "updatedAt": "2026-08-13T09:41:32Z", "timestamp": "2026-08-13T09:41:32Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "bug", "cpu", "kimi" ], "author": "akesar01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:19ed40eeceaf906cea93", "signalId": "github:vllm-project/vllm:pull_request:50685", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50685", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Keep Qwen3Next layer boundaries sequence parallel", "text": " ## Purpose Fixes #50681. Qwen3.6-35B-A3B produces corrupted output during single-token decode when expert parallelism and MoE sequence parallelism are enabled with DP2 and TP2. The Qwen3Next model path infers the hidden-state layout from the first tensor dimension, but during TP2 single-token decode the full input and each padded local shard can all have one row. Shape alone therefore cannot determine whether attention must first all-gather the input. This PR gives supported all-MoE Qwen3Next/Qwen3.5 models a fixed model-wide sequence-parallel contract, following the same structure used by Kimi K3: - shard once before entering the decoder stack; - all-gather before attention and reduce-scatter its output in every layer; - keep every decoder-layer boundary sequence parallel; and - gather once after the final norm. There is no runtime layout state to infer or pass through decoder-layer arguments. Qwen3Next configurations that mix dense and MoE layers retain the existing layer-local MoE sequence-parallel path instead of switching layouts inside the model loop. Qwen3.5 MoE inherits the shared model forward and decoder forward implementations, so no duplicate change is needed in `qwen3_5.py`. The MTP implementations call decoder layers directly and bypass the shared model forward. They therefore shard before that direct call and gather after the MTP norm to honor the decoder's fixed contract. This is compatibility with the new contract, not a claim that MTP already exhibited #50681 on `main`. No open PR references #50681, and searches for Qwen3Next single-token sequence-parallel all-gather fixes found no duplicate work. I used AI assistance during the analysis and implementation of this change. I reviewed every changed line and verified the results reported below. ## Test Plan Use `Qwen/Qwen3.6-35B-A3B` in BF16 with DP2, TP2, EP4, eager execution, greedy decoding, and the default `allgather_reducescatter` all-to-all backend: ```bash vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 2 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --enforce-eager \\ --max-model-len 512 \\ --max-num-seqs 4 \\ --gpu-memory-utilization 0.80 ``` Send a deterministic 64-token chat-completions request for `Explain quantum computing in simple terms:` with thinking disabled. Run code checks only on the three changed files: ```bash pre-commit run --files \\ vllm/model_executor/models/qwen3_next.py \\ vllm/model_executor/models/qwen3_next_mtp.py \\ vllm/model_executor/models/qwen3_5_mtp.py git diff --check ``` ## Test Result The four-GPU run confirmed the intended topology and backend: ```text world_size=4 [EP Rank 0/4] Expert parallelism is enabled. Using AgRsAll2AllManager all2all manager. ``` On current `main` (`89c8401c8a`), the non-MTP request reproduced corrupted and repetitive output: ```text To **E****E****E******** **Quant2** **Quant2** Quant2020** ``` With this change, the same request returned coherent output: ```text To understand quantum computing, it helps to first look at how our current computers work, and then see how quantum computers break the rules. ``` For scope validation, current `main` was also run with real Qwen3.6 MTP using one and two speculative tokens. Both produced coherent output. With two speculative tokens, 34 of 58 draft tokens were accepted, confirming that the draft path was active rather than entirely masked by target rejection. The MTP file changes in this PR are therefore contract adaptation, not a separate pre-existing output-corruption fix. All pre-commit hooks for the three changed files passed, including Ruff and mypy, and `git diff --check` passed. No test files were added. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] Before-and-after E2E results. - [x] AI assistance disclosure.
    ", "url": "https://github.com/vllm-project/vllm/pull/50685", "createdAt": "2026-08-01T13:18:59Z", "updatedAt": "2026-08-13T09:37:24Z", "timestamp": "2026-08-13T09:37:24Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "bug", "ready", "qwen" ], "author": "kzwrime", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:da7bdbc0f1c540ecef9a", "signalId": "github:vllm-project/vllm:pull_request:51987", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51987", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Attention] Add FlashInfer XQA decode support on SM12x\" (#49718)", "text": "Reverts #49718 ([Attention] Add FlashInfer XQA decode support on SM12x). ## Why `GPQA Eval (GPT-OSS) (DGX Spark)` regressed in nightly [build 83511](https://buildkite.com/vllm/ci/builds/83511) (commit `02ac178`) and is still red on the later build 83539. - Baseline nightly 83443 (same agent `spark-0df9-1`): **passed**, score 0.5619, 9m27s, zero harmony parse errors. - Build 83511: `FlashInfer resolved query dtypes: prefill=torch.bfloat16, decode=torch.bfloat16, decode_backend=xqa, kv_cache_dtype=torch.bfloat16, arch=sm121` — the dedicated XQA decode path this PR enables for `is_device_capability_family(120)`. That log line does not exist at baseline. - gpt-oss-20b decode output becomes gibberish: **1081** `openai_harmony.HarmonyError: unexpected tokens remaining in message header` (0 at baseline), and the eval exceeds its 1800s budget — `RuntimeError: Evaluation timed out`, 33m14s vs 9m27s. - The sibling GPQA jobs on 2xB200 and 2xH100 passed, so the blast radius is confined to SM12x, matching this PR's gating. gpt-oss uses attention sinks, which this PR routes through the new SM12x XQA path (`FlashInfer supports sinks on SM12x XQA and SM100 trtllm-gen`) — that is the most likely source of the corrupt decode output. Reverting restores the previous SM12x decode selection. A re-land should gate the dedicated XQA path (at minimum for sink-enabled models) until it is validated on GB10/sm121. - Failure count linked to this PR: 1 (`GPQA Eval (GPT-OSS) (DGX Spark)`) - Build: 83511 _Auto-generated by CI failure analyzer._", "url": "https://github.com/vllm-project/vllm/pull/51987", "createdAt": "2026-08-12T12:04:41Z", "updatedAt": "2026-08-13T09:37:06Z", "timestamp": "2026-08-13T09:37:06Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "nvidia" ], "author": "vllm-agent", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2843cfb417fa441f1ed2", "signalId": "github:vllm-project/vllm:pull_request:52134", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Fix `WhisperEncoderLayer.forward` docstring in `dots3_note`", "text": "## Purpose The docs build emits griffe warnings for `vllm/models/dots3_note/nvidia/audio_encoder.py`: ``` WARNING - griffe: audio_encoder.py:351: Parameter 'attention_mask' does not appear in the function signature WARNING - griffe: audio_encoder.py:353: Parameter 'layer_head_mask' does not appear in the function signature ``` The `WhisperEncoderLayer.forward` docstring was inherited from the upstream HF Whisper implementation and never updated for this layer's signature, which takes packed variable-length inputs (`cu_seqlens_*`, `max_seqlen_*`) and rotary embeddings instead of `attention_mask`/`layer_head_mask`. This documents the parameters the method actually takes. It also corrects the return annotation, which claimed `torch.Tensor` while the method returns `tuple[Any, ...]` (hidden states, plus attention weights when `output_attentions=True`). Docstring only, plus one annotation. No runtime behaviour change. ## Not a duplicate - `gh pr list --state open --search \"griffe docstring parameter\"` returns only #51342, which fixes an unrelated annotation in `vllm/benchmarks/throughput.py`. - `gh pr list --state open --search \"dots3_note audio_encoder\"` returns nothing. ## Test Plan Run griffe's Google-style docstring parser over the module and check for warnings. ## Test Result Before: ``` WARNING: module.py:351: Parameter 'attention_mask' does not appear in the function signature WARNING: module.py:353: Parameter 'layer_head_mask' does not appear in the function signature ``` After: no warnings. `pre-commit run --files vllm/models/dots3_note/nvidia/audio_encoder.py` passes, including `mypy` and `ruff`. No model evaluation is included because the change touches no executed code. ## Note AI assistance (Claude Code) was used for this change. I have reviewed every changed line. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52134", "createdAt": "2026-08-13T08:58:58Z", "updatedAt": "2026-08-13T09:33:25Z", "timestamp": "2026-08-13T09:33:25Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ready", "build-docs" ], "author": "hmellor", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:c6ff87fa0f190e6d9457", "signalId": "github:vllm-project/vllm:pull_request:52005", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52005", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix .../mrope.py::apply_interleaved_rope() when torch.compile is used in torch==2.13", "text": "## Purpose This PR fixes incorrect outputs of `torch.compile` `.../mrope.py::apply_interleaved_rope()` when it's used with `torch==2.13` (which newest vLLM uses), In `torch==2.11` it worked correctly. We fix it by computing the same output in a way that `torch.compile` doesn't break . This method does only indexing and assigning so compiling it should not introduce errors. Added test case, comparing eager to `torch.compile` without updated function test case returns. ``` E Mismatched elements: 98229 / 524288 (18.7%) E Greatest absolute difference: 6.25 at index (3749, 9) E Greatest relative difference: 32512.0 at index (1541, 5) ``` [Issue](https://github.com/pytorch/pytorch/issues/193183) in pytorch github ## Test Plan ``` python -m pytest tests/kernels/core/test_mrope.py::test_apply_interleaved_rope tests/kernels/core/test_mrope.py::test_apply_interleaved_rope_torch_compile ``` Additionally it can be tested by running request to `nvidia/Cosmos3-Edge` (need to apply fixes from [PR51989](https://github.com/vllm-project/vllm/pull/51989)), server started with `VLLM_DISABLE_COMPILE_CACHE=1 vllm serve nvidia/Cosmos3-Edge --reasoning-parser nemotron_v3`: ``` curl -sS http://localhost:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ --data-binary @- <<'JSON' | { \"model\": \"nvidia/Cosmos3-Edge\", \"messages\": [ { \"role\": \"user\", \"content\": [ { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" } }, { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg\" } }, { \"type\": \"text\", \"text\": \"Identify the images independently. In the final answer, write exactly two short lines. Start the first line with 'Image 1:' and the second line with 'Image 2:'. Do not return JSON.\" } ] } ], \"max_tokens\": 1024, \"temperature\": 0, \"stream\": false, \"chat_template_kwargs\": { \"enable_thinking\": true } } JSON jq -r '.choices[0].message.content' ``` With the fix it outputs: ``` Image 1: A bee on a pink flower in a garden setting. Image 2: A rabbit in a blue coat standing on a dirt path with flowers and a house in the background. ```` Without: ``` Image 1: A rabbit in a blue coat standing on a path with flowers. Image 2: A cartoon rabbit dressed as a boy on a dirt path with a house and hills. ``` ## Test Result ``` 2 passed, 14 warnings in 0.77s ``` ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52005", "createdAt": "2026-08-12T15:03:07Z", "updatedAt": "2026-08-13T09:29:58Z", "timestamp": "2026-08-13T09:29:58Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "torch.compile", "verified" ], "author": "bastefaniak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8bbe238f297a560b4f51", "signalId": "github:vllm-project/vllm:pull_request:52133", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52133", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend] Add Hunyuan A13B tool parser", "text": "## Purpose The Rust frontend exposes parser selection and automatic model matching, but it did not provide the Python frontend's `hunyuan_a13b` tool parser. Serving `tencent/Hunyuan-A13B-Instruct` through the Rust frontend therefore could not parse the model's `[...]` JSON-array format into OpenAI tool-call deltas. This PR adds a Hunyuan A13B parser using the existing shared incremental JSON tool-call core. It supports parallel calls, empty call arrays, nested argument objects, non-ASCII text, arbitrary streaming chunk boundaries, whitespace between `` and the JSON array, and argument properties named `name` without treating them as new call headers. To match the Python parser, it also strips the leading chat-template `助手：` prefix from natural-language responses and discards text after a completed tool-call block, including an empty block. The parser is registered under the Python-compatible `hunyuan_a13b` name, with model-name matching for the hyphenated and underscored Hunyuan A13B forms. This contributes to the Rust frontend parity roadmap in #44280. Duplicate-work check: I reviewed #44280's claims and searched current open PRs for `hunyuan rust`, `hunyuan_a13b`, `Hunyuan A13B Rust Frontend`, `HunyuanA13BToolParser`, and `tool_calls JSON array Rust Frontend`. No open PR adds this parser. Related PRs cover HY-V3 reasoning (#48800), Python Hunyuan parser fixes (#49535 and #47954), and a Python Hermes parser-engine migration (#51937), so they do not overlap this Rust implementation. AI assistance was used to research, implement, test, and review this change. The commit includes the required attribution trailer. This is intentionally a draft so the account owner can complete the final human review before marking it ready. ## Test Plan ```bash cargo fmt --all --check cargo nextest run -p vllm-parser cargo nextest run -p vllm-chat factory_new_resolves_default_patterns cargo nextest run -p vllm-chat validate_parser_overrides_rejects_unknown_tool_parser cargo clippy -p vllm-parser -p vllm-chat --all-targets -- -D warnings git diff --check origin/main...HEAD ``` The parser unit tests cover: - ordinary text without a tool call; - chat-template prefix removal across streaming chunk boundaries; - preservation of a non-leading `助手：` occurrence in ordinary content; - parallel calls and deeply nested arguments; - prefix text and non-ASCII arguments; - whitespace and newlines before the JSON array; - empty arrays in complete and character-by-character streaming input; - discarded suffix text after complete and empty tool-call blocks; - rejection of an invalid trailing comma; - arbitrary streaming chunk boundaries; - nested argument keys named `name`; - a truncated tool call failing closed. ## Test Result - `cargo fmt --all --check`: passed - Full `vllm-parser` nextest suite: 431 passed (including 15 Hunyuan tests) - factory model-pattern test: 1 passed - unknown-parser registry snapshot test: 1 passed - Clippy with warnings denied for `vllm-parser` and `vllm-chat`: passed - `git diff --check`: passed Clippy emitted only a dependency-level future-incompatibility notice for `proc-macro-error2`; it emitted no lint failure for this change. An additional full `vllm-chat` run passed 268 tests before eight network-transport integration tests hit the local sandbox's `Operation not permitted` restriction and fail-fast cancelled the remainder; the two chat tests affected by this change passed independently. No model evaluation is required because this adds frontend parsing parity and does not change model inference or generated tokens. ---
    Essential Elements of an Effective PR Description Checklist - [x] Purpose and related roadmap are documented. - [x] Exact test commands are provided. - [x] Test results are reported. - [x] No model documentation update is required; this adds a frontend parser for an already supported model/parser format.
    ", "url": "https://github.com/vllm-project/vllm/pull/52133", "createdAt": "2026-08-13T08:56:42Z", "updatedAt": "2026-08-13T09:23:29Z", "timestamp": "2026-08-13T09:23:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rust" ], "author": "Sunbeam23333", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4967ae955bf7d45e0d2b", "signalId": "github:vllm-project/vllm:pull_request:51974", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51974", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "feat: Add support for profile_prefix payload in HTTP /start_profile endpoint", "text": "## Proposal: Upstream API Support for Dynamic Profiling Options Allow users to pass dynamic settings to the profiler over the API (e.g., toggling native options or `experimental_config` flags) without requiring engine server restarts. This will help leverage the various options supported by native [`torch.profiler.profile`](https://docs.pytorch.org/docs/2.13/profiler.html#api-reference) ### Current Gap Currently, there is a discrepancy in profiling capabilities between online and offline serving: * **❌ Online Serving (`api_server.py`):** The `/start_profile` endpoint currently does not let pass any dynamic profiling options to a live production server. * **✅ Offline Serving (`vllm.LLM`):** Developers can programmatically pass a string argument locally via `llm.start_profile(profile_prefix=\"...\")`. Downstream hardware-specific profiling backends can rely on this to absorb configuration requirements on-the-fly. ### Purpose & Implementation To safely enable dynamic profiling arguments upstream, we chose the most lightweight architectural path possible instead of inventing a new configuration schema. **This PR:** 1. **Patches the REST router (`api_router.py`):** Optionally extracts a lightweight JSON body mapping `{\"profile_prefix\": \"...\"}` or a query parameter string. 2. **Updates the Abstract Protocol (`EngineClient`):** Safely permits the protocol signature to seamlessly accept `async def start_profile(self, profile_prefix: str | None = None)`. *Why this approach?* The internal vLLM v1 architecture (including `AsyncLLM.start_profile`) **already natively expects and supports the `profile_prefix` parameter**. We simply bridge the HTTP layer gap. ## Test Plan 1. Start an API server: ```bash vllm serve facebook/opt-125m ``` 2. Trigger the profiler over HTTP, tunnelling some options inside the JSON body: ```bash curl -X POST http://localhost:8000/start_profile \\ -H \"Content-Type: application/json\" \\ -d '{\"profile_prefix\": \"host_tracer_level:3;e2e_enable_fw_throttle:true\"}' # example ``` 3. Issue standard generation requests to capture metrics. 4. Stop the profiler: ```bash curl -X POST http://localhost:8000/stop_profile ``` ## Test Result & Proof of Concept We captured TPU traces and uploaded and viewed them on XProf and got the results as expected from the passed options in the `/start_profile` API. ### Example trace on XProf \"image\" ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    Signed-off-by: Rushabh Lalwani ", "url": "https://github.com/vllm-project/vllm/pull/51974", "createdAt": "2026-08-12T09:35:06Z", "updatedAt": "2026-08-13T09:20:33Z", "timestamp": "2026-08-13T09:20:33Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend" ], "author": "rushabh-46", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:23fd03fae0bb49076e45", "signalId": "github:vllm-project/vllm:pull_request:44927", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44927", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(structured_output): pass new_token_ids to should_advance() to fix MTP spec-decode off-by-one", "text": "Fixes #34650. ## Problem When Multi-Token Prediction (MTP) speculative decoding accepts a draft token, `_update_after_schedule` pre-increments `num_computed_tokens` by `(1 main + N spec)` tokens **before** model execution. After `update_from_output` appends the accepted tokens, the index-based delta in `should_advance()` starts one position past the main token. This means a `    ` token emitted as the main token is silently missed: `reasoning_ended` is never set to `True`, and grammar constraints (JSON mode, structured output) are **permanently disabled** for the rest of the request. Users running Qwen3 models with MTP enabled see unstructured output with no error or warning. ## Root Cause `should_advance()` infers which new tokens to scan by computing a delta from `num_computed_tokens`: ```python delta_from = request.num_computed_tokens - request.num_output_placeholders start = delta_from if delta_from >= 0 else max(len(all_token_ids) + delta_from, 0) ``` After an MTP step, `num_computed_tokens` has already been pre-incremented to include the spec tokens, so `start` lands past the `` token — it is never examined by `is_reasoning_end_streaming`. ## Fix Add an optional `new_token_ids: list[int] | None = None` parameter to `should_advance()`. The one call site inside `update_from_output()` already holds the accepted token list, so passing it directly bypasses the broken index arithmetic: ```python # scheduler.py — caller passes accepted tokens directly if new_token_ids and self.structured_output_manager.should_advance( request, new_token_ids ): # structured_output/__init__.py — use provided delta when available if new_token_ids is not None: delta: Iterable[int] = new_token_ids else: # existing path for non-MTP callers: async-scheduling-aware delta delta_from = request.num_computed_tokens - request.num_output_placeholders ... delta = itertools.islice(all_token_ids, start, None) if reasoner.is_reasoning_end_streaming(all_token_ids, delta): ``` All other call sites (`update_draft_token_ids`, `update_draft_token_ids_in_output`) omit the new parameter and continue using the existing async-scheduling-aware delta, which is correct for those contexts. ## Tests `tests/v1/structured_output/test_reasoning_structured_output.py` — added `test_mtp_reasoning_grammar_not_disabled`: - Simulates an MTP step by pre-incrementing `num_computed_tokens` as `_update_after_schedule` would - Calls `update_from_output` with a `` token appended - Asserts `reasoning_ended` is `True` (was `False` before fix) - Asserts `should_advance()` returns `True`, confirming grammar is still enabled", "url": "https://github.com/vllm-project/vllm/pull/44927", "createdAt": "2026-06-08T20:48:57Z", "updatedAt": "2026-08-13T09:20:20Z", "timestamp": "2026-08-13T09:20:20Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "structured-output", "needs-rebase", "v1" ], "author": "nac7", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3580c4e642971ae63a61", "signalId": "github:vllm-project/vllm:pull_request:51909", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51909", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Cached blocks never hit eviction first", "text": "## Purpose Most cached blocks with block_hash that never hit. Now they are appended to the free list mixed with fewer hit blocks. We should differentiate the both cases because of the non-hit blocks amount that is huge and far greater than hit blocks. So we should append the non-hit blocks first and then append the hit blocks so that non-hit blocks are allocated first in free list and keep alive for hit blocks. ## Test Plan 1. default testcases python -m pytest tests/v1/core/test_kv_cache_utils.py -v python -m pytest tests/v1/core/test_prefix_caching.py -v python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py -v 2. under stress test watching the amount of blocks which are hit ever and reallocated again in free list VLLM_CACHE_ROOT=/data/hf/vllm/ vllm bench throughput --model /data/hf/Qwen3.6-35B-A3B --input-len 1024 --output-len 1024 --dtype float16 --num-prompts 2000 --tensor-parallel-size 4 --enable-prefix-caching ## Test Result 1. default testcases python -m pytest tests/v1/core/test_kv_cache_utils.py -v 76 passed, 14 warnings python -m pytest tests/v1/core/test_prefix_caching.py -v 89 passed, 14 warnings python -m pytest tests/v1/core/test_single_type_kv_cache_manager.py -v 9 passed, 14 warnings 2. under stress test watching the amount of blocks which are hit ever and reallocated again in free list ...... INFO 08-13 09:03:54 [loggers.py:310] Engine 000: Avg prompt throughput: 7629.3 tokens/s, Avg generation throughput: 999.6 tokens/s, Running: 171 reqs, Waiting: 84 reqs, GPU KV cache usage: 99.8%, Prefix cache hit rate: 0.0% ...... **before : 479 blocks after : 406 blocks** the reallocated cached hit blocks decrease **15.24%** ---", "url": "https://github.com/vllm-project/vllm/pull/51909", "createdAt": "2026-08-12T02:15:54Z", "updatedAt": "2026-08-13T09:19:10Z", "timestamp": "2026-08-13T09:19:10Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "shanrow-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:434d8968ab0cc5465118", "signalId": "github:vllm-project/vllm:issue:52137", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52137", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: split local/external prefix-cache hits in `prompt_tokens_details`", "text": "### 🚀 The feature, motivation and pitch ## Background With distributed prefix caching (KV connectors such as Mooncake / lmcache), cache hits in a single prefill can come from two different sources: - **local**: hits on this instance's GPU prefix cache; - **external**: hits fetched from a remote KV store / another instance via the KV connector. The scheduler already tracks `num_local_cached_tokens` / `num_external_cached_tokens` separately in `PrefillStats`, but `OutputProcessor` only forwards their sum (`num_cached_tokens`), so the breakdown is lost before it reaches the OpenAI serving layer — users cannot tell, per request, where cache hits came from. Exposing this information per-request may also be handy for pricing purposes. ## Current behavior With `--enable-prompt-tokens-details` enabled, the response currently looks like: ```json \"prompt_tokens_details\": { \"cached_tokens\": 3627, \"created_cache_tokens\": 0, \"multimodal_tokens\": null } ``` ## Expected behavior Add two optional keys, `local_cached_tokens` / `external_cached_tokens`, alongside the existing fields: ```json \"prompt_tokens_details\": { \"cached_tokens\": 2320, \"created_cache_tokens\": 0, \"multimodal_tokens\": null, \"local_cached_tokens\": 1216, \"external_cached_tokens\": 1104 } ``` Invariant: `cached_tokens == local_cached_tokens + external_cached_tokens`. ## Requirements - Strictly additive change: existing fields (`cached_tokens` / `created_cache_tokens` / `multimodal_tokens`) must be preserved untouched; - New fields default to `None` and must not affect existing output when the flag is off or the values carry no meaning; - Cover both streaming and non-streaming paths for chat and completion endpoints, and also benefit offline `LLM()` users. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52137", "createdAt": "2026-08-13T09:18:49Z", "updatedAt": "2026-08-13T09:18:49Z", "timestamp": "2026-08-13T09:18:49Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request" ], "author": "fataswellassad", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b034bfd9a21d11c8c3ee", "signalId": "github:vllm-project/vllm:pull_request:52127", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52127", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build][CPU] Shrink triton-cpu-build layer by dropping build artifacts", "text": "## Summary - The `vllm-triton-cpu-build` stage in `docker/Dockerfile.cpu` clones and builds `triton-lang/triton-cpu`, but only the resulting wheel is needed by later stages/the final image. - The cloned `triton-cpu` source/build tree and Triton's downloaded LLVM/MLIR toolchain (under `/root/.triton`) were being retained in the final layer, growing the stage to 3.46GB. - This PR removes the `triton-cpu` source tree after the wheel is built, and moves `/root/.triton` into a cache mount so it never lands in the image layer (while still persisting across builds for faster rebuilds). - As a side effect, this also addresses why this stage in CI (e.g. https://buildkite.com/vllm/ci/builds/83692) can look \"not cached\": BuildKit's remote cache import is lazy — a cached step's actual layer blobs are still pulled on demand by downstream stages that bind to it, so a large cached layer can still take a long time to materialize even though the step itself reports `CACHED`. ## Test plan - [x] `docker build --target vllm-triton-cpu-build -f docker/Dockerfile.cpu .` on both base branch and this branch; compared final stage image sizes via `docker history`/`docker save`: **3.46GB → 1.44GB**. - [x] `docker run --rm ls -la /vllm-workspace/dist` on the new image: wheel is present and unchanged; `triton-cpu` source dir and `/root/.triton` are both absent from the layer. - [x] Confirmed via `git diff` that no other stage is touched.", "url": "https://github.com/vllm-project/vllm/pull/52127", "createdAt": "2026-08-13T08:34:50Z", "updatedAt": "2026-08-13T09:17:05Z", "timestamp": "2026-08-13T09:17:05Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "cpu" ], "author": "bigPYJ1151", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e231cef71b783cbf708f", "signalId": "github:vllm-project/vllm:pull_request:51360", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51360", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add reusable TP1 initialized-engine snapshots", "text": " ## Purpose @elizabetht's RFC #34303 and #37921/#37925 split CUDA checkpoint work into in-process primitives and leave CRIU disk persistence for a later phase. #35934 proposes a CRIU-safe engine suspend and resume path, while PRs #42790 and #46089 propose provider-owned pod snapshot hooks. @fergusfinn's data in #34303 also showed that full initialized snapshots can work, with large images and stale external TCP connections as central costs. @simon-mo's closed, reference-only #40323 autoresearch showcase also included inert snapshot scaffolding for a pre-model image of Python imports and a CUDA context. It explicitly excluded model weights, model configuration, compile state, and CUDA graphs. This PR instead implements and measures a host-owned artifact captured after TP1 engine initialization and a deterministic generation canary, before HTTP bind. It is also the measured successor to my closed PR #48996. PR #48996 captured only the import stage. Its matched warm comparison was flat at about 29.08 seconds versus 29.09 seconds, n=3 per arm, because the expensive engine state still had to be constructed after restore. This PR moves the boundary after engine initialization and a deterministic generation canary, but before the HTTP listener binds. The user-facing workflow is: ```bash vllm snapshot create MODEL --snapshot-dir PATH --revision MODEL_SHA [serve args] vllm snapshot inspect PATH vllm snapshot restore PATH --host 127.0.0.1 --port 8000 ``` Creation captures the initialized TP1 process tree and CUDA state with CRIU and NVIDIA's CUDA plugin. Restore validates the saved host and software identity before CRIU runs, restores the tree, binds the requested HTTP address, and checks the saved token again before returning. It does not silently fall back to ordinary startup. The lightweight `vllm_bootstrap` wrapper is intentional. Importing the normal CLI loads the vLLM runtime and PyTorch before CRIU can restore the saved process, consuming much of the activation win. Only `vllm snapshot restore` takes the lightweight path. Every other `vllm` command delegates to the existing CLI entry point unchanged. This is adjacent to, but does not replace, the in-process suspend and resume work. The current product boundary stays TP1 so it can preserve the initialized process tree without claiming multi-GPU communicator restore. ## Product envelope and tradeoffs This feature is opt-in and intentionally exchanges local disk and host privilege for activation time. - Linux x86-64, one NVIDIA GPU, TP1, PP1, and DP1 only. - Same host, GPU, driver, kernel, Python, PyTorch, vLLM source and binary, model revision, engine arguments, selected environment, and plugin identity. - Root or passwordless `sudo`, CRIU, `cuda-checkpoint`, and NVIDIA's CRIU CUDA plugin. - `io_uring` disabled before vLLM starts. For an unprivileged process, `kernel.io_uring_disabled=1` is sufficient; mode `2` disables it host-wide. - One plaintext HTTP frontend without API authentication, TLS, custom middleware, or a Unix domain socket. - No external established TCP peer at capture. Tree-owned internal connections are recorded. - Private local artifact that may contain process memory, model state, and secrets. - No concurrent restore of one artifact and no automatic normal-start fallback. The documentation explains these constraints, the sensitive-artifact boundary, and the disk, preparation-time, privilege, compatibility, and portability costs. ## Update, August 8 After the original `16999dbe788f` head passed full CI, I added one opt-in artifact-size path to the same product. `vllm snapshot create --minimize-snapshot-state` uses the existing sleep level 2 lifecycle after the correctness canary and before capture. It releases the model-weight and KV-cache allocations, then reloads the weights and recreates the KV cache after restore and before binding HTTP. The default full-state path is unchanged. The dense A10 decision cell used experimental source `4154f9b61775048f7cb679c4581706fd620426ff`, Qwen3-8B FP16 TP1, warm model-weight pages, and a cold snapshot artifact: | Path | Allocated artifact | Activation to first correct token | | --- | ---: | ---: | | ordinary | 24,231,231,488 B | 23.7633 s median, n=3 | | minimized | 5,144,281,088 B | 21.2396 s median, n=3 | The minimized path removed 19,086,950,400 allocated bytes, or 78.8 percent, and reduced the activation clock by 2.5237 seconds, or 10.6 percent. Every accepted restore returned the exact oracle token, completed at least six post-restore requests, and left no GPU process behind. The public option was then tested on source `e790b9b2018a6727bea069fa11bf1d1e64626e2b` with Qwen3-0.6B FP16 TP1 on an RTX 4090 Laptop GPU. It reduced the allocated artifact from 11,623,870,464 to 3,698,221,056 bytes, or 68.2 percent. The ordinary artifact restored once and the minimized artifact restored twice. Each restore passed the exact-token oracle plus three additional requests and drained the GPU. Page residency was uncontrolled, so this is a mechanism smoke and not another performance claim. Feature production-code changes since that tested source are a reviewer-requested hardening of existing CRIU link-remap validation and the lightweight package rename from `vllm_cli` to `vllm_bootstrap` after a reviewer found a collision with the independent `vllm-cli` PyPI project. Neither changes the measured activation path. Deterministic regressions cover the prior remap check/read race and the console entry-point/package-discovery boundary. Current head `f86a1e269b` rebases the tested implementation plus those two review fixes onto vLLM main `8a9f9f762a`; the remaining feature changes since the tested source are documentation and tests. The five commits added since the previous reviewed base touched none of this PR's 16 files. The state-minimization measurements use a different model and starting state from the original 9.4234-second A10 restore result below. The savings are not additive. Warm model-file pages can make the smaller artifact faster, while cold or slow storage can make reloading weights slower than restoring the full process image. ## Test plan Focused CPU tests: ```bash PYTHONPATH=$PWD pytest -q \\ tests/snapshot \\ tests/entrypoints/unit_tests/test_launch_cli.py ``` Changed-file repository checks: ```bash pre-commit run \\ --from-ref 8a9f9f762a47e24ed8b48a8472f6757346cb30ae \\ --to-ref HEAD ``` The RTX 4090 Laptop GPU cell used: - source `6d4507450d6ad2b226b5883904e009b9d11617ec` - base `dd856e48bbf969e3f0e561e8c76f6e92c76e0795` - RTX 4090 Laptop GPU, 16,376 MiB, driver 595.71.05 - Ubuntu kernel 7.0.0-28, Python 3.12.3, torch 2.13.0+cu132, CRIU 4.2.1 - `Qwen/Qwen3-0.6B` at revision `c1899de289a04d12100db370d81485cdf75e47ca` - FP16, maximum model length 512, GPU memory utilization 0.50, TP1, PP1, and DP1 - warm Hugging Face, TorchInductor, Triton, and vLLM caches - `VLLM_NO_USAGE_STATS=1` The Python source was the exact clean feature commit. The precompiled native payload was reused from `adc3e03517d2e7333a3bb2083bb4d394a2986876` after a path-limited diff found no C++, CUDA, Rust, FlashAttention, or vendored native-source change. The only `setup.py` change was an optional dependency minimum. The final NVIDIA A10 acceptance cell used: - exact source `16999dbe788f2a1d7e6b0de243e874193388e352` - base `c810e5ee9976ad86b81d1277b53e76d0ee639414` - NVIDIA A10, 23,028 MiB, driver 570.148.08 - Ubuntu 22.04.5, kernel 6.8.0-60, Python 3.10.12, torch 2.13.0+cu129, CUDA 12.9, CRIU 4.2.1 - the same model revision and engine settings as the 4090 cell - warm Hugging Face, TorchInductor, Triton, and vLLM caches - documented unprivileged `kernel.io_uring_disabled=1` This exact head contains the two repairs selected by the first A10 run. It captures the caller's environment before vLLM injects internal defaults, and it rejects active `io_uring` descriptors with an actionable error before CRIU runs. ## Test result Focused tests: ```text 65 passed ``` All applicable changed-file pre-commit hooks passed, including Ruff, formatting, typos, markdownlint, Python 3.10 mypy, SPDX, lazy imports, forbidden imports, and configuration checks. Each hardware cell restored one artifact five times and ran five matched ordinary warm starts. Both arms used the same source, model revision, caches, engine settings, console entry point, clock boundary, and correctness request within that cell. ### RTX 4090 Laptop GPU | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 15.7790 s | 5 | | initialized snapshot restore | 5.5488 s | 5 | | reduction | 10.2302 s, 64.8% | 5 per arm | Snapshot creation took 37.50 seconds and recorded 11,595,128,620 artifact bytes, about 11.60 GB. The measured break-even was the fourth repeated activation. ### NVIDIA A10 | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 23.9121 s | 5 | | initialized snapshot restore | 9.4234 s | 5 | | reduction | 14.4888 s, 60.6% | 5 per arm | Snapshot creation took 42.3594 seconds and recorded 15,580,855,788 artifact bytes, about 15.58 GB. The measured break-even was the third repeated activation. Every restore and warm control in both cells returned token ID `12095` with text value \" Paris\" for the prompt `The capital of France is`. The final A10 cell exercised both product defects found by the first A10 run. With the host default `kernel.io_uring_disabled=0`, creation stopped before CRIU and named the owning PIDs. Mode `1` then completed creation and all five restores. The saved manifest omitted vLLM-injected internal defaults, while changing caller-provided `VLLM_USE_FLASHINFER_SAMPLER` still produced `snapshot mismatch: environment` before CRIU ran. The runner restored the host sysctl and left the GPU process list empty. These break-even points belong only to their measured machine, model, and cache state. Larger models, slower disks, or different GPU memory settings can change both artifact size and restore time. Both hardware cells disabled usage reporting. #51219 separately closes the default telemetry HTTP session, so this PR does not duplicate that fix or claim a default-telemetry hardware result. This PR does not claim a model-quality improvement. The generation canary checks that the restored engine reproduces the saved deterministic token before it is declared ready. AI assistance was used. I reviewed the changes and can defend the implementation, constraints, and measurements. ## Checklist - [x] The purpose names the RFC and adjacent open work. - [x] The test plan includes focused tests, repository checks, and hardware. - [x] The result includes matched before and after measurements with n labels. - [x] The documentation covers setup, security, compatibility, and tradeoffs.", "url": "https://github.com/vllm-project/vllm/pull/51360", "createdAt": "2026-08-07T05:06:49Z", "updatedAt": "2026-08-13T09:13:26Z", "timestamp": "2026-08-13T09:13:26Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "frontend", "ready" ], "author": "matteso1", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:081cb62a0fcfc0e6ae50", "signalId": "github:vllm-project/vllm:pull_request:51862", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51862", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 Remove prefill pipeline stall in chunk KDA", "text": "## Purpose On ROCm's Kimi-K3 path, each prefill/mixed step has a stall in `prepare_chunk_indices`, caused by two interacting factors: ``` def prepare_chunk_indices(cu_seqlens: torch.Tensor, chunk_size: int) -> torch.Tensor: indices = torch.cat( [ torch.arange(n) for n in triton.cdiv(prepare_lens(cu_seqlens), chunk_size).tolist() ] ) return torch.stack([indices.eq(0).cumsum(0) - 1, indices], 1).to(cu_seqlens) ``` - `tolist()` D2H copy makes the host wait until the device queue is emptied - Host ops executing before the next H2D copy, while the device is sitting idle The data that the host waits on is already sitting on the host. At each step, `GDNAttentionMetadataBuilder` builds `chunk_indices` and `chunk_offsets` from `query_start_loc_cpu` and saves them in attention metadata. However, the ROCm Kimi-K3 path does not use it, and the stall waiting on and rebuilding the data from the device is a duplication of the same work. This PR addresses this and removes the stall by - Making the chunk kda kernels optionally accept `chunk_indices` and `chunk_offsets`. When they're provided, they don't need to be rebuilt through `prepare_chunk_indices`. - Passing these metadata to the kernels to actually remove the stall. - For the `prepare_chunk_offsets` call in metadata builder, building them directly on the device, instead of having a synchronization point waiting on the host-side python loop and host-device copy before the next device workload can be queued. These remove the stall from `prepare_chunk_indices`, and achieves TTFT and performance gain especially in low concurrency scenarios where the stall cannot be hidden by the heavy device workloads. ## Test Plan E2E benchmark and lm_eval with gsm8k on MI355X Server command ``` vllm serve moonshotai/Kimi-K3 \\ --served-model-name moonshotai/Kimi-K3 \\ --trust-remote-code \\ --moe-backend auto \\ --tensor-parallel-size 8 \\ --load-format auto \\ --gpu-memory-utilization 0.95 \\ --mm-encoder-tp-mode data \\ --max-num-seqs 128 \\ --max-num-batched-tokens 4096 \\ --max-model-len 10240 \\ --no-enable-prefix-caching \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --attention-backend TRITON_MLA ``` Benchmark ``` vllm bench serve \\ --model moonshotai/Kimi-K3 --backend vllm --trust-remote-code \\ --dataset-name random \\ --random-input-len 8192 --random-output-len 128 \\ --max-concurrency \"$C\" --num-prompts $((C * 4)) \\ --temperature 0 --ignore-eos --seed 1234 ``` lm_eval ``` lm_eval --model local-chat-completions \\ --model_args \"model=moonshotai/Kimi-K3,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,max_gen_toks=2048,timeout=3600\" \\ --tasks gsm8k --num_fewshot 20 \\ --apply_chat_template --fewshot_as_multiturn \\ --seed 1234 ``` ## Test Result gsm8k on MI355X |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 20|exact_match|↑ |0.9757|± |0.0042| | | |strict-match | 20|exact_match|↑ |0.9765|± |0.0042| Benchmark results | ISL/OSL | Conc | Config | Total tokens/s | Delta (%) | Total reqs/s | Delta (%) | Output tokens/s | Delta (%) | mean TTFT (ms) | Delta (%) | mean TPOT (ms) | Delta (%) | mean ITL (ms) | Delta (%) | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | 8192/128 | 8 | main | 8052.42 | -| 0.97 | - | 123.88 | - | 1813.30 | - | 50.47 | - | 50.47 | - | | 8192/128 | 8 | PR | 8316.70 | +3.28% | 1.00 | +3.09% | 127.95 | +3.29% | 1766.05 | -2.61% | 48.78 | -3.35% | 48.78 | -3.35% | | 8192/128 | 16 | main | 10067.08 | - | 1.21 | - | 154.88 | - | 2481.99 | - | 83.84 | - | 83.84 | - | | 8192/128 | 16 | PR | 10358.07 | +2.89% | 1.24 | +2.48% | 159.35 | +2.89% | 2447.98 | -1.37% | 81.17 | -3.18% | 81.17 | -3.18% | | 8192/128 | 32 | main | 11674.19 | - | 1.40 | - | 179.60 | - | 3729.02 | - | 148.50 | - | 148.50 | - | | 8192/128 | 32 | PR | 11766.98 | +0.79% | 1.41 | +0.71% | 181.03 | +0.80% | 3713.48 | -0.42% | 147.20 | -0.88% | 147.20 | -0.88% | | 8192/128 | 64 | main | 12760.26 | - | 1.53 | - | 196.31 | - | 6267.18 | - | 275.10 | - | 275.10 | - | | 8192/128 | 64 | PR | 12822.07 | +0.48% | 1.54 | +0.65% | 197.26 | +0.48% | 6255.47 | -0.19% | 273.57 | -0.56% | 273.57 | -0.56% | ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/51862", "createdAt": "2026-08-11T19:00:58Z", "updatedAt": "2026-08-13T09:11:46Z", "timestamp": "2026-08-13T09:11:46Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "rocm", "ready", "kimi", "k3" ], "author": "kliuae", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f2a74c896e7c866d61af", "signalId": "github:vllm-project/vllm:pull_request:52135", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52135", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(missing_documentation): Class `RequestOutputKind` in vllm/sampling_params.py has no", "text": "## TLDR **Gap:** Class `RequestOutputKind` in vllm/sampling_params.py has no docstring — add parameter docs, return type, and example [BLOCKED:Tests failed after retry: ==================================== ERRORS ==================================== ___________________ ERROR collecting vllm/sampling_params.py _____ **Wedge type:** `missing_documentation` **Issue:** https://github.com/vllm-project/vllm/blob/main/vllm/sampling_params.py#L182 ## Changes - `vllm/sampling_params.py` **Diff size:** 43 lines across 1 file(s) ## Pre-submission checklist - [x] Minimal change — touches at most 3 files - [x] Tests updated (if test suite present) - [x] No CI/CD, Dockerfile, or lock file modifications - [x] Diff reviewed for secrets ## AI Assistance Disclosure This contribution was AI-assisted using Hermes Agent (Nous Research). Co-authored-by: Hermes Agent ", "url": "https://github.com/vllm-project/vllm/pull/52135", "createdAt": "2026-08-13T09:07:35Z", "updatedAt": "2026-08-13T09:07:45Z", "timestamp": "2026-08-13T09:07:45Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "nickhac", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:566d713f8a90a95f83ff", "signalId": "github:vllm-project/vllm:pull_request:50062", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50062", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP", "text": "## Summary This PR adds the scheduler and KV-cache-manager support required for multi-module MTP (one MTP module per speculative step, e.g. Inkling's 8-depth checkpoint). It is the companion to #48892, which introduced the speculator itself to Model Runner V2. The core property this PR protects: the multi-module drafter reads **ahead** of the computed tokens during prefill. MTP module `m` computes position `p`'s KV from token `p + m + 1`, so at every chunked-prefill boundary the drafter consumes the next `num_speculative_tokens` (N) known prompt tokens — the \"prefill lookahead\" — to write exact KVs into all modules. That lookahead, plus the decode-time rejection re-prefill (which rewrites up to N−1 trailing positions), creates three hazards that the scheduler and KV cache layer must handle: chunk boundaries landing where lookahead tokens don't exist, caching/freeing KV that can still change, and prefix-cache hits serving KV that encodes another request's continuation. ## Changes ### Config (`vllm/config/speculative.py`) - Inkling MTP hf-config override now exposes **all** checkpoint MTP depths (`n_predict = num_nextn_predict_layers` instead of clamping to 1), and the \"exactly one speculative token\" restriction is removed. Module `i` drafts speculative token `i`. ### Speculator selection (`vllm/v1/worker/gpu/spec_decode/__init__.py`) - `init_speculator` routes to `MultiModuleMTPSpeculator` when `use_multi_module_mtp()` (method `\"mtp\"` with >1 usable MTP layer). ### Scheduler (`vllm/v1/core/sched/scheduler.py`) - New unified field `num_prefill_lookahead`: how many positions past the computed tokens the drafter reads during prefill (N for multi-module MTP, 1 for other eagle-family drafters, 0 without spec decode). All three consumers below are projections of it. - **`_reserve_prefill_lookahead`** (both scheduling loops, after encoder truncation): never end a prefill chunk with `0 < remaining < N` tokens before the prefill end — either finish the prefill or leave ≥ N for the next chunk. Without this, a boundary near the prompt end has no real lookahead tokens; the drafter falls back to sampled drafts and the trailing modules' KVs at those positions are polluted permanently (they fall outside every future query window, so nothing rewrites them). Vacuous no-op for eagle-family (lookahead 1). - **Encoder scheduling shift generalized**: `shift_computed_tokens` to `_try_schedule_encoder_inputs` is now `num_prefill_lookahead` (previously the hardcoded eagle `+1`). MM spans starting inside the lookahead window get encoded one chunk early, so the drafter's future-token embeddings are available; when the encoder budget can't cover a span, the existing rollback lands the boundary at `span_start − N`, keeping the lookahead window all-text. (The runner-side consumption of these early embeddings lands in a follow-up PR; until then MM placeholder lookahead tokens use text embeddings.) - Passes `num_prefill_lookahead` to `KVCacheManager`. ### KV cache manager / coordinator (`kv_cache_manager.py`, `kv_cache_coordinator.py`, `single_type_kv_cache_manager.py`) - `num_prefill_lookahead` is threaded into `KVCacheCoordinator`, which derives `num_reprefillable_tokens = max(0, lookahead − 1)`. - **Delayed caching**: `cache_blocks` only hash-registers tokens up to `num_computed − num_reprefillable_tokens` (mirrored in the hybrid coordinator's EAGLE lookahead-block eligibility). During decode, rejection re-prefill can rewrite the last N−1 tokens' draft KVs; registering them earlier would expose unverified KV to other requests and mutate blocks after they are shared. - **Prefix-hit soundness assert**: for every EAGLE-flagged group, `block_size >= num_prefill_lookahead`. The last N slots of *any* cached prefix hold draft KVs computed from tokens past the block hash (the writer's continuation — lookahead tokens at a chunk boundary, or sampled/draft tokens later). The pre-existing EAGLE last-block drop already recomputes the trailing block on every hit — with the new request's own lookahead tokens, making the rewrite exact — but a single dropped block only covers all N polluted slots if the block size is large enough. Rather than generalizing the drop to multiple blocks, we assert (real configs use block 16/64 with N ≤ 8). ### Sliding-window retention (`kv_cache_interface.py`, `kv_cache_utils.py`, `single_type_kv_cache_manager.py`) - New `SlidingWindowSpec.extra_retained_tokens` (tagged as N−1 for multi-module MTP in `get_kv_cache_configs`): the SWA free boundary lags by N−1 tokens, and `max_memory_usage_bytes` accounts for the extra retained blocks. Rejection re-prefill recomputes positions up to N−1 behind the tip, and each recomputed position needs its full attention window; without the lag, those windows reach into already-freed (null) blocks and the corrected KVs would be computed from garbage. Single source of truth on the spec keeps pool sizing, the admission cap, and block eviction consistent. Prior spec methods never needed this knob because no drafter re-processed positions behind the tip (eagle's ±1 is a token-id shift, not a position shift). ## Why this is not duplicating an existing PR Multi-module MTP is not supported by any open PR; the eagle/single-module paths this builds on (`drop_eagle_block`, encoder shift, `use_eagle` plumbing) are extended in place rather than duplicated. ## Testing - `tests/v1/core/test_scheduler.py` — 137 passed (includes the eagle encoder-shift regression test, chunked-prefill, and preemption paths). - `tests/v1/core/test_prefix_caching.py` — 89 passed. - `tests/config/test_speculative_draft_hf_overrides.py` — updated for the all-depths Inkling override. - Direct construction check of the new assert: accepts `block_size=16, N=8` and spec-decode-off; rejects `block_size=4, N=8` with a clear message. - End-to-end acceptance-rate evaluation with the Inkling multi-module MTP model (chunked prefill + prefix caching enabled): [results to be added]. AI assistance (Claude) was used for implementation and review of this PR; all changes were human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50062", "createdAt": "2026-07-28T01:07:12Z", "updatedAt": "2026-08-13T09:03:51Z", "timestamp": "2026-08-13T09:03:51Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "ready", "v1", "mrv2" ], "author": "TheEpicDolphin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c29689c97a2ef496ffe0", "signalId": "github:vllm-project/vllm:issue:52104", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52104", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm_c fused_add_rms_norm rounds the residual sum before variance", "text": "### Summary The CUDA `vllm_c` implementation of `fused_add_rms_norm` and the native vLLM IR implementation use different rounding orders for BF16/FP16 inputs. At upstream commit [`903d2efe`](https://github.com/vllm-project/vllm/commit/903d2efe7eb6d045d03c0a8192b1ac7bec50d168): - The [native IR implementation](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/vllm/ir/ops/layernorm.py#L37-L55) computes `x.float() + residual.float()` and uses that unrounded FP32 sum to calculate the variance. - The [vectorized `vllm_c` CUDA kernel](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/csrc/libtorch_stable/layernorm_kernels.cu#L133-L169) stores the sum in an `_f16Vec` before `sum_squares()`. This rounds the residual sum to BF16/FP16 before calculating the variance. - The [generic CUDA kernel](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/csrc/libtorch_stable/layernorm_kernels.cu#L191-L215) has the same behavior because `z` is `scalar_t`. This matters because CUDA selects `[\"native\"]` with Inductor, but selects `[\"vllm_c\", \"native\"]` when compilation is disabled ([dispatch code](https://github.com/vllm-project/vllm/blob/903d2efe7eb6d045d03c0a8192b1ac7bec50d168/vllm/platforms/cuda.py#L686-L700)). Therefore, changing only `CompilationMode` can change numerical results and, after accumulation through many layers, greedy output tokens. I observed this with GLM-4.7-Flash in BF16: the `CompilationMode.NONE` path diverged from the compiled/native path. Making the CUDA kernel use the FP32 residual sum for variance removed the token divergence in that test. ### Minimal reproducer Run this against unmodified vLLM `main` on a CUDA GPU: ```python import torch from vllm import _custom_ops as ops torch.manual_seed(0) device = \"cuda\" dtype = torch.bfloat16 tokens = 4 hidden_size = 2048 epsilon = 1e-6 x = torch.randn(tokens, hidden_size, device=device, dtype=dtype) residual = torch.randn_like(x) weight = torch.randn(hidden_size, device=device, dtype=dtype) # Native IR semantics: keep input + residual in FP32 through variance. summed_fp32 = x.float() + residual.float() variance_fp32 = summed_fp32.square().mean(dim=-1, keepdim=True) native = summed_fp32 * torch.rsqrt(variance_fp32 + epsilon) native = (native.to(weight.dtype) * weight).to(dtype) native_residual = summed_fp32.to(dtype) # vllm_c custom op mutates both tensors in place. actual = x.clone() actual_residual = residual.clone() ops.fused_add_rms_norm(actual, actual_residual, weight, epsilon) torch.cuda.synchronize() diff = (actual.float() - native.float()).abs() print(\"output mismatches:\", torch.count_nonzero(actual != native).item()) print(\"max output diff:\", diff.max().item()) print(\"mean output diff:\", diff.mean().item()) print( \"residual mismatches:\", torch.count_nonzero(actual_residual != native_residual).item(), ) ``` The exact mismatch count can vary with the GPU/build, but the two implementations are structurally different: `vllm_c` calculates variance from the rounded low-precision sum, while native IR calculates it from the FP32 sum. For scale, simulating the current `vllm_c` arithmetic on an A100 with PyTorch 2.11.0/CUDA 13.0 and the seed/shapes above produced: ```text tokens=1: 368 / 2048 output elements differ; max abs diff 0.03125 tokens=4: 1465 / 8192 output elements differ; max abs diff 0.0625 tokens=16: 5791 / 32768 output elements differ; max abs diff 0.0625 ``` ### Expected behavior The `vllm_c` and native implementations should use the same defined arithmetic/rounding semantics so that selecting `CompilationMode.NONE` does not introduce a different residual-normalization calculation. ### Proposed fix For both CUDA kernel specializations: 1. Compute each residual sum as FP32: `float sum = float(input) + float(residual)`. 2. Accumulate variance from that FP32 `sum`. 3. Publish `residual = scalar_t(sum)`. 4. Normalize using the FP32 `sum`, while preserving the intended output/weight multiplication rounding policy. This requires recomputing the sum during the second pass or temporarily retaining it; reading the already-rounded `residual` in the second pass preserves the current mismatch. ### Related issues This appears distinct from: - #49616 / #49639, which concern rounding the normalized value before multiplying by the RMSNorm weight. - #42379, which concerns the dtype used for weight multiplication. - #48271, which concerns batch-size-dependent reduction behavior. - #43390, which concerns integer overflow. This report is specifically about rounding `input + residual` before variance. ### Environment used during investigation - GPU: NVIDIA A100-SXM4-80GB - PyTorch: 2.11.0+cu130 - CUDA: 13.0 - Driver: 580.105.08 - Dtype: BF16 - Model-level observation: GLM-4.7-Flash - Relevant configuration: `CompilationMode.NONE` ### Before submitting a new issue - [x] I searched existing issues and PRs for this specific residual-sum rounding mismatch. This issue was prepared with AI assistance; the source comparison and reported measurements were manually validated on the hardware above.", "url": "https://github.com/vllm-project/vllm/issues/52104", "createdAt": "2026-08-13T05:44:10Z", "updatedAt": "2026-08-13T09:02:42Z", "timestamp": "2026-08-13T09:02:42Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "hjzccc", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8496593d3bab56a7da43", "signalId": "github:vllm-project/vllm:pull_request:51906", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51906", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add routed-experts prompt offset", "text": "## Summary - Add `routed_experts_prompt_start` to OpenAI chat/completion requests and `SamplingParams`, allowing clients to omit an already-known prompt prefix from returned R3. - Centralize NumPy-to-base64 serialization used by existing R3 responses and document the `int32` expert-ID representation. - Keep OpenAI streaming behavior unchanged: R3 remains supported only on existing non-streaming responses. ## Why this is not duplicate work No open PR implements the R3 prompt offset or shared R3 response serialization as a frontend change. #49555 returns sparse-attention indexer top-k, which is a distinct feature; this PR only factors shared R3-facing plumbing. ## Validation - `uvx pre-commit run` — passed. - `PYTHONPATH=. /home/aoshen/vllm/.venv/bin/python -m pytest tests/test_sampling_params.py tests/entrypoints/openai/test_stop_token_ids.py tests/utils_/test_serial_utils.py -q` — 49 passed. - R3 OpenAI integration coverage remains in `tests/entrypoints/openai/test_return_routed_experts.py`. Model evaluation was not run: this changes request/response transport and validation only; it does not change routing computation or generated model outputs. AI assistance was used to implement and validate this PR.", "url": "https://github.com/vllm-project/vllm/pull/51906", "createdAt": "2026-08-12T02:00:29Z", "updatedAt": "2026-08-13T09:00:55Z", "timestamp": "2026-08-13T09:00:55Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "frontend", "ready" ], "author": "aoshen02", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b61e6ebaa64e9815695c", "signalId": "github:vllm-project/vllm:pull_request:49852", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49852", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MRV2][Multimodal] Enable encoder cuda graph for model runner v2", "text": "## Purpose - Enable encoder cuda graph on model runner v2. ## Test Plan ``` pytest -s -v tests/models/multimodal/generation/test_vit_cudagraph.py ``` ## Test Result All tests should pass ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/49852", "createdAt": "2026-07-26T04:08:30Z", "updatedAt": "2026-08-13T09:00:53Z", "timestamp": "2026-08-13T09:00:53Z", "metrics": { "reactions": 5, "comments": 3 }, "labels": [ "ready", "v1", "nvidia", "mrv2" ], "author": "Isotr0py", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:379cc91fc04b2153d935", "signalId": "github:vllm-project/vllm:pull_request:44544", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] AITER FP8 ASM prefill backend", "text": " ## Purpose Add AITER persistent ASM kernels as a MLA prefill backend. Only for FP8 on gfx950, where it is now the default. #42509 added the (causal) AITER ASM prefill kernel as part of the AITER MLA backend, but it only works for _non-chunked prefills_. For chunked batches _we still currently fall back to FA_. This PR integrates the non-causal ASM prefill kernel so it can be used together with chunked prefill. Dependent on: - [[Bugfix][MLA] Correct final_lse in PS MLA prefill kernel for chunked prefill](https://github.com/ROCm/aiter/pull/3606) (aiter#3606). That PR adds `max_kvlen` to the formal parameters of `get_ps_metadata_info_v1` so we can gate on that. **Hence safe to merge this PR without the AITER PR being merged and released and baked into vllm docker.** ### (TLDR) Perf improvement - 1k/1k ISL/OSL: basically unchanged perf (tested at 32-256 concurrency) - 8k/1k ISL/OSL: **7-9% lower TTFT**, **2-6% higher QPS** (tested at 32-128 concurrency) _Note: I did not yet delete the PS ASM code path that currently sits in `vllm/v1/attention/backends/mla/rocm_aiter_mla.py` which is activated only for pure causal prefills. We need to keep this as we otherwise would have a performance regression until the gate on aiter#3606 goes through. Also makes it fully backwards compatible, even though it might be architecturally advisable to delete the PS ASM code path from rocm_aiter_mla.py once the AITER_ASM backend is fully available._ ## Test Plan ### 1. Patch nightly image with this branch, and re-build aiter from aiter#3606
    ```Dockerfile # docker/Dockerfile.rocm_aiter_asm_src # Build: # docker build --no-cache -f docker/Dockerfile.rocm_aiter_asm_src \\ # --build-arg AITER_REPO=https://github.com/simondanielsson/aiter.git \\ # --build-arg AITER_BRANCH=fix/mla-prefill-ps-final-lse \\ # -t vllm-rocm-aiter-asm-prefill:src . # # Override the base if you want a pinned nightly: # --build-arg BASE_IMAGE=rocm/vllm-dev:nightly_main_20260604 # Override the PR if you rebased / opened a new one: # --build-arg PR_NUMBER=44544 ARG BASE_IMAGE=vllm/vllm-openai-rocm:nightly-a16dbd5b8572d4128be9f10b9dcff4999b594b25 ### ### Stage 1: build AITER from source against THIS image's torch. ### FROM ${BASE_IMAGE} AS build_aiter ARG AITER_REPO=https://github.com/simondanielsson/aiter.git ARG AITER_BRANCH=fix/mla-prefill-ps-final-lse # Restrict to the arch you actually run on; add gfx942 back if you need MI300. ARG AITER_ROCM_ARCH=gfx950 RUN set -eux; \\ apt-get update && apt-get install -y --no-install-recommends git curl; \\ rm -rf /var/lib/apt/lists/* # Clone with submodules (CK etc.) exactly like Dockerfile.rocm_base. RUN set -eux; \\ git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO} /aiter; \\ cd /aiter && git submodule update --init --recursive; \\ git rev-parse HEAD > /aiter_sha.txt # Build deps. We deliberately do NOT touch torch here: the nightly base already # ships the torch we must ABI-match against, so requirements.txt must not pull a # different torch. If it tries to, pin/strip it rather than letting it upgrade. RUN set -eux; \\ cd /aiter; \\ pip install pyyaml; \\ pip install -r requirements.txt # AOT-compile kernels into the wheel (PREBUILD_KERNELS=1) for gfx950, matching # the stock build. This is the path the runtime actually exercises; a JIT build # (PREBUILD_KERNELS unset) is a different code path and not what ships. RUN set -eux; \\ cd /aiter; \\ PREBUILD_KERNELS=1 GPU_ARCHS=${AITER_ROCM_ARCH} \\ python3 setup.py bdist_wheel --dist-dir=/aiter/dist; \\ ls -l /aiter/dist/*.whl ### ### Stage 2: patched nightly vLLM + the freshly built aiter wheel. ### FROM ${BASE_IMAGE} AS final ARG PR_NUMBER=44544 ARG PATCH_URL=https://github.com/vllm-project/vllm/pull/${PR_NUMBER}.diff # Apply the vLLM PR diff onto the installed vllm package in site-packages. RUN set -eux; \\ apt-get update && apt-get install -y --no-install-recommends curl patch; \\ rm -rf /var/lib/apt/lists/*; \\ VLLM_DIR=\"$(python3 -c 'import vllm, os; print(os.path.dirname(vllm.__file__))')\"; \\ REPO_ROOT=\"$(dirname \"${VLLM_DIR}\")\"; \\ echo \"Applying ${PATCH_URL} into ${REPO_ROOT}\"; \\ curl -fsSL \"${PATCH_URL}\" -o /tmp/pr.patch; \\ cd \"${REPO_ROOT}\"; \\ git apply -p1 --include=\"vllm/**\" /tmp/pr.patch; \\ rm /tmp/pr.patch; \\ find \"${VLLM_DIR}\" -name '__pycache__' -type d -exec rm -rf {} + # Install the aiter we just built. --no-deps so pip cannot drag in a different # torch; --force-reinstall so it replaces whatever aiter the nightly shipped. RUN --mount=type=bind,from=build_aiter,src=/aiter/dist,target=/aiter_dist \\ set -eux; \\ pip install --no-deps --force-reinstall /aiter_dist/*.whl # Record which aiter SHA we built so the running container is self-describing. COPY --from=build_aiter /aiter_sha.txt /aiter_sha.txt ``` Build: ``` docker build --build-arg AITER_REV=1781510439 --build-arg PR_REV=60e1866dbd1d6deb1dc9b7f0141590388a1b45f4 -f docker/Dockerfile.rocm_aiter_asm_src-cached -t vllm-rocm-aiter-asm-prefill:local . ```
    ### 2. `vllm bench serve` on DSv3, vs main Run model ```shell docker run \\ --rm \\ --name p-replay-dsv3 \\ --init --network host --ipc host --privileged \\ --cap-add SYS_PTRACE --security-opt seccomp=unconfined \\ --ulimit memlock=-1 --ulimit stack=67108864 \\ --shm-size 256G \\ --group-add video --group-add render \\ --device /dev/kfd --device /dev/dri --device /dev/infiniband \\ -v /sys:/sys \\ -v \"/mnt/nfs_share/huggingface/:/root/.cache/huggingface\" \\ -e HF_HOME=/root/.cache/huggingface \\ -e HF_TOKEN=$HF_TOKEN \\ -e HF_HUB_OFFLINE=1 \\ -e HF_HUB_ENABLE_HF_TRANSFER=0 \\ -e NCCL_MIN_NCHANNELS=112 \\ -e SAFETENSORS_FAST_GPU=1 \\ -e VLLM_ROCM_USE_AITER=1 \\ vllm-rocm-aiter-asm-prefill:local \\ deepseek-ai/DeepSeek-V3 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.8 \\ --tensor-parallel-size 8 \\ --max-model-len 16K \\ |& tee asm_aiter_backend.log ``` - For 1k/1k ISL/OSL, sweep concurrencies 32-256 - For 8k/1k, sweep 32-128 conc. Always warmup with 2x concurrency # prompts, and run for 10x. ```bash docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c 'vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280' ``` ### 3. Acc with DSv3 FP8: this branch vs main ```shell docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c \"pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k\" ``` ### 4. Profile sanity check We need to profile at high enough concurrency to see that the expected kernels are being called during mixed batches with chunked prefills, e.g. using the following commands. Specifically, we need to check 1. main falls back to FA for chunked prefill (even for the causal part!) 2. this branch invokes the ASM kernels for both causal and non-causal prefill during chunked prefill Run profile with 128 concurrent requests, warmup for 32 requests, and run 256 prompts. ### 5. Unit tests ``` python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_backends.py -k \"AITER_ASM and fp8-\" python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_prefill_selector.py -k \"aiter\" ``` ## Test Result ### 2. vllm bench serve #### 128 conc, 8k/1k ISL/OSL Showing the full bench output for this workload. The remaining workload results I just show in a single table. **This branch:** ``` ============ Serving Benchmark Result ============ Successful requests: 1280 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 815.86 Total input tokens: 10484480 Total generated tokens: 1310720 Request throughput (req/s): 1.57 Output token throughput (tok/s): 1606.55 Peak output token throughput (tok/s): 3968.00 Peak concurrent requests: 133.00 Total token throughput (tok/s): 14457.39 ---------------Time to First Token---------------- Mean TTFT (ms): 4032.52 Median TTFT (ms): 1423.93 P99 TTFT (ms): 46993.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.66 Median TPOT (ms): 76.42 P99 TPOT (ms): 82.60 ---------------Inter-token Latency---------------- Mean ITL (ms): 75.66 Median ITL (ms): 37.64 P99 ITL (ms): 406.58 ================================================== ``` **Nightly (vllm/vllm-openai-rocm:nightly-43914dd743ab0500abcd69fe072e02465c944dcf)** ``` ============ Serving Benchmark Result ============ Successful requests: 1280 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 863.50 Total input tokens: 10484480 Total generated tokens: 1310720 Request throughput (req/s): 1.48 Output token throughput (tok/s): 1517.91 Peak output token throughput (tok/s): 3968.00 Peak concurrent requests: 133.00 Total token throughput (tok/s): 13659.75 ---------------Time to First Token---------------- Mean TTFT (ms): 4394.12 Median TTFT (ms): 1569.67 P99 TTFT (ms): 50166.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 79.97 Median TPOT (ms): 81.64 P99 TPOT (ms): 85.45 ---------------Inter-token Latency---------------- Mean ITL (ms): 79.97 Median ITL (ms): 37.22 P99 ITL (ms): 440.01 ================================================== ``` #### Full bench serve results #### 1k/1k Mostly unchanged (within noise factor), because decode heavy workload. | Branch | Concurrency | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | TTFT change (%) | |---|---|---|---|---|---|---| | Main | 32 | 643.5 | 19.55 | 1.51 | | | | This branch | 32 | 656.5 | 19.48 | 1.52 | +0.5% | +2% | | Main | 64 | 837.5 | 26.43 | 2.27 | | | | This branch | 64 | 839.6 | 25.7 | 2.35 | +3.5% | +0.2% | | Main | 128 | 996.54 | 36.86 | 3.29 | | | | This branch | 128 | 1003.5 | 36.5 | 3.27 | -0.6% | +0.7% | | Main | 256| 947.5 | 47.54 | 5.1 | | | | This branch | 256 | 979.5 | 48.5 | 5.00 | -1.9% | +3% | #### 8k/1k 7-9% improved TTFT, 2-6% improved QPS. | Branch | Concurrency | TTFT (P50 ms) | TPOT (P50 ms) | QPS | QPS change (%) | TTFT change (%) | |---|---|---|---|---|---|---| | Main | 32 | 1936.2 | 29.9 | 0.96 | | | | This branch | 32 | 1787.95 | 29.1 | 0.99 | +2.8% | -7.6% | | Main | 64 | 1896.8 | 47.5 | 1.24 ||| | This branch | 64 | 1763.7 | 45.2 | 1.30 | +4.8% | -7% | | Main | 128 | 1569.67 | 81.63 | 1.48 | || | This branch | 128 | 1423.93 | 76.4 | 1.57 | +6% | -9.3% | ### 3. Accuracy #### This branch, with AITER changes ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9431|± |0.0064| | | |strict-match | 5|exact_match|↑ |0.9424|± |0.0064| ``` #### vllm/vllm-openai-rocm:nightly-43914dd743ab0500abcd69fe072e02465c944dcf ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ``` ### 4. Profile sanity check ### On `vllm/vllm-openai-rocm:nightly-3f0a91bb96f8d72e0498b95c166e817deae14d62` **First prefill batch**: causal ASM kernel already gets called (`aiter::mla_pfl_qh192_vh128_m32x8_n128x1_causal1`, in purple) \"image\" **Mixed batched but with pure (new) prefills:** ASM kernel also already called \"image\" **Mixed batch with cached prefill (extends)**: FA fallback (this is the `void ck_tile::kentry<256, 1, ck_tile::FmhaFwdKernel` etc), plus lots of small eager quant ops right before it. **This is what we want to get rid of with this PR.** \"image\" ### This branch **First pure prefill batch:** (causal) ASM kernel called (in purple) \"image\" **Mixed batched but with pure (new) prefills:** (causal) ASM called as expected \"image\" **Mixed batch with cached prefill (extends)**: (**NEW**) Two prefill ASM kernels & 1 decode ASM called \"image\" The first one is the causal (see the trailing `causal1`), and the second one is the non-causal _extend_ kernel: \"image\" **Worth noting**: The get_ps_metadata function is blocking (it's doing 5 sync memcpys unfortunately) so host side looks like this. Doesn't seem like a big issue though since we are only doing this once per forward. \"image\" ### 5. Unit tests Skips coming from non-unit scale tests (e.g. `test_backend_correctness[MLAPrefillBackendEnum.AITER_ASM-2.0-3.0-fp8-16-deepseek-ai/DeepSeek-R1-spec_decode_medium`) because AITER ASM only supports unit scales. ``` $ python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_backends.py -k \"AITER_ASM and fp8-\" ... -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =================================================== 48 passed, 48 skipped, 1344 deselected, 23 warnings in 154.41s (0:02:34) ==================================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ``` $ python3 -m pytest -v /app/repos/vllm/tests/v1/attention/test_mla_prefill_selector.py -k \"aiter\" ============================================================================== test session starts ============================================================================== platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /app/repos/vllm configfile: pyproject.toml plugins: asyncio-1.4.0, anyio-4.13.0 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 19 items / 13 deselected / 6 selected tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability0-fp8-True-True-None] PASSED [ 16%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability1-auto-True-False-fp8] PASSED [ 33%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability2-fp8-True-False-compute capability] PASSED [ 50%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmValidation::test_validate_configuration[capability3-fp8-False-False-R1] PASSED [ 66%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmSelectorPriority::test_aiter_asm_wins_on_gfx950_fp8 PASSED [ 83%] tests/v1/attention/test_mla_prefill_selector.py::TestAiterAsmSelectorPriority::test_falls_through_to_flash_attn_when_not_fp8 PASSED [100%] ``` ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    **BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/44544", "createdAt": "2026-06-04T14:51:11Z", "updatedAt": "2026-08-13T08:59:42Z", "timestamp": "2026-08-13T08:59:42Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "v1" ], "author": "simondanielsson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8edae115faaea2175d42", "signalId": "github:vllm-project/vllm:issue:51798", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51798", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Kimi-K3-NVFP4 on 8xB300 produces degenerate, incoherent output in the reasoning channel on v0.27.0", "text": "### Your current environment We are not able to run `collect_env.py` against the affected build. It was a production deployment that has since been rolled back, so that container no longer exists. The information below is therefore split into the node-level facts (identical across both builds, since it is the same physical machine) and the identity of the affected build. **Node — unchanged between the working and the failing build** | | | |---|---| | GPUs | 8x NVIDIA B300 SXM6 AC (288 GB each), NV18 full mesh | | NVIDIA driver | 595.71.05 | | CUDA runtime | 13.0.88 | | Host OS | Talos Linux (immutable, no host userland), kernel `6.18.34-talos` | | Container userland | Ubuntu 24.04.3 LTS (x86_64), glibc 2.39, Python 3.12.3 — from the vLLM image | | CPU | 224 vCPU, 2 NUMA nodes, KVM guest | **Affected build:** the published `v0.27.0` release image (`vllm/vllm-openai:v0.27.0`) — vLLM 0.27.0, torch 2.13.0+cu130, FlashInfer 0.6.16.post3, Triton 3.7.1, transformers 5.15.0, as pinned for that tag. ### 🐛 Describe the bug ## What we run `RedHatAI/Kimi-K3-NVFP4` served on a single node of 8x B300, tensor parallel 8, weights loaded. Serving flags: ``` --tensor-parallel-size=8 --gpu-memory-utilization=0.95 --max-model-len=262144 --max-num-batched-tokens=8192 --max-num-seqs=64 --kv-cache-dtype=fp8 --attention-config={\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"} --enable-prefix-caching --load-format=fastsafetensors --no-enable-flashinfer-autotune --trust-remote-code --enable-auto-tool-choice --tool-call-parser=kimi_k3 --reasoning-parser=kimi_k3 --enable-prompt-tokens-details ``` ## What we observed On `v0.27.0` the model produces degenerate, incoherent text in the reasoning channel. The prompt was as simple as `hello how are you`. The reasoning content came back as: ``` G now now now now now now now now now now now now now now now now now now now now now now { how are now now now now now now ) [ how are now } [ are now now now now now now now now are now [ is now now is now [ G is now now is now now ) G is now [ now now is now now now now now now now now now now now = is now now = now now is now - is now -G is now - now now - G now -G is this - this - this ``` - **It appeared immediately, on a trivially short prompt.** This did not develop after hours of serving, under load, or after a long-context request. It was present essentially from the first request after the version change, with a prompt of five words. - **Nothing crashed or reported an error.** No CUDA errors, no illegal memory access, no assertion failures, no NaN-related aborts, no engine restarts. The pod stayed `Running` with 0 restarts and kept answering requests at normal latency. The failure was completely silent from an operational standpoint: health checks and metrics looked entirely normal while the output was garbage. That is the part we found most concerning — nothing in our monitoring would have caught this. ## What changed to trigger it We had been serving this model on a **nightly build from 2026-08-09** without this symptom. We then moved the deployment to the **`v0.27.0` release tag** and changed nothing else — same physical node, same GPUs, same driver, same weights, same flags. The degenerate output appeared with that change. As mitigation we rolled the deployment back to that same nightly. We want to be precise about the limits of this evidence: we changed builds, we did not bisect individual commits, so we cannot point at a specific change as the cause. What we can say is that the `v0.27.0` release tag shows it on this configuration and a nightly from a few days before the release did not. Kimi-K3 support is very new and appears to be changing quickly, so we wondered whether the release branch is simply missing model fixes that landed on `main` shortly before the tag was cut — but that is speculation on our part, and someone with a view of what went into the branch is far better placed to judge it than we are. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51798", "createdAt": "2026-08-11T10:01:53Z", "updatedAt": "2026-08-13T08:59:21Z", "timestamp": "2026-08-13T08:59:21Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "kimi", "k3" ], "author": "pyhp2017", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6f927f850e48b924a60e", "signalId": "github:vllm-project/vllm:pull_request:52044", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52044", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle DeepseekV4ForCausalLM in benchmark_moe get_model_params", "text": "## Summary Fixes [#52042](https://github.com/vllm-project/vllm/issues/52042): `benchmarks/kernels/benchmark_moe.py` crashes for `deepseek-ai/DeepSeek-V4-Flash-0731` with `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'`. While fixing it we audited every registered MoE architecture against `get_model_params()`'s dispatch table and found the same latent crash for ~28 more architectures (any MoE model whose config doesn't use the Mixtral `num_local_experts` convention). This PR addresses the whole class of bug, not just DeepSeek-V4. ## Root cause `get_model_params()` dispatches on the model architecture string. Any architecture not matched by an `elif` branch fell through to a silent \"default Mixtral\" branch that reads `config.num_local_experts`. For DeepSeek-V4 — and Gemma-4, EXAONE, OLMoE, Pangu, MiMo-V2, Step3.5, ERNIE-4.5-MoE, etc. — the config uses a different field convention, so it crashed with an opaque `AttributeError`. ## Changes 1. **DeepSeek-V4 fix**: `DeepseekV4ForCausalLM` added to the DeepSeek-V2/V3 branch (`n_routed_experts` / `num_experts_per_tok` / `moe_intermediate_size` — confirmed against the model's `config.json`). 2. **Register all remaining MoE architectures** from `registry.py` under their actual config conventions: - `n_routed_experts` family: `DeepseekForCausalLM`, `LongcatFlashForCausalLM`, `MiMoV2ForCausalLM`, `MiMoV2FlashForCausalLM`, `NemotronHPuzzleForCausalLM`, `PanguEmbeddedForCausalLM`, `PanguProMoEV2ForCausalLM`, `PanguUltraMoEForCausalLM` - `num_experts` / `moe_intermediate_size` family: `Qwen3_5MoeForCausalLM`, `AfmoeForCausalLM`, `BailingMoeForCausalLM`, `BailingMoeV2ForCausalLM`, `BailingMoeV2_5ForCausalLM`, `ExaoneMoeForCausalLM`, `LagunaForCausalLM`, `Lfm2MoeForCausalLM`, `Param2MoEForCausalLM`, `SarvamMLAForCausalLM`, `SarvamMoEForCausalLM` - `num_experts` / `intermediate_size` family: `Cohere2MoeForCausalLM`, `FlexOlmoForCausalLM`, `HYV3ForCausalLM`, `JambaForSequenceClassification`, `MiniCPMForCausalLM`, `OlmoeForCausalLM` - `num_local_experts` family (explicit branch, was the silent fallback): `MixtralForCausalLM`, `Llama4ForCausalLM`, `Llama4ForConditionalGeneration`, `ArcticForCausalLM`, `GptOssForCausalLM`, `GraniteMoeForCausalLM`, `GraniteMoeHybridForCausalLM`, `GraniteMoeSharedForCausalLM`, `MiniMaxM2ForCausalLM`, `PhiMoEForCausalLM` - `top_k_experts` family: `Gemma4ForCausalLM` - `moe_num_experts` family: `Ernie4_5_MoeForCausalLM`, `Step3TextForCausalLM`, `Step3p5ForCausalLM` 3. **Replace the silent Mixtral fallback with an explicit `ValueError`** for unsupported architectures, so future models fail with an actionable message (\"add it to get_model_params()\") instead of a misleading `AttributeError`. 4. **Testability**: `get_model_params` is extracted into a dependency-free `benchmarks/kernels/benchmark_moe_utils.py` (no torch/ray/triton imports), so it can be unit-tested without the benchmark's heavy deps. ## Testing - New unit test `tests/kernels/test_benchmark_moe_model_params.py`: all 59 registered MoE architectures resolve the correct `(E, topk, intermediate_size, hidden_size)` per their field convention, plus the `ValueError` for unsupported architectures. Runs with no GPU / no ray (`15 passed`). - Real configs via transformers: `deepseek-ai/DeepSeek-V3` → `(256, 8, 2048, 7168)`; `mistralai/Mixtral-8x7B-v0.1` → `(8, 2, 14336, 4096)`; DeepSeek-V4 config data → `(256, 6, 2048, 4096)`.", "url": "https://github.com/vllm-project/vllm/pull/52044", "createdAt": "2026-08-12T20:54:27Z", "updatedAt": "2026-08-13T08:58:00Z", "timestamp": "2026-08-13T08:58:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "performance", "deepseek" ], "author": "SayHelloToWorld", "state": "open", "assignees": [ "jeejeelee" ], "change": "updated" } }, { "id": "event:a1a35818334353f97f7d", "signalId": "github:vllm-project/vllm:pull_request:52128", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52128", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Include MM limits predicate in model compute_hash to prevent compile-cache poisoning", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52128", "createdAt": "2026-08-13T08:35:41Z", "updatedAt": "2026-08-13T08:57:18Z", "timestamp": "2026-08-13T08:57:18Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "vschandramourya", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:b6fda634cb75f10e8e6a", "signalId": "github:vllm-project/vllm:pull_request:52129", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52129", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Demote xgrammar accept_tokens rejection log to DEBUG (spec decode replay spam)", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52129", "createdAt": "2026-08-13T08:36:00Z", "updatedAt": "2026-08-13T08:56:46Z", "timestamp": "2026-08-13T08:56:46Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "structured-output" ], "author": "vschandramourya", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:557a12da1678f5e53ade", "signalId": "github:vllm-project/vllm:pull_request:52130", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52130", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Don't use the live processor cache for V1 multimodal profiling", "text": "Withdrawn by author.", "url": "https://github.com/vllm-project/vllm/pull/52130", "createdAt": "2026-08-13T08:36:20Z", "updatedAt": "2026-08-13T08:56:24Z", "timestamp": "2026-08-13T08:56:24Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "vschandramourya", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:727e2cfd5cead630e5d3", "signalId": "github:vllm-project/vllm:pull_request:51484", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51484", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MM][CG][WIP] Encoder CUDA graph support for HunyuanVL", "text": "## Summary Working on encoder CUDA graph support for HunYuanVL (`HunYuanVLForConditionalGeneration`), tracked in #38175. Claiming this model in the tracker. This PR starts with a prerequisite refactor: `HunYuanVisionTransformer` currently processes images by splitting the batch into a Python list and running each image through every layer separately (`O(num_layers * num_images)` split/cat calls). This commit packs all images into a single sequence and uses `cu_seqlens`-scoped attention instead, matching the pattern used by Qwen2-VL/Qwen3-VL. This is a standalone perf win and a prerequisite for graph capture (a per-image Python loop across layers isn't graph-friendly). **Not yet included:** the actual `SupportsEncoderCudaGraph` wiring. HunyuanVL's patch merger reshapes tokens into an explicit `(h, w)` spatial grid per image before a Conv2d-based merge step (with per-row newline tokens and begin/end/sep tokens). Unlike Qwen2-VL's merger, which works on a flat sequence and only cares about total token count, HunyuanVL's merger needs the real aspect ratio, not just the token budget — two images with the same token count but different `(h, w)` produce different merge outputs. This means the \"uniform dummy shape per token budget\" trick used for Qwen2-VL/Qwen3-VL doesn't directly apply here. This looks like the same class of problem noted for MiniCPM-V in #38175 (needs a secondary capture axis / per-shape graph pool rather than token-budget-only grouping). Will follow up with the capture/replay/forward methods once that's sorted out — flagging here in case anyone has thoughts on approach before I go further. ## Why this isn't a duplicate Checked #38175's checklist and comments, `gh pr list --search \"hunyuan\"` — no open PR or issue comment claims HunyuanVL for encoder CUDA graph work as of this PR. ## Test plan - [x] `ruff check` / `ruff format --check` pass on the changed file - [x] Standalone (non-vLLM) PyTorch script confirming packed `cu_seqlens`-masked SDPA attention is numerically equivalent to the old per-image split-attention loop (max abs diff ~1.8e-7) - [ ] Unit tests (`tests/models/multimodal/...`) — not yet run, no GPU available in the environment this was written in - [ ] E2E benchmark — not yet run, same reason I don't have GPU access in my current dev environment, so the actual model forward pass (patch embed → packed attention → merger) is unverified end-to-end. I'll need to run this on GPU hardware before it's mergeable — treating this as a draft/WIP starting point, not something ready for review yet. ## AI assistance disclosure Portions of this PR (the refactor and this description) were written with AI assistance (Claude). I've reviewed the diff and will run and verify tests myself before taking this out of draft.", "url": "https://github.com/vllm-project/vllm/pull/51484", "createdAt": "2026-08-08T07:10:47Z", "updatedAt": "2026-08-13T08:54:48Z", "timestamp": "2026-08-13T08:54:48Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "documentation", "multi-modality", "nvidia" ], "author": "gangula-karthik", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3f911cc31092af996d78", "signalId": "github:vllm-project/vllm:pull_request:44385", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44385", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for chunk_local_cumsum kernels", "text": "## Purpose Adds unit tests for the `chunk_local_cumsum` GDN/FLA kernels, which currently have none. Tests call the public entry point, so both the scalar (3D) and vector (4D) paths are covered, along with varlen (`cu_seqlens` plus caller-supplied chunk indices, as GDN/KDA pass them), partial tail chunks, reverse mode, and the bf16 downcasting store. ## Test Plan `pytest tests/kernels/test_chunk_local_cumsum.py` ## Test Result 88 passed on Arc Pro B70; same cases passed on CUDA.", "url": "https://github.com/vllm-project/vllm/pull/44385", "createdAt": "2026-06-03T06:30:39Z", "updatedAt": "2026-08-13T08:54:27Z", "timestamp": "2026-08-13T08:54:27Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:4eda34af9fd2de93a5b0", "signalId": "github:vllm-project/vllm:pull_request:50496", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50496", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Apertus 1.5", "text": "## General Information This change enables support for serving Apertus v1.5 multimodal (hf tags apertus-ai/Apertus-v1.5-8B, apertus-ai/Apertus-v1.5-70B) Special thanks to @blancsw for creating a workable refactor from https://github.com/swiss-ai/vllm/tree/apertus_integration that optimizes the multimodal inference pipeline and provides a foundation for future integration. Also, special thanks to @tomas-polach for the bug-fix contribution, @AryanAhadinia for the reasoning parser, and @robmsmt for the double BOS-token fix. This PR depends on transformers [PR](https://github.com/huggingface/transformers/pull/47662). ## Test Plan ``` vllm serve apertus-ai/Apertus-v1.5-70B \\ --max-model-len 100000 \\ --tensor-parallel-size 4 \\ --enable-chunked-prefill \\ --async-scheduling \\ --enable-prefix-caching \\ --max-num-seqs 128 \\ --max-num-batched-tokens 65536 \\ --block-size 16 \\ --enable-auto-tool-choice \\ --tool-call-parser apertus \\ --reasoning-parser apertus \\ --chat-template-content-format string ``` ## Tests added Audio tests were not added because default sample rate supported is 16Khz and Apertus 1.5 works with 24Khz hence leading to test failures. and multiple audio case does not exist. ``` XDG_CACHE_HOME=/tmp/vllm-test-cache /opt/venv/bin/python -m pytest \\ tests/models/multimodal/processing/test_common.py \\ -k 'Apertus-v1.5-8B' -v XDG_CACHE_HOME=/tmp/vllm-test-cache /opt/venv/bin/python -m pytest \\ tests/models/multimodal/generation/test_common.py \\ -k apertus_1p5 -v ```", "url": "https://github.com/vllm-project/vllm/pull/50496", "createdAt": "2026-07-31T00:24:08Z", "updatedAt": "2026-08-13T08:54:15Z", "timestamp": "2026-08-13T08:54:15Z", "metrics": { "reactions": 8, "comments": 13 }, "labels": [ "documentation", "new-model", "ready", "multi-modality", "tool-calling" ], "author": "Anunay-Yadav", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a3412b6d6d4522f8f70c", "signalId": "github:vllm-project/vllm:pull_request:52132", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52132", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Enhance engine snapshot management and API lifecycle", "text": "## Purpose Add an opt-in Engine Snapshot deep-sleep path that keeps the API frontend, HTTP listener, and launcher alive while snapshotting and exiting the `EngineCoreProc` subtree. On wake, vLLM validates the snapshot and runtime environment, restores the EngineCore and CUDA state, adopts the restored process, reconnects the frontend, and reopens request admission only after health and semantic checks pass. Design, roadmap, compatibility constraints, and complete experimental results are documented in [RFC #52125: vLLM Native Reusable Engine Snapshot Sleep (L3)](https://github.com/vllm-project/vllm/issues/52125). This work overlaps with #51360 in its use of CRIU and CUDA Checkpoint/Restore for an initialized engine. The snapshot boundary and user-facing lifecycle differ: - #51360 captures the complete initialized service process tree before the HTTP listener is bound and restores it during instance startup. - This PR implements runtime deep sleep. The frontend and HTTP listener remain available for status and wake requests while only the EngineCore subtree is captured, terminated, restored, and reattached. ## Change Summary - Add the Engine Snapshot lifecycle: `READY → DRAINING → PREPARING → SNAPSHOTTING → HIBERNATED` and `HIBERNATED → RESTORING → ATTACHING → VERIFYING → READY`. - Add idle-only request admission control and readiness handling around snapshot transitions. - Add EngineCore detach, expected-exit handling, restored-process adoption, and frontend reconnection. - Add `fake` and `criu_cuda` snapshot providers. - Add atomic snapshot artifact commit, manifest validation, diagnostics, rollback, and cleanup. - Add four L3 resource policies: - `full`: preserve weights, KV cache, and runtime state in the CUDA image. - `discard_kv`: discard the idle KV cache while preserving weights and runtime state in the CUDA image. - `host_backup`: retain weights in host memory, discard the KV cache, and copy weights back to GPU after restore. - `reload_weights`: discard weights and KV cache before capture, then reload weights from model files after restore. - Add development-mode CLI configuration and the Level 3 sleep, wake, and status endpoints. - Add allocator, worker, process-lifecycle, provider, manager, API, middleware, CLI, and serving tests. - Document prerequisites, supported scope, endpoints, snapshot contents, and deployment restrictions. ## Current Scope The initial implementation supports: - Same-machine restore. - One host and one GPU. - `TP=1`, `PP=1`, and `DP=1`. - One Python API server using `UniProcExecutor`. - Idle-only capture with no in-flight request preservation. - `fake` and CRIU/CUDA providers. - Development-mode enablement through `VLLM_SERVER_DEV_MODE=1`. The feature is opt-in and does not change the default `vllm serve` process topology or request path. Cross-machine restore, 1→N restore, multi-GPU snapshot groups, distributed topologies, and in-flight request preservation remain follow-up work described in the RFC. ## Test Plan - Run Ruff on the changed Python files. - Run Python 3.10 mypy on the changed typed modules. - Run the snapshot unit-test suite: `pytest -q tests/snapshot` - Run the Engine Snapshot CLI parser tests: `pytest -q tests/entrypoints/openai/test_cli_args.py -k engine_snapshot` - Exercise the lifecycle and failure paths with the `fake` provider. - Run repeated real CRIU/CUDA capture and restore cycles for all four resource policies. - Verify restored inference output, process cleanup, physical HBM release, snapshot artifacts, and phase-level timing. ## Test Result Test environment: - NVIDIA H20, single GPU. - Qwen3-8B BF16. - `TP=1`, `PP=1`, `DP=1`, one API server. - Non-eager execution. - Model, compilation cache, CRIU images, and snapshots stored on tmpfs. - Six runs per group: one warmup followed by five measured runs. - `reload_weights` was repeated across three groups for 15 measured runs. Median results: | L3 policy | Measured runs | Checkpoint time | Restore time | Artifact size | | --- | ---: | ---: | ---: | ---: | | `reload_weights` | 15 | 2.837 s | 5.948 s | 4.251 GiB | | `discard_kv` | 5 | 10.335 s | 9.361 s | 18.339 GiB | | `host_backup` | 5 | 19.015 s | 10.394 s | 23.960 GiB | | `full` | 5 | 27.242 s | 22.990 s | 49.659 GiB | Restore time is measured from the `/wake_up` request to the first non-empty token from a fixed streaming request, with the final response content also checked. Correctness and resource results: - All 30 formal L3 runs restored successfully and passed the fixed semantic checks. - Every pre-restore model page-residency check reported 100%. - Engine `read_bytes` delta was 0 in all measured runs. - All 30 `HIBERNATED` checks reported 0 MiB for both project-process HBM and full-GPU HBM. - Ruff passed. - Python 3.10 mypy passed. - Snapshot tests: `75 passed`. - CLI parser tests: `3 passed, 30 deselected`. ## AI Assistance AI assistance was used for implementation, testing, analysis, and drafting. The submitter reviewed the changes and owns the final submission. ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52132", "createdAt": "2026-08-13T08:52:39Z", "updatedAt": "2026-08-13T08:53:18Z", "timestamp": "2026-08-13T08:53:18Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "frontend" ], "author": "fuhefei", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:daae12b6aae600bafc5a", "signalId": "github:vllm-project/vllm:pull_request:52087", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52087", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Preserve constant effective K schedule semantics", "text": "## Purpose `num_speculative_tokens_per_batch_size` is currently treated as runtime-variable whenever the schedule is present. Some schedules are syntactically non-uniform but resolve to one positive effective K after the configured runtime maximum is applied: ```text num_speculative_tokens = 3 schedule = [(1, 8, 5), (9, 16, 4)] effective K values = {min(5, 3), min(4, 3)} = {3} ``` Such a schedule cannot change the target verification width. The current behavior has three effects: 1. V1 downgrades `FULL_AND_PIECEWISE` to `PIECEWISE`. 2. Data parallelism disables the schedule and falls back to the raw maximum K, which changes the configured effective K. 3. Prefix-cache-hit requests entering their first decode step are not padded to the effective verification width. This PR: - classifies a schedule as constant only when every entry resolves to the same positive K after runtime clamping; - normalizes the runtime maximum to that effective K so the Scheduler, GPU Runner, metrics, and CUDA Graph descriptor use one width; - applies the V1 graph downgrade and DP fallback only when the effective verification width can vary; and - pads prefix-cache-hit requests with the constant effective K. Malformed schedules, genuinely variable schedules, and K=0 stay on the conservative path. Existing schedule validation remains in place. No CUDA kernel, drafter, verifier, or rejection-sampling rule is changed. ## Why this is not duplicate work Before implementation I checked the issue and searched open PRs by issue number and by the areas `constant effective K`, `speculative schedule`, `full cudagraph capture sizes`, `separate decode capture sizes`, and related terms. - #48944 adds a context-length axis to K selection; this change only classifies schedules that collapse to one runtime width. - #50488 ensures wide uniform-decode batches are represented in the capture grid; this change addresses an earlier configuration/shape classification boundary. - #52000 dispatches more uniform-decode batches to padded FULL graphs; it does not preserve a constant schedule's effective K under data parallelism. - #49652 handles fixed-width autoregressive draft-decode capture; it does not classify the target verification schedule. - #52070 covers always-K=0 behavior. K=0 is deliberately excluded here. ## Test plan ### Focused behavior tests ```bash HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \\ /vllm-workspace/.venv/bin/python -m pytest -q \\ tests/v1/spec_decode/test_dynamic_sd.py ``` ```text 26 passed, 15 warnings in 7.59s ``` Coverage includes: - no schedule, one entry, same-K entries, values clamped to one K, variable K, K=0, and malformed entries; - preserving effective K=2 under DP when the raw maximum is K=3; - prefix-cache-hit first-decode padding with the effective K; and - direct Dispatcher assertions that the normalized runtime shape is uniform and resolves to `CUDAGraphMode.FULL`. ### Repository hooks ```bash uv tool run pre-commit run --files \\ tests/v1/spec_decode/test_dynamic_sd.py \\ vllm/config/speculative.py \\ vllm/config/vllm.py \\ vllm/v1/core/sched/scheduler.py ``` All applicable hooks passed, including Ruff, mypy, typos, SPDX, forbidden-import, and configuration checks. `git diff --check` passed. ## GPU validation Environment: - NVIDIA RTX PRO 6000 Blackwell Server Edition; - Qwen3-8B BF16, V1 model runner; - N-gram speculation, raw maximum K=3, effective schedule K=2; - random input/output length 128/128. ### Single-GPU paired matrix Each concurrency level used five paired repetitions with alternating execution order. | Concurrency | Base PIECEWISE | Patched FULL | Paired change | 95% CI | |---:|---:|---:|---:|---:| | 1 | 94.12 | 92.45 | -1.78% | [-1.98%, -1.57%] | | 2 | 174.05 | 169.16 | -2.81% | [-2.91%, -2.71%] | | 4 | 344.90 | 367.47 | +6.55% | [+6.20%, +6.90%] | | 8 | 658.70 | 638.82 | -3.02% | [-3.31%, -2.73%] | A five-run isolation control with the patched build forced to PIECEWISE was approximately neutral at concurrency 1 (94.05 vs 94.12 output tok/s). This indicates that effective-K normalization itself is not the main regression source; FULL-versus-PIECEWISE performance crosses over by workload shape on this setup. These results do not support a universal performance claim. They show that shape safety and performance admission are separate decisions. A follow-up should select FULL only for empirically winning token/batch buckets, with PIECEWISE as the fail-closed default. ### Two-GPU data-parallel stress test Two DP ranks were tested at temperature 0 for three repetitions at concurrency 7 and 16: - 2304 total requests completed, 0 failed; - base logs show the schedule disabled and a fallback to K=3; - patched logs show both ranks initialized with effective K=2 and retained the schedule. The throughput numbers are intentionally not presented as a same-K speedup because the base and patched runs execute different K. This test validates configuration semantics, rank consistency, and absence of a DP deadlock. ## Current assessment The DP behavior is a concrete configuration-semantics fix: a schedule whose effective K is always 2 should not silently become K=3 merely because DP is enabled. The effective-width propagation and direct Dispatcher test also fix a gap where graphs could be captured without the runtime descriptor selecting the expected replay path. The FULL graph policy needs further reviewer discussion or a separate performance-admission change because the measured crossover is non-monotonic. This PR remains a draft while that scope is resolved. ## AI assistance disclosure OpenAI Codex assisted with prior-art checks, implementation and test drafting, GPU benchmark execution, and documentation. The human submitter must review every changed line, rerun the relevant tests, and be able to defend the change end-to-end before requesting merge.", "url": "https://github.com/vllm-project/vllm/pull/52087", "createdAt": "2026-08-13T03:17:52Z", "updatedAt": "2026-08-13T08:51:35Z", "timestamp": "2026-08-13T08:51:35Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "speculative-decoding", "nvidia" ], "author": "feednetinfra", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9d8fe2fad6ae4d0710de", "signalId": "github:vllm-project/vllm:pull_request:51021", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Gate Torch FP8 scaled-MM on architecture support", "text": "## Purpose Reject Torch FP8 scaled-MM kernels when the underlying PyTorch operator is unavailable. The existing implementation compares a CUDA-style numeric capability against the SM89 threshold. On ROCm, gfx1100 reports capability `110`, so it incorrectly passes this check even though RDNA3 has no native FP8 matrix support and `torch._scaled_mm` rejects the device at runtime. This change mirrors PyTorch’s current ROCm architecture gate for per-tensor, channel-wise, and row-wise Torch FP8 kernels while preserving existing CPU, CUDA, and XPU behavior. It does not change general FP8 support for Triton, upcast, or emulation paths. This does not duplicate #43615: that PR enables RDNA4 paths but does not fix the inherited per-tensor/channel-wise eligibility check on gfx1100. Once `torch.cuda.is_scaled_mm_supported()` is available in vLLM’s supported PyTorch baseline, the mirrored architecture gate can be replaced with that query. AI assistance was used; all changed lines were reviewed by the submitter. ## Test Plan ```bash ruff check vllm/model_executor/kernels/linear/scaled_mm/pytorch.py ruff format --check vllm/model_executor/kernels/linear/scaled_mm/pytorch.py git diff --check HIP_VISIBLE_DEVICES=0 python /tmp/validate_vllm_fp8_scaled_mm.py HIP_VISIBLE_DEVICES=1 python /tmp/validate_vllm_fp8_scaled_mm.py ``` The standalone script validates: - CPU, XPU, CUDA, and ROCm routing behavior. - gfx1100 rejects all three Torch FP8 scaled-MM variants. - gfx1201 accepts all three variants. - The underlying PyTorch operator rejects gfx1100. - Tensor-wise and row-wise execution succeeds on gfx1201 and matches an FP32 reference. ## Test Result - Ruff passed. - Formatting passed. - `git diff --check` passed. - Platform-routing matrix passed: - CPU/XPU supported. - CUDA FP8 capability preserved. - Unsupported ROCm rejected. - Supported ROCm accepted. gfx1100: ```text selected=[False, False, False], operator=rejected ``` gfx1201: ```text selected=[True, True, True], tensorwise=ok, rowwise=ok, relative_error=0.001601 ``` The local vLLM extensions were ABI-incompatible with the installed PyTorch, so validation covered vLLM’s Python selection logic and the underlying PyTorch operator directly rather than a full end-to-end model serving run. ---
    Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] No documentation update is required for this kernel-selection fix.
    ", "url": "https://github.com/vllm-project/vllm/pull/51021", "createdAt": "2026-08-04T12:43:29Z", "updatedAt": "2026-08-13T08:46:58Z", "timestamp": "2026-08-13T08:46:58Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "ready" ], "author": "sstamenk", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8fdc79725fc364b53a4d", "signalId": "github:vllm-project/vllm:pull_request:50431", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50431", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[sleep functionality] code refactor about sleep/wake_up", "text": "## Purpose 1. For now, frontend multiprocessing the sleep and wake_up are utility commands and in `async_llm` the call will await the request future. So, it is a sync operation. The log should be deleted as v0 has been completely deleted. 2. remove hard code strings 3. more detailed logs about the scheduler state change ## Test Plan NA ## Test Result NA ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/50431", "createdAt": "2026-07-30T13:06:06Z", "updatedAt": "2026-08-13T08:46:36Z", "timestamp": "2026-08-13T08:46:36Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "v1" ], "author": "andyxning", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c76b39f5909dcc86a855", "signalId": "github:vllm-project/vllm:pull_request:49577", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49577", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Mask Replay", "text": "## Summary This PR adds experimental support for **sampling distribution replay**. A sampling mask represents the vocabulary support retained after top-k/top-p filtering. It is not an attention mask and does not affect causal attention or KV-cache behavior. When enabled, vLLM returns the sampling support for each generated token in a CSR-style representation: ```python SamplingMask( token_ids=[...], offsets=[0, ..., len(token_ids)], ) ``` The feature is opt-in and does not change default generation behavior. ## Motivation With top-p sampling, rollout probabilities are normalized over a truncated vocabulary, while training commonly recomputes probabilities over the full vocabulary. This creates a systematic mismatch in importance ratios and KL estimates. Sampling replay returns the rollout-time support so training can recompute log-probabilities over the same distribution. ## Experiment We compare GRPO training with sampling replay enabled and disabled under top-p sampling. For each generated token: - `logp_diff = training_logprob - rollout_logprob` - `importance_ratio = exp(logp_diff)` Ideally, the rollout and training distributions match, so `logp_diff` is close to `0` and `importance_ratio` is close to `1`. ![Sampling replay metrics](https://github.com/user-attachments/assets/baa591e0-7a21-4b0a-9a62-488685ec2789) ### Mean Importance Ratio With replay enabled, the mean importance ratio remains tightly centered around `1`. Without replay, it is consistently below `1` and shows larger fluctuations. This indicates that replay restores the rollout-time top-p normalization during training, while the non-replay baseline compares the truncated rollout distribution with a full-vocabulary training distribution. ### Mean Log-Probability Difference With replay enabled, the mean log-probability difference remains close to `0`. Without replay, it has a persistent negative bias. This is expected: a token probability normalized over the full vocabulary is generally lower than the probability normalized over the retained top-p support. ### Standard Deviation of the Log-Probability Difference Replay lowers the standard deviation of the log-probability difference. This means it improves not only the average alignment, but also token-level consistency between rollout and training probabilities. Lower variance produces more stable importance weights and reduces the impact of probability-ratio outliers. ### Approximate KL Replay produces a lower and more stable approximate KL estimate, with fewer large spikes. This indicates that the training-time distribution stays closer to the rollout-time distribution throughout optimization. ### Conclusion Across all four metrics, sampling replay improves rollout/training distribution consistency: - `importance_ratio_mean` stays close to `1`; - `logp_diff_mean` stays close to `0`; - `logp_diff_std` is lower; - `approx_kl` is lower and more stable. These results demonstrate the intended algorithmic effect of sampling replay: improved probability alignment and lower-variance importance weights. This experiment does not, by itself, establish an improvement in downstream reward or task accuracy. ## Limitations The current implementation requires: - Model Runner V2; - `logprobs_mode=\"processed_logprobs\"`; - `SamplingParams.logprobs=1`; - `temperature > 0`. Speculative decoding, custom samplers, and custom logits processors are not currently supported.", "url": "https://github.com/vllm-project/vllm/pull/49577", "createdAt": "2026-07-23T10:22:03Z", "updatedAt": "2026-08-13T08:44:42Z", "timestamp": "2026-08-13T08:44:42Z", "metrics": { "reactions": 3, "comments": 33 }, "labels": [ "documentation", "frontend", "ready", "v1", "mrv2", "rust" ], "author": "vx120", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:4b77d7b60fe92802de7c", "signalId": "github:vllm-project/vllm:pull_request:52123", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52123", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Update CODEOWNERS", "text": "## Purpose As discussed offline with @Isotr0py ## Test Plan ## Test Result ---
    Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
    ", "url": "https://github.com/vllm-project/vllm/pull/52123", "createdAt": "2026-08-13T08:21:34Z", "updatedAt": "2026-08-13T08:44:34Z", "timestamp": "2026-08-13T08:44:34Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "ci/build" ], "author": "DarkLight1337", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:0c4392846ab398116783", "signalId": "github:vllm-project/vllm:pull_request:50525", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50525", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Fix NUMA node binding for multi-TP CPU workers", "text": "## Purpose On CPU, each tensor-parallel worker bound its memory to the NUMA node of the **first allowed CPU** (`get_allowed_cpu_list()[0]`), so every rank in a multi-TP run landed on the **same** NUMA node. This defeats NUMA locality: all workers compete for one node's memory bandwidth while other nodes sit idle, and the per-worker KV cache sizing is computed against a single node. This PR binds each rank round-robin across the visible memory nodes: ```python current_numa_node = allowed_memory_nodes[rank % len(allowed_memory_nodes)] ``` so worker `rank` maps to a distinct NUMA node when multiple nodes are available. Because the node now comes directly from `get_visible_memory_node()`, it is always a valid memory node by construction; the previous `cpu_core.numa_node not in allowed_memory_nodes` guard and the s390x synthetic-book-ID fallback are no longer reachable and have been removed. A `logger.debug` line records the node each rank binds to. ## Not a duplicate This is distinct from #47336, which distributes **data-parallel EngineCore processes** across NUMA nodes at the process level. This PR fixes **tensor-parallel workers** within a single engine (`CPUWorker.__init__`) that were all binding to the first CPU's node. Different layer, different mechanism, no overlap in changed logic. ## Notes AI assistance was used while preparing this change. The author has reviewed every changed line and understands the change end-to-end.", "url": "https://github.com/vllm-project/vllm/pull/50525", "createdAt": "2026-07-31T05:15:37Z", "updatedAt": "2026-08-13T08:39:35Z", "timestamp": "2026-08-13T08:39:35Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "cpu" ], "author": "Sundaresan-G", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a826577852d619c1bc95", "signalId": "github:vllm-project/vllm:pull_request:52122", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52122", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MiniCPM-V] Fix AssertionError in get_dummy_mm_data when passing VideoDummyOptions to _get_dummy_images", "text": "**Issue**: pytest tests/lora/test_minicpmv_tp.py::test_minicpmv_lora raises AssertionError on non-CUDA platforms (e.g., XPU). **Root Cause**: Commit 9a276d6375 added a runtime assertion to _get_dummy_images in dummy_inputs.py: assert overrides is None or isinstance(overrides, ImageDummyOptions) However, MiniCPMVDummyInputsBuilder.get_dummy_mm_data in minicpmv.py had always been passing video_overrides (type VideoDummyOptions) directly to _get_dummy_images when constructing per-frame dummy images for video. This type mismatch was silently ignored before the assertion was introduced. **Why upstream CI missed it**: test_minicpmv_lora is decorated with @pytest.mark.skipif(current_platform.is_cuda_alike(), ...), so it is skipped entirely on CUDA CI. **Fix**: In minicpmv.py, convert VideoDummyOptions before passing to _get_dummy_images: - Apply num_frames override to num_video_frames - Convert width/height into an ImageDummyOptions instance", "url": "https://github.com/vllm-project/vllm/pull/52122", "createdAt": "2026-08-13T08:06:44Z", "updatedAt": "2026-08-13T08:39:30Z", "timestamp": "2026-08-13T08:39:30Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready" ], "author": "mayuyuace", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6e993387a05531caa712", "signalId": "github:vllm-project/vllm:pull_request:52126", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52126", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: prevent PyNvVideoCodec decoder slot limit bypass via ClassVar shadowing", "text": "Move mutable decoder-pool state (_decoder_slots, _active_decoder_slots, _decoder_slot_cond, _max_decoder_slots) out of ClassVar attributes on PyNvVideoCodecVideoBackendMixin into a module-level _PyNvDecoderPool singleton. This prevents subclass augmented assignment (cls._active += 1) from creating independent shadow counters per concrete class, which allowed multiple subclasses to independently exceed the configured process-wide hw_decoders limit.", "url": "https://github.com/vllm-project/vllm/pull/52126", "createdAt": "2026-08-13T08:29:00Z", "updatedAt": "2026-08-13T08:33:06Z", "timestamp": "2026-08-13T08:33:06Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "multi-modality" ], "author": "jperezdealgaba", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e4dd529b0050bdc89f69", "signalId": "github:vllm-project/vllm:pull_request:49793", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode][Perf] Fuse the MTP trailing all-reduce; local-argmax draft tokens", "text": "Two optimizations on the DeepSeek-V3.2 / GLM-5.2 MTP draft path. - **Fuse the trailing all-reduce into the final RMSNorm** on the non-sequence-parallel path, as the main model already does at layer boundaries. The sequence-parallel path is unchanged. - **Greedy draft tokens via vocab-parallel local argmax** (`get_top_tokens`), skipping the full-vocab all-gather in `compute_logits`. The proposer already probes for this via `use_local_argmax_reduction`; this adds the model side.", "url": "https://github.com/vllm-project/vllm/pull/49793", "createdAt": "2026-07-25T08:16:24Z", "updatedAt": "2026-08-13T08:31:18Z", "timestamp": "2026-08-13T08:31:18Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "speculative-decoding", "ready", "v1", "mrv2" ], "author": "zhou9402", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c6851069c8cec7fb0152", "signalId": "github:vllm-project/vllm:issue:46589", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:46589", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Segfault encountered in qwen3.5-27B long video (>2G)inference", "text": "### Your current environment
    The output of python collect_env.py ```text Your output of `python collect_env.py` here 2026-06-24 07:11:18 (110 B/s) - 'collect_env.py' saved [35090/35090] Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (aarch64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 15.0.7 CMake version : version 4.3.2 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cpu Is debug build : False CUDA used to build PyTorch : None ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.11.15 (main, May 13 2026, 07:56:17) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.10.0-216.0.0.115.oe2203sp4.aarch64-aarch64-with-glibc2.35 ============================== CPU Info ============================== Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: HiSilicon BIOS Vendor ID: HiSilicon Model name: Kunpeng-920 BIOS Model name: HUAWEI Kunpeng 920 5250 Model: 0 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 4 Stepping: 0x1 BogoMIPS: 200.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs L1d cache: 12 MiB (192 instances) L1i cache: 12 MiB (192 instances) L2 cache: 96 MiB (192 instances) L3 cache: 192 MiB (8 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-23 NUMA node1 CPU(s): 24-47 NUMA node2 CPU(s): 48-71 NUMA node3 CPU(s): 72-95 NUMA node4 CPU(s): 96-119 NUMA node5 CPU(s): 120-143 NUMA node6 CPU(s): 144-167 NUMA node7 CPU(s): 168-191 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] numpy==1.26.4 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+cpu [pip3] torch_npu==2.10.0 [pip3] torchaudio==2.10.0+cpu [pip3] torchvision==0.25.0+cpu [pip3] transformers==5.5.3 [pip3] triton==3.5.0 [pip3] triton_ascend==3.2.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/Ascend/cann-9.0.0/lib64:/usr/local/Ascend/cann-9.0.0/lib64/plugin/opskernel:/usr/local/Ascend/cann-9.0.0/lib64/plugin/nnengine:/usr/local/Ascend/cann-9.0.0/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/python3.11.15/lib::/usr/local/lib OMP_NUM_THREADS=1 TORCH_DEVICE_BACKEND_AUTOLOAD=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
    ### 🐛 Describe the bug log: ``` (APIServer pid=89271) INFO 06-24 02:35:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% !!!!!!! Segfault encountered !!!!!!! File \"\", line 0, in avio_seek File \"\", line 0, in avformat_open_input File \"Objects/typeobject.c\", line 1103, in type_call File \"Objects/call.c\", line 214, in _PyObject_MakeTpCall File \"Python/ceval.c\", line 4769, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 59, in method_vectorcall File \"Objects/call.c\", line 257, in _PyVectorcall_Call File \"Objects/call.c\", line 328, in _PyObject_Call File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 89, in method_vectorcall File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"Python/ceval.c\", line 7349, in do_call_core File \"Python/ceval.c\", line 5376, in _PyEval_EvalFrameDefault File \"./Include/internal/pycore_ceval.h\", line 73, in _PyEval_EvalFrame File \"Python/ceval.c\", line 6434, in _PyEval_Vector File \"./Include/internal/pycore_call.h\", line 92, in _PyObject_VectorcallTstate File \"Objects/classobject.c\", line 67, in method_vectorcall File \"./Modules/_threadmodule.c\", line 1124, in thread_run File \"Python/thread_pthread.h\", line 241, in pythread_wrapper File \"\", line 0, in 0xffffffffffffffff ``` ``` vllm serve /home/model_weights/Qwen3.5-27B \\ --served-model-name \"qwen3.5\" \\ --host 0.0.0.0 \\ --port 8888 \\ --data-parallel-size 1 \\ --tensor-parallel-size 8 \\ --max-num-batched-tokens 16384 \\ --max-num-seqs 128 \\ --gpu-memory-utilization 0.92 \\ --compilation-config '{\"cudagraph_capture_sizes\":[1,6,12,18,24,30,36,42,48,50,54,72,78,84,90,96,102,108,144,192], \"cudagraph_mode\":\"FULL_DECODE_ONLY\"}' \\ --speculative_config '{\"method\": \"qwen3_5_mtp\", \"num_speculative_tokens\": 5, \"enforce_eager\": true}' \\ --trust-remote-code \\ --async-scheduling \\ --allowed-local-media-path / \\ --mm-processor-cache-gb 0 \\ ``` long video: https://modelscope.cn/datasets/Byters/HourVideo/tree/master/videos 01236d91-23a2-49be-8f9f-d25cd3e126f6.mp4 test curl: ``` video_path=${1:-'/xxx/xxx.mp4'} curl http://localhost:8888/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d \"{ \\\"model\\\": \\\"qwen3.5\\\", \\\"messages\\\": [ {\\\"role\\\": \\\"system\\\", \\\"content\\\": \\\"You are a helpful assistant.\\\"}, {\\\"role\\\": \\\"user\\\", \\\"content\\\": [ {\\\"type\\\": \\\"video_url\\\", \\\"video_url\\\": {\\\"url\\\": \\\"file://${video_path}\\\"}}, {\\\"type\\\": \\\"text\\\", \\\"text\\\": \\\"Summarize the video content in one sentence.\\\"} ]} ] }\" ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/46589", "createdAt": "2026-06-24T07:36:42Z", "updatedAt": "2026-08-13T08:30:54Z", "timestamp": "2026-08-13T08:30:54Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug" ], "author": "shuixiaoer", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:10284af0bd536a107c4e", "signalId": "github:vllm-project/vllm:pull_request:50732", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50732", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[PD] Emit inactive KV blocks for decode affinity", "text": "Publish GPU block inactivity when the last request reference is released so external schedulers can maintain accurate decode load signals. ## Purpose Add a new KV cache event, `BlockInactive`, so external PD schedulers can track when a GPU prefix-cached block is no longer actively referenced. Today, KV event consumers can observe `BlockStored` and `BlockRemoved`, but cannot reliably detect that the last request released a block while it may still remain in the prefix cache. Without a refcount-zero signal, decode-affinity load accounting (`active_blocks`) tends to only increase or needs fragile heuristics. This PR: - Adds `BlockInactive` to `KVEventBatch`, emitted from `BlockPool.free_blocks()` when `ref_cnt` drops to 0 for a hashed GPU block (`medium=MEDIUM_GPU`). - Publishes it through the existing KV event path (`take_events()` / ZMQ). No new scheduler-loop wiring. - Gates emission for Prefill-only instances: `kv_role == \"kv_producer\"` still emits `BlockStored` / `BlockRemoved`, but does **not** emit `BlockInactive`, so Prefill frees do not pollute decode load signals. - Decode (`kv_consumer`), union (`kv_both`), and non-PD setups continue to emit Inactive via `KVTransferConfig.should_emit_block_inactive` (`kv_role != \"kv_producer\"`). Related issue: #50730 ## Test Plan pytest tests/v1/core/test_prefix_caching.py \\ -k \"free_blocks_emits_block_inactive or free_blocks_skips_block_inactive or kv_transfer_should_emit_block_inactive\" \\ -q Coverage: free_blocks emits BlockInactive when enable_block_inactive_events=True. Prefill-style gate (enable_block_inactive_events=False) still emits BlockStored, but not BlockInactive. KVTransferConfig.should_emit_block_inactive is False for kv_producer, True for kv_consumer / kv_both / default. ## Test Result Unit tests above pass locally. Existing BlockStored / BlockRemoved behavior is unchanged. Old KV event consumers can ignore the unknown BlockInactive type. Validated end-to-end in a 1P1D (one Prefill + one Decode) setup. Additional tests are still in progress. Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50732", "createdAt": "2026-08-02T08:21:18Z", "updatedAt": "2026-08-13T08:26:55Z", "timestamp": "2026-08-13T08:26:55Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "needs-rebase", "kv-connector" ], "author": "tianmengx12-max", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:46178e7c5d4e19bebec3", "signalId": "github:vllm-project/vllm:issue:51975", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51975", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Bugs in kimi-k3 docker image", "text": "### Your current environment
The output of python collect_env.py ```text N/A - discussion about docker image ```
### 🐛 Describe the bug The [kimi-k3](https://hub.docker.com/layers/vllm/vllm-openai/kimi-k3/images/sha256-fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e) image and other [kimi k3 images](https://hub.docker.com/r/vllm/vllm-openai/tags?name=kimi) are outdates and still feature a bug that was fixed in vLLM commit beca88e59ea75a7aa1af72a5ae50188fa91d4e3d (\"[BugFix][K3] Skip moe_intermediate padding when EP is enabled\". ). The vllm [recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3?nodes=4) for kimi-k3 still refers to this image with a bug. The desired fix is to push the fixed kimi-k3 docker image to docker hub. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51975", "createdAt": "2026-08-12T09:41:02Z", "updatedAt": "2026-08-13T08:26:37Z", "timestamp": "2026-08-13T08:26:37Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "kimi", "k3" ], "author": "rwood-97", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e57f06f830714a587888", "signalId": "github:vllm-project/vllm:pull_request:44834", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU][Zen] Route Int8 MoE inference through zentorch on AMD", "text": "## Purpose Routes CPU MoE on AMD Zen through a zentorch-backed Int8 W8A8 fused-MoE backend: the per-expert int8 GEMMs are dispatched through torch.ops.zentorch.*, mirroring the existing zentorch W8A8 dense-linear integration. On non-Zen CPUs, or when zentorch is not installed, the int8 MoE oracle falls through to the existing backends, so behaviour elsewhere is unchanged. - CPUInt8Experts: per-expert int8 experts kernel dispatching through torch.ops.zentorch (zentorch_dynamic_qlinear / zentorch_fused_moe). - Int8 MoE oracle: add the CPU_ZEN backend, gated on is_zen_cpu(); - LLM-Compressor per-expert W8A8 checkpoint loading (GPT-OSS), including the SwiGLU-OAI interleaved weight layout for the fused W8A8 path. - Centralize the topk_softmax / sigmoid CPU fallback in _custom_ops. ## Test Plan lm_eval --model vllm --model_args pretrained=amd/gpt-oss-20b-BF16-w8a8-llmcompressor-v0.10.0.2,dtype=bfloat16,max_model_len=4096 --tasks gsm8k --batch_size auto --trust_remote_code --num_fewshot 5 --apply_chat_template --log_samples --gen_kwargs \"max_gen_toks=2048\" ## Test Result Verified GPT-OSS / Qwen3 MoE W8A8 inference on AMD Zen CPU. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/44834", "createdAt": "2026-06-08T05:51:47Z", "updatedAt": "2026-08-13T08:26:06Z", "timestamp": "2026-08-13T08:26:06Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "rocm", "ci/build", "v1", "cpu", "gpt-oss", "quantization", "mistral" ], "author": "ganeshr10", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f32bbebca08f43b27240", "signalId": "github:vllm-project/vllm:pull_request:37383", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:37383", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Doc] Add comprehensive --speculative-config documentation", "text": "## Summary - Adds a new `--speculative-config` Reference section to the speculative decoding documentation (`docs/features/speculative_decoding/README.md`) that enumerates all user-facing configuration keys accepted by `--speculative-config` (CLI) and the `speculative_config` Python dict - Documents every key's type, default value, valid ranges/values, and which speculative method each key applies to - Includes usage examples for CLI and Python API, plus practical examples for draft model with quantization, EAGLE with tree speculation, n-gram with custom window, and suffix decoding with tuned parameters - All information is derived directly from `vllm/config/speculative.py` (`SpeculativeConfig` dataclass) - Fixed `enforce_eager` description: it controls CUDA graph capture for the speculative model (not the draft model's `ModelConfig.enforce_eager`), and defaults to allowing CUDA graphs when `None` Fixes #35573 ## Test plan - [x] Cross-reference each documented key against `vllm/config/speculative.py` to confirm accuracy - [x] Cross-reference `enforce_eager` usage in `vllm/v1/worker/gpu_model_runner.py` and `vllm/v1/spec_decode/eagle.py` to confirm description - [ ] Verify the documentation renders correctly with `mkdocs serve` - [ ] Confirm no regressions in existing documentation sections (Lossless guarantees, Resources, etc.)", "url": "https://github.com/vllm-project/vllm/pull/37383", "createdAt": "2026-03-18T04:23:51Z", "updatedAt": "2026-08-13T08:24:17Z", "timestamp": "2026-08-13T08:24:17Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "documentation", "needs-rebase" ], "author": "karanb192", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:7c3e977fe50adaca94b3", "signalId": "github:vllm-project/vllm:issue:37304", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:37304", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Language Models Test (Extended Generation) test_models[False-False-5-32-bigcode/starcoder2-3b] test issue", "text": "### Your current environment main ### 🐛 Describe the bug I got the test to fail under the following two situations: 1. We're trying to upgrade PyTorch from 2.10 to 2.11. The test succeeds in PyTorch 2.10, but fails in PyTorch 2.11 ([logs](https://buildkite.com/vllm/ci/builds/56028#019ce7ae-eb9b-41be-90ae-310763ec6438)) 2. In PyTorch 2.10, if I add [--enforce-eager](https://github.com/vllm-project/vllm/blob/3717a4dd475e6a936df0c84b043743310368e766/tests/models/language/generation/test_common.py#L178), the test fails. The error message looks the same in both cases. ([logs](https://buildkite.com/vllm/ci/builds/56028#019ce7ae-eb9b-41be-90ae-310763ec6438)) ``` [2026-03-13T16:18:07Z] E AssertionError: Test1: -- [2026-03-13T16:18:07Z] E Matched tokens: [222, 40, 494, 447, 9009, 98, 828, 366] [2026-03-13T16:18:07Z] E hf: '\\n# + [markdown] id=\"5f7o229v00-L\"\\n# **1950** - ' {59: -4.088154315948486, 58: -4.088154315948486, 56: -4.213154315948486, 55: -4.213154315948486, 54: -4.213154315948486} [2026-03-13T16:18:07Z] E vllm: '\\n# + [markdown] id=\"68284477\"\\n# **1950**\\n#\\n# - ' {59: Logprob(logprob=-4.071040153503418, rank=1, decoded_token='6'), 56: Logprob(logprob=-4.196040153503418, rank=2, decoded_token='3'), 54: Logprob(logprob=-4.196040153503418, rank=3, decoded_token='1'), 57: Logprob(logprob=-4.196040153503418, rank=4, decoded_token='4'), 55: Logprob(logprob=-4.196040153503418, rank=5, decoded_token='2')} ``` Here's a [link to the test](https://github.com/vllm-project/vllm/blob/3717a4dd475e6a936df0c84b043743310368e766/tests/models/language/generation/test_common.py#L198-L203) I'm trying to figure out if we should block the PyTorch 2.11 release on this test. Given datapoint (2): the test fails when adding --enforce-eager, I think there is something wrong with the test and that we shouldn't block on this. I don't have very much context over this test though. @hmellor I see you poked around at this file recently for Gemma and Gemma2. Do you have any thoughts here? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37304", "createdAt": "2026-03-17T14:47:03Z", "updatedAt": "2026-08-13T08:22:28Z", "timestamp": "2026-08-13T08:22:28Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "bug", "stale" ], "author": "zou3519", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e186815c687b9cdd2e7c", "signalId": "github:vllm-project/vllm:issue:29341", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:29341", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: sleep level 2 causes gibberish outputs", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 20.04.6 LTS (x86_64) GCC version : (Ubuntu 10.5.0-1ubuntu1~20.04) 10.5.0 Clang version : Could not collect CMake version : version 3.27.7 Libc version : glibc-2.31 ============================== PyTorch Info ============================== PyTorch version : 2.9.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime) Python platform : Linux-5.15.0-1048-aws-x86_64-with-glibc2.31 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.1.105 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA H100 80GB HBM3 Nvidia driver version : 575.57.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 48 bits physical, 48 bits virtual CPU(s): 96 On-line CPU(s) list: 0-95 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7R13 Processor Stepping: 1 CPU MHz: 3491.190 BogoMIPS: 5299.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 3 MiB L1i cache: 3 MiB L2 cache: 48 MiB L3 cache: 384 MiB NUMA node0 CPU(s): 0-47 NUMA node1 CPU(s): 48-95 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.5.2 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.16.0 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.3.0 [pip3] nvidia-ml-py==13.580.82 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.3.20 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.9.0 [pip3] torchaudio==2.9.0 [pip3] torchvision==0.24.0 [pip3] transformers==4.57.2 [pip3] triton==3.5.0 [conda] flashinfer-python 0.5.2 pypi_0 pypi [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.16.0 pypi_0 pypi [conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi [conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi [conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.3.0 pypi_0 pypi [conda] nvidia-ml-py 13.580.82 pypi_0 pypi [conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi [conda] nvidia-nvshmem-cu12 3.3.20 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi [conda] pyzmq 27.1.0 py312hcf8288c_1 [conda] torch 2.9.0 pypi_0 pypi [conda] torchaudio 2.9.0 pypi_0 pypi [conda] torchvision 0.24.0 pypi_0 pypi [conda] transformers 4.57.2 pypi_0 pypi [conda] triton 3.5.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-10 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib:/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/nvidia/nvjitlink/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda-12.1/lib:/usr/local/cuda-12.1/lib64:/usr/local/cuda-12.1:/usr/local/cuda-12.1/targets/x86_64-linux/lib/:/usr/local/cuda-12.1/extras/CUPTI/lib64:/usr/local/lib:/usr/lib::/opt/amazon/openmpi/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/efa/lib VLLM_LOGGING_LEVEL=ERROR PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
### 🐛 Describe the bug When using the sleep mode level 2, the model produces gibberish completions: ```python from vllm import LLM llm = LLM(model=\"openai/gpt-oss-20b\", enable_sleep_mode=True) llm.sleep(level=2) llm.wake_up() prompts = [[{\"role\": \"user\", \"content\": \"Where is the Machu Picchu located?\"}]] outputs = llm.chat(prompts) print(repr(outputs[0].outputs[0].text)) ``` ``` 'ocado \\'\" chemical optimal UriWord Beef nwanyị Mehmet性质Usuarioಡೆಯ profanity դեպիSleep Columbia' ``` **notes**: - with sleep level = 1, there is no such issue: ``` 'analysisUser asks location. Need to answer: Machu Picchu located' ``` - the same issue occurs with version 0.10.2, 0.11.0, 0.11.1 and 0.11.2 - the same issue occurs with transformers 4.57.0 and 5.0.0.dev0 - the same issue occurs with models Qwen3 and GPT-OSS, so I guess it affects all models. - `model_impl=\"transformers\"` doesn't solve the issue ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29341", "createdAt": "2025-11-24T20:31:46Z", "updatedAt": "2026-08-13T08:21:44Z", "timestamp": "2026-08-13T08:21:44Z", "metrics": { "reactions": 0, "comments": 15 }, "labels": [ "bug", "stale" ], "author": "qgallouedec", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:690f8a998b62d95b86f7", "signalId": "github:vllm-project/vllm:issue:51737", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51737", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Fix]: MTP speculative decoding crashes with RuntimeError on heterogeneous per-layer head_dim, fixed below and created a PR", "text": "### Bug Description Serving Gemma 4 31B with its native assistant model (`google/gemma-4-31B-it-assistant`) on `vllm/vllm-openai:latest` crashes during parameter loading with `RuntimeError: start (0) + length (4096) exceeds dimension size (2048)`. ### Root Cause Gemma 4 uses heterogeneous attention layers (45 sliding layers with `head_dim=256`, 15 full layers with `global_head_dim=512`). During MTP draft layer parameter loading, `_ColumnvLLMParameter.load_qkv_weight` uses global `global_head_dim=512` for all layers, slicing past the 2048-dim bounds of sliding layers. ### Reproduction ```bash docker run --gpus all --ipc host vllm/vllm-openai:latest \\ --model RedHatAI/gemma-4-31B-it-FP8-dynamic \\ --speculative-config '{\"method\": \"draft_model\", \"model\": \"google/gemma-4-31B-it-assistant\", \"num_speculative_tokens\": 8}' ``` ### Traceback ```text File \"vllm/model_executor/parameter.py\", line 195, in load_qkv_weight param_data = param_data.narrow(self.output_dim, shard_offset, shard_size) RuntimeError: start (0) + length (4096) exceeds dimension size (2048) ``` ### Proposed Fix **`vllm/model_executor/parameter.py`**: ```python max_param = param_data.shape[self.output_dim] max_loaded = loaded_weight.shape[self.output_dim] shard_offset = min(shard_offset, max_param) shard_size = min(shard_size, max_param - shard_offset) loaded_start = min(shard_id_int * shard_size, max_loaded) shard_size = min(shard_size, max_loaded - loaded_start) if shard_size <= 0: return ``` **`vllm/model_executor/model_loader/weight_utils.py`**: ```python if param.size() != loaded_weight.size() and param.dim() == 1 and loaded_weight.dim() == 1: min_len = min(param.size(0), loaded_weight.size(0)) param.data[:min_len].copy_(loaded_weight[:min_len]) ```", "url": "https://github.com/vllm-project/vllm/issues/51737", "createdAt": "2026-08-10T22:21:56Z", "updatedAt": "2026-08-13T08:19:51Z", "timestamp": "2026-08-13T08:19:51Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "quivent", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a074692677959bf5f22e", "signalId": "github:vllm-project/vllm:issue:51744", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51744", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm/vllm-openai:latest fails to start Gemma4 with Transformers 5.15.0", "text": "### Your current environment ```text Docker image: vllm/vllm-openai:latest vLLM: 0.27.0 Transformers: 5.15.0 Model: yasu-oh/gemma-4-31B-it-qat-NVFP4 Tensor parallel size: 2 KV cache dtype: fp8 ``` The Transformers version in the image can be confirmed with: ```bash docker run --rm \\ --entrypoint python3 \\ vllm/vllm-openai:latest \\ -c 'import vllm, transformers; print(\"vLLM:\", vllm.__version__); print(\"Transformers:\", transformers.__version__)' ``` Output: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` ### 🐛 Describe the bug The current official release image `vllm/vllm-openai:latest` contains: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` When serving: ```text yasu-oh/gemma-4-31B-it-qat-NVFP4 ``` vLLM fails during startup with: ```text transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via config.per_layer_config[i].head_dim. ``` The exception occurs in: ```text vllm/transformers_utils/model_arch_config_convertor.py ``` while accessing the Gemma4 `head_dim`. ### Workaround Downgrading only Transformers to 5.14.1 while keeping the same `vllm/vllm-openai:latest` base image and serving configuration resolves the issue. ```dockerfile FROM vllm/vllm-openai:latest RUN /usr/bin/python3 -m pip install --no-cache-dir 'transformers==5.14.1' ``` With Transformers 5.14.1: * the same model loads successfully, * the vLLM engine initializes successfully, * the API server starts successfully, * `/v1/chat/completions` returns HTTP 200, * 32 concurrent requests run successfully. ## Related PRs * #49797 ? Fix Gemma 4 for upcoming Transformers version * #49959 ? Handle Transformers 5.15 Gemma4 schema ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51744", "createdAt": "2026-08-11T00:42:58Z", "updatedAt": "2026-08-13T08:15:51Z", "timestamp": "2026-08-13T08:15:51Z", "metrics": { "reactions": 5, "comments": 14 }, "labels": [ "bug" ], "author": "yasu-oh", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b4d04dbbabe79cb44a26", "signalId": "github:vllm-project/vllm:pull_request:51251", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51251", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Configure custom encoder cache managers from VllmConfig", "text": "## Purpose Expose custom encoder cache manager configuration through `VllmConfig` for both online and offline inference. Custom encoder cache managers may require policy-specific parameters in addition to the encoder cache size. This PR provides a generic configuration path while preserving compatibility with existing built-in and constructor-only cache managers. ## Changes This PR: - Adds an opaque `manager_config` field to `EncoderCacheManagerConfig`. Its contents are interpreted by the selected custom cache manager. - Exposes `ec_manager_config` through `EngineArgs`. - Adds the `--ec-manager-config` CLI option for online serving. - Supports the same configuration through `LLM(ec_manager_config=...)` for offline inference. - Resolves custom encoder cache managers using fully qualified class names. - Allows custom managers that need the complete `VllmConfig` to optionally implement: ```python @classmethod def from_vllm_config( cls, *, cache_size: int, vllm_config: VllmConfig, ): ... ``` - Preserves compatibility with constructor-only custom managers: ```python manager_cls(cache_size=cache_size) ``` - Keeps the existing construction paths of built-in `EncoderCacheManager` and `EncoderDecoderCacheManager` unchanged. A custom manager is not required to inherit from `EncoderCacheManager` or implement `from_vllm_config`, as long as it satisfies the interfaces consumed by the scheduler and model runner. ## Relationship to #48218 This PR is a follow-up to #48218, which was merged as commit `833483f`. #48218 introduced the generic encoder cache manager extension foundation, including: - selecting a custom encoder cache manager by fully qualified class name; - passing `EncoderCacheManagerMetadata` from the scheduler to workers; - overridable model-runner hooks for encoder cache lookup, storage, eviction, and request cleanup. However, custom managers were still constructed using only `cache_size`, and the encoder cache manager configuration was not consistently exposed through online and offline `EngineArgs`. This PR completes the configuration path without changing the lifecycle hooks introduced by #48218. Hardware-specific cache policies remain implemented in their corresponding platform plugins. ## Usage ### Online inference ```bash vllm serve MODEL \\ --ec-manager-config '{ \"encoder_cache_manager_cls\": \"vllm_ascend.ec_manager.score_ec_manager.ScoreEncoderCacheManager\", \"manager_config\": { \"cpu_cache_slots\": 100000, \"max_clock\": 15, \"clock_decay_every\": 64, \"watermark\": 0.2, \"promote_percentile\": 0.2 } }' ``` ### Offline inference ```python from vllm import LLM llm = LLM( model=\"MODEL\", ec_manager_config={ \"encoder_cache_manager_cls\": ( \"vllm_ascend.ec_manager.score_ec_manager.\" \"ScoreEncoderCacheManager\" ), \"manager_config\": { \"cpu_cache_slots\": 100000, \"max_clock\": 15, \"clock_decay_every\": 64, \"watermark\": 0.2, \"promote_percentile\": 0.2, }, }, ) ``` `manager_config` is intentionally opaque to vLLM. Validation and interpretation of manager-specific fields are owned by the selected custom manager. Specifying a non-empty `manager_config` without `encoder_cache_manager_cls` raises a configuration error.", "url": "https://github.com/vllm-project/vllm/pull/51251", "createdAt": "2026-08-06T10:36:50Z", "updatedAt": "2026-08-13T08:08:01Z", "timestamp": "2026-08-13T08:08:01Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "ready" ], "author": "hotTea123", "state": "closed", "assignees": [ "Isotr0py" ], "change": "updated" } }, { "id": "event:0d3b62a1ee658790372e", "signalId": "github:vllm-project/vllm:pull_request:46370", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46370", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Initial support for fault tolerant ep using scale-down", "text": "## Purpose This PR adds **scale-down** and **auto-recovery** support to the vLLM fault tolerance (FT) framework for DP+EP MoE deployments (external LB mode). The existing FT framework (`--enable-fault-tolerance`) supports `retry` — reinitializing the DP process group after a transient fault where all ranks survive. When a DP rank permanently dies (hardware failure, OOM kill, etc.), `retry` alone is insufficient: the dead rank's expert slots become unreachable and EPLB placement tables reference non-existent slots. This PR introduces two complementary capabilities: 1. **Scale-down** (`POST /fault_tolerance/apply` with `instruction: \"scale_down\"`): Removes dead DP ranks, masks the corresponding EP ranks at the all2all kernel level, redistributes orphaned logical experts onto surviving ranks using EPLB redundant slots, selectively reloads only reassigned expert weights from disk, rebuilds the DP Gloo group over the surviving slots, and resumes inference — all without a full restart. 2. **Auto-recovery** (`--fault-tolerance-config='{\"auto_recovery\": true}'`): Lets the engine self-heal without external orchestrator commands. On fault, the engine queries the all2all mask from its workers and unions it cluster-wide via the DP store: if no new dead ranks are detected (transient fault), it auto-retries; otherwise it derives the dead DP ranks from the EP mask and auto-scales-down. ### Design - **Slots model: EP topology unchanged** From vLLM's point of view, a removed rank remains as an empty slot: `parallel_config.data_parallel_size` / `data_parallel_rank` stay frozen at their initial values and DP rank ids never change, so there is exactly one coordinate system (original) and no densification anywhere. `removed_dp_ranks` is always in original coordinates and is cumulative-safe — it may include ranks removed by earlier rounds; the engine applies only the increment. Likewise the all2all kernel keeps the original EP world size with dead EP ranks masked via `update_mask()` (both `deepep_low_latency` and `nixl_ep`), which avoids model re-init, graph recompile, and CUDA graph re-capture. Only the rebuilt Gloo `cpu_group` uses dense internal ranks over `sorted(alive)`, computed at reinit time and never written back to config. - **EPLB is disabled after scale-down and re-enabled by a future scale-up.** Post-scale-down the EP/EPLB groups still span the original EP world size, so dynamic rebalancing would attempt collectives with dead ranks; EPLB async rebalancing is therefore suppressed (`eep_eplb_suppressed`) until a scale-up restores the slots. - **Failed recovery is retryable.** The engine commits the topology change (`data_parallel_master_ip`, cumulative dead set) only after the Gloo group reinit succeeds, so a failed attempt leaves a consistent state that can be re-dispatched. Invalid commands (own-rank removal, fully stale removal, <1 survivor) are rejected up front with an observable `ft_error` — the orchestrator re-reads status and re-issues. - **The orchestrator can pass an explicit `request_id` to identify the recovery round.** All engines in one round share the same `request_id`, which namespaces that round's coordination keys in the DP store so concurrent or stale rounds cannot cross-talk. When empty, it falls back to the engine's local reinit epoch. - **Auto-recovery rejects external commands while it is in progress.** A `_recovery_lock` serializes auto-recovery (busy-loop thread) against external commands (input-sockets thread); a command arriving during an ongoing recovery is rejected with a recorded `ft_error`, keeping a single recovery driver per engine at any time. ### Implementation Notes - **TP sibling lockstep barrier (TP > 1).** `_synchronize_dp_ranks` (v1) and `sync_cudagraph_and_dp_padding` (v2) run a per-step Gloo barrier over the TP CPU group when FT is enabled and `tp_size > 1`, preventing intra-TP deadlock: a sibling whose peer stopped arriving fails on the host within `cpu_distributed_timeout_seconds`, never leaving an orphaned TP collective (e.g. fused allreduce kernels without timeout) running on its device stream. The TP CPU group is rebuilt during worker `retry`. - **nixl_ep dispatcher sync.** After masking, each MoE layer's cached `num_dispatchers` is rewritten to the nixl_ep kernel's `active_rank_bound` (highest surviving EP rank + 1); the kernel sizes combine output by `active_rank_bound` and asserts the width matches. DeepEP-LL keeps a fixed num_ranks-wide layout and needs no sync. - **Dead-column neutralization.** Under the slots model the DP allreduce tensors keep their original width and a dead rank's column stays 0 after the SUM allreduce, which is not neutral for min/`all(==1)` aggregations. After the allreduce, each rank locally fills dead columns (from `get_dp_group().dead_dp_ranks`) with aggregate-neutral values. - **EPLB async state reset.** A new `ft_reset_epoch` on the EPLB state lets the async worker detect a recovery and abandon in-flight transfers instead of applying stale results; the async thread survives exceptions when FT is enabled. - **Side-stream mask query.** Worker `query_mask` runs on a separate CUDA stream so a poisoned main stream cannot block the mask read that auto-recovery and status reporting depend on. ### Known limitations - `scale_down` requires EPLB with `num_redundant_experts > 0`. - Requires an FT-capable all2all backend (`deepep_low_latency` or `nixl_ep`) and external LB mode with one engine per API server. - EPLB dynamic rebalancing stays suppressed after scale-down (lifted by a future `scale_up`). - Recovery is not guaranteed for faults occurring at certain stages (e.g. mid weight-transfer in EPLB). - Auto-recovery cannot self-heal the death of DP rank 0 (the coordination TCPStore master is hosted by rank 0); the engine fails closed and waits for an external `scale_down` command with `dp_store_port`/`dp_master_ip`. ## Test Plan E2E tests live in `tests/v1/fault_tolerance/test_fault_tolerance_e2e.py` (gated on `has_nixl_ep`): 1. `test_injected_fault_retry_recovers_all_ranks` (DP=2) — injected fault, all ranks UNHEALTHY, `retry`, all ranks HEALTHY and serving. 2. `test_scale_down_removes_dead_rank_and_recovers` (DP=4, EPLB with 32 redundant experts) — SIGKILL rank 1's worker; survivors go UNHEALTHY, victim's engine goes DEAD and rejects `retry` with a recorded `ft_error`; `scale_down(removed_dp_ranks=[1])` to all survivors; all survivors return to HEALTHY and serve completions. Manual cluster validation (external LB, 4 independent API servers, one engine each): - Model: DeepSeek-V2-Lite-Chat (MoE, 64 logical experts), `--data-parallel-size 4 --tensor-parallel-size 2` (EP world size 8), `--enable-expert-parallel`, `--all2all-backend nixl_ep`, `--enable-eplb` with 64 redundant experts (async), `--enable-fault-tolerance`, `--cpu-distributed-timeout-seconds 30`, `engine_recovery_timeout_sec=500`. - Drive baseline inference on all ranks; kill one worker of DP rank 1 (TP1 sibling) during active inference; verify survivors detect the fault and go UNHEALTHY while the victim's engine goes DEAD; send `scale_down(removed_dp_ranks=[1])` to the 3 survivors; verify expert redistribution + weight reload, Gloo group rebuilt over the surviving slots (original rank coordinates unchanged), all survivors HEALTHY and serving. ## Test Results Manual scale-down validation (2026-08-12, config above): **PASSED**. This run exercises the TP=2 single-worker-kill scenario end to end. Sanitized full log (host IP / GPU model / local paths redacted) attached. | Phase | Observation | Timestamp | |---|---|---| | Cluster ready | 4 ranks (DP=4 × TP=2, EP world 8) initialized, serving baseline traffic | 17:49:24 – 17:50:48 | | Fault injected | One worker of DP rank 1 (TP1 sibling, EP rank 3) killed during active inference | 17:50:48 | | First survivor-side error | Survivors' TP1 workers hit gloo `Connection reset by peer` on the per-TP-rank DP group | 17:50:48 | | Victim DEAD | Engine 1's executor shut down its remaining worker; busy loop cancelled → status DEAD; its API server stays up and keeps answering `/fault_tolerance/status` | 17:50:57 | | Survivors UNHEALTHY | Engines 0/2/3 raise from the nixl_ep kernel timeout with mask `[0,0,1,1,0,0,0,0]` (EP ranks 2,3 = DP rank 1, both TP workers) | 17:51:18 | | scale_down dispatched | `POST /fault_tolerance/apply` `removed_dp_ranks=[1]` → 202 on R0/R2/R3 | ~17:51:44 | | Expert redistribution | 257 (layer, expert) pairs reloaded from disk per worker (652–733 tensors); `num_dispatchers` synced to `active_rank_bound=8` | 17:51:45 | | Scale-down complete | Gloo DP group rebuilt 4→3 (dense group ranks R0→0, R2→1, R3→2; original `data_parallel_rank` frozen); Engines 0/2/3 → HEALTHY, `cumulative dead_dp_ranks=[1]` | 17:51:45 | | Post-recovery inference | All 3 survivors resume serving `/v1/chat/completions` with HTTP 200 | 17:51:45 – 17:52:07 | Key observations: - Recovery itself (mask replay + redistribution + weight reload + group reinit) completes in ~1s once dispatched; end-to-end latency is dominated by fault detection (immediate for the TP1 siblings via gloo connection-reset; ~30s for the surviving engines via the nixl_ep kernel timeout). - No hangs, no orphaned TP collectives — the TP sibling lockstep barrier keeps survivor device streams clean. E2E pytest: ``` .venv/bin/python -m pytest tests/v1/fault_tolerance/test_fault_tolerance_e2e.py -v -s ``` - **AI assistance**: this change was developed with AI assistance (kimi-k3); every changed line was reviewed by the human submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update.
", "url": "https://github.com/vllm-project/vllm/pull/46370", "createdAt": "2026-06-22T12:56:18Z", "updatedAt": "2026-08-13T08:06:06Z", "timestamp": "2026-08-13T08:06:06Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "fangyuchu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d212a3c930c284243642", "signalId": "github:vllm-project/vllm:issue:52071", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52071", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: speculative decoding under pipeline parallelism produces wrong output with --no-async-scheduling", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Reproduced at `pipeline_parallel_size` 2, 4 and 8, with two different speculative methods and two model families. Code references are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug With `--no-async-scheduling`, speculative decoding under pipeline parallelism produces wrong output — or a CUDA fault — instead of being rejected or working. Turning async scheduling back on fixes it. ### Root cause `next_decode_eligible_step` is assigned in exactly one place in the tree: ``` vllm/v1/core/sched/async_scheduler.py:49 request.next_decode_eligible_step = self.current_step + self.pp_size ``` initialised in one place: ``` vllm/v1/request.py:167 self.next_decode_eligible_step = 0 ``` and read in exactly one place: ``` vllm/v1/core/sched/scheduler.py:509 if self.current_step < request.next_decode_eligible_step: # V2+PP+async: enforce `pp_size` steps between same-req decodes # to match worker-side sampled-tokens broadcast slot ring cadence. ``` The base `Scheduler` — which is what `--no-async-scheduling` selects — reads that field and never sets it. It stays `0`, the guard never fires, and nothing keeps a request's decodes `pp_size` steps apart. The worker-side sampled-token broadcast ring is then read out of phase. ### What that produces, measured A probe at the point where a step's input ids are built (`combine_sampled_and_draft_tokens`), PP=2: ``` last_sampled=[16925] drafts=[16925, 16925, 16925] healthy last_sampled=[0] drafts=[0, 0, 0] ~25% of spec steps ``` Token id 0 is embedded as the anchor. On the model under test token 0 is `!`, and the generated text fills with `!`. The same skew has a second face — the step gets scheduled without the accepted token that its drafts hang off: ``` healthy: hidden=(4,1024) logits_indices=[0, 1, 2, 3] qlen=4 num_logits=4 ndraft=3 failing: hidden=(3,1024) logits_indices=[-1, 0, 1, 2] qlen=3 num_logits=4 ndraft=3 ``` `logits_start = query_end - num_logits = 3 - 4 = -1`, so `hidden_states[input_batch.logits_indices]` either wraps to the last row silently (torch semantics) or trips: ``` Assertion `ind >=0 && ind < ind_dim_size && \"vectorized gather kernel index out of bounds\"` failed ``` The same skew also reaches `_prepare_prefill_inputs_kernel` in `v1/worker/gpu/spec_decode/autoregressive/speculator.py`, where `query_len -= num_rejected` can reach 0 and the unmasked `tl.store(draft_input_ids_ptr + query_start - 1, ...)` writes out of bounds — Triton adds negative indices to the base pointer rather than wrapping. ### Reproduced with - `method: mtp` on GLM-4.5-Air (AWQ), PP=4 - `method: dspark` on Kimi-K3, PP=8 - PP=2 on a 16-layer synthetic target ### Controls Two controls that rule out the obvious alternatives, same tree, same session: ``` speculation OFF, async on vs async off : outputs IDENTICAL Qwen3-0.6B, PP=1 vs PP=2, no speculation : outputs IDENTICAL ``` So the pipeline path is greedy-stable on its own, and the scheduler mode alone changes nothing. The defect appears only when speculation and PP are combined with async scheduling disabled. Also worth noting: with async scheduling ON, plain decode throughput on the same config was ~1.9× better than with it off (94.6 → 48.8 ms/token TPOT, GLM-4.5-Air PP=4, concurrency 1), so the flag is expensive in its own right. ### Suggested fix Any of these beats the current behaviour, in descending order of preference: 1. Set the cadence in the base scheduler too, so the guard works regardless of scheduling mode. 2. Reject `speculative_config` + `pipeline_parallel_size > 1` + `--no-async-scheduling` at config time, with a message naming the cadence. 3. At minimum, document that speculative decoding under PP requires async scheduling. Silently emitting `!` is the worst of the available outcomes, and it is what happens today. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52071", "createdAt": "2026-08-12T23:52:42Z", "updatedAt": "2026-08-13T08:04:24Z", "timestamp": "2026-08-13T08:04:24Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "kimi" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0c6b2dc33fdb893b1fcb", "signalId": "github:vllm-project/vllm:pull_request:49652", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49652", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Fix autoregressive draft decode capture with dynamic SD", "text": "## Purpose Fixes #48494. Dynamic speculative decoding (DSD) derives the decode query lengths for each scheduled speculative-token count from the manager's configured `decode_query_len`: ```text num_new_sampled_tokens = decode_query_len - max_speculative_tokens runtime_query_len = scheduled_speculative_tokens + num_new_sampled_tokens ``` This derivation is valid when `decode_query_len` includes the maximum number of draft tokens, as it does for target verification and autoregressive draft prefill. It is not valid for autoregressive draft decode: every draft step processes exactly one token per request and dispatches the CUDA graph with `num_tokens == num_reqs` and `uniform_token_count == 1`. For example, with `max_speculative_tokens=3`, applying the DSD derivation to the draft decode manager's `decode_query_len=1` produces query lengths `1`, `-1`, and `-2` for scheduled K values `3`, `1`, and `0`. The non-positive descriptors eventually reach `InputBatch.make_dummy` and fail: ```text assert 0 < num_reqs <= num_tokens ``` This PR: - adds `use_dynamic_decode_query_len` to `CudaGraphManager`, defaulting to `True` to preserve existing behavior; - disables dynamic query-length derivation only for the autoregressive draft decode manager, whose query length is fixed at one; - keeps dynamic query-length derivation enabled for target verification, autoregressive draft prefill, DFlash, and all other existing callers; and - adds a production-wiring regression test for a representative `K=3/1/0` DSD schedule. This keeps full CUDA graph execution enabled. It does not clamp invalid shapes, swallow the assertion, or fall back to eager execution. Duplicate-work checks found no open PR referencing #48494 and no existing autoregressive Dynamic-SD CUDA Graph fix. Related PR #47737 skips non-positive query lengths for DSpark's dynamic draft manager when a K=0 tier disables drafting. That is complementary: the autoregressive draft decode manager has a fixed query length of one for every draft step and must not derive its shape from the DSD schedule at all. In particular, filtering non-positive derived lengths can leave it with no FULL decode graph when the runtime schedule does not contain the configured upper-bound K. PR #47131 implements D-cut verification-depth pruning and is also unrelated to this capture-shape bug. ## Test Plan ### Unit and static checks ```bash .venv/bin/python -m pytest -q \\ tests/v1/spec_decode/test_dynamic_sd_cug.py \\ tests/v1/cudagraph/test_cudagraph_manager.py .venv/bin/ruff check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py .venv/bin/ruff format --check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py git --no-pager diff --check ``` The regression test covers: - a valid K=3/1/0 runtime schedule below the configured upper-bound K=4; - dynamic draft-prefill query lengths; - fixed draft-decode query length `1`; and - full-CUDA-graph dispatch for every batch size from 1 through 8. ### GPU red/green reproduction Environment: - 2x NVIDIA A800, tensor parallel size 2; - V2 model runner; - Qwen3.5 architecture with native `Qwen3_5MTP`; - GDN attention path; - full CUDA graphs; and - DSD schedule `K=3/1/0`. ### Real-weight end-to-end smoke test The patched branch was also tested with the public `Qwen/Qwen3.5-0.8B-Base` checkpoint, including its native MTP weights: ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-0.8B-Base \\ --tensor-parallel-size 2 \\ --max-model-len 1024 \\ --max-num-seqs 8 \\ --gpu-memory-utilization 0.50 \\ --dtype bfloat16 \\ --language-model-only \\ --speculative-config '{ \"method\": \"mtp\", \"model\": \"Qwen/Qwen3.5-0.8B-Base\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [ [1, 2, 3], [3, 4, 1], [5, 8, 0] ], \"draft_tensor_parallel_size\": 2 }' \\ --compilation-config '{ \"mode\": 3, \"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [8] }' ``` The same checkpoint and request set were then run with `--enforce-eager` as a numerical reference. ## Test Result ### Unit and static checks ```text Base: vllm-project/vllm@1479bd9e9 Environment: Python 3.11.2, torch 2.11.0+cu130, 2x NVIDIA A800-SXM4-40GB pytest: 6 passed ruff check: All checks passed ruff format --check: 3 files already formatted pre-commit: 10 applicable Python hooks passed git diff --check: passed ``` The pre-commit run covered Ruff check/format, typos, mypy 3.10, SPDX headers, root lazy imports, filenames, forbidden imports, new `torch.cuda` API usage, and boolean context-manager checks. ### GPU red/green result Before this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/3 failure: InputBatch.make_dummy -> AssertionError ``` After this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/1 server: Application startup complete ``` ### Real-weight result - 9/9 requests succeeded across concurrency 1, 3, and 5, exercising scheduled K values 3, 1, and 0. - Speculative-decoding metrics confirmed that the MTP path generated and accepted draft tokens. - For the patch-active K=3 and K=1 tiers, all four full-CUDA-graph results matched eager results field-for-field. Repeating the eager run produced the same four results. - K=0 does not execute autoregressive draft decode and was therefore excluded from the patch-path numerical comparison. - Neither the full-CUDA-graph nor eager log contained `AssertionError`, `InputBatch.make_dummy`, or an `ERROR` record. The end-to-end smoke test used Qwen3.5 native MTP rather than the Gemma4 checkpoint from #48494. Both use the shared `AutoRegressiveSpeculator` draft decode manager fixed by this PR. This test validates the control flow and small-model numerical behavior; it does not make a large-model performance claim. ## AI assistance OpenAI Codex assisted with root-cause analysis, code drafting, and test drafting. I reviewed the changes and validation results, and this PR is ready for maintainer review.", "url": "https://github.com/vllm-project/vllm/pull/49652", "createdAt": "2026-07-24T02:00:33Z", "updatedAt": "2026-08-13T08:02:15Z", "timestamp": "2026-08-13T08:02:15Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "bug", "speculative-decoding", "v1", "nvidia", "mrv2" ], "author": "CZT0", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a87b086b5aa42f9ef84a", "signalId": "github:vllm-project/vllm:pull_request:52024", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52024", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Perf][ROCm] Dual-stream decode with hipgraphs\"", "text": "Reverts vllm-project/vllm#48223", "url": "https://github.com/vllm-project/vllm/pull/52024", "createdAt": "2026-08-12T17:05:00Z", "updatedAt": "2026-08-13T08:01:42Z", "timestamp": "2026-08-13T08:01:42Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "rocm", "ready" ], "author": "simondanielsson", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:eb7fd1886f3aed81e677", "signalId": "github:vllm-project/vllm:pull_request:50594", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50594", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix MiniMax M3 prompt reasoning initialization", "text": "## Purpose Fixes #46042. Fixes #50549. MiniMax-M3 adaptive reasoning can leak a leading `` into streaming `delta.content`. The existing initialization scans the entire rendered prompt with `is_reasoning_end()`. Because the prompt can contain paired `` markers as instructions or examples, the parser may incorrectly conclude that reasoning has already ended before generation starts. This change: - adds a prompt-boundary-specific reasoning-state hook while preserving the existing behavior by default - initializes MiniMax-M3 from `thinking_mode`: `enabled` starts in reasoning, `disabled` starts after reasoning, and `adaptive` lets generated markers determine the state - uses the hook in Chat Completions, the Responses API, streaming parsing, and structured-output initialization - preserves the deferred `adaptive=None` state through the structured-output gate instead of falling back to a full-prompt scan - scopes generated-marker checks to output tokens so instructional prompt markers cannot activate the grammar early - constrains unresolved adaptive sampling to grammar-valid tokens plus every decoded reasoning-marker tokenization - replays adaptive direct-content tokens through the grammar, including ambiguous prefixes and speculative draft state - respects the final reasoning boundary when continuing an existing assistant message, including explicit open continuations under disabled mode, and scopes continuation marker scans to that final message only - preserves open and closed `thinking` content-part state during continuation - propagates incomplete Responses reasoning items to frontend and engine parsers with the same request-local state - clears continuation-only parser state and refreshes the frontend parser after Responses built-in tool turns rerender the prompt - forwards request-local reasoning mode and prompt state through offline `LLM.chat()` and `enqueue_chat()` into the synchronous engine - keeps batch reasoning parsers request-local so continuation state cannot leak between conversations - initializes each engine-side structured-output parser from its prompt exactly once - adds focused regressions for instructional markers, adaptive marker boundaries, structured output, and all MiniMax-M3 thinking modes ## Related work and scope - #45718 fixed split-token MiniMax-M3 marker parsing, but not prompt-boundary state initialization. - #50152 identified the same full-prompt scan failure and demonstrated the production symptom, but closed without merging. - #48550 also addresses prompt-mode initialization while adding broader speculative-decoding and tool-parser changes. - This PR limits the fix to prompt-boundary reasoning initialization across online and offline chat paths and Responses, plus the structured-output state transitions needed to preserve it. ## Test Plan - Run focused parser and structured-output regressions: `uv run pytest -q tests/reasoning/test_minimax_m3_reasoning_parser.py tests/v1/structured_output/test_reasoning_structured_output.py` - Run offline chat prompt-state regression: `uv run pytest -q tests/entrypoints/llm/test_offline_reasoning_state.py` - Run batch prompt-state regressions: `uv run pytest -q tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py -k batch_forwards_prompt_reasoning_state` - Run Responses continuation regressions: `uv run pytest -q tests/entrypoints/openai/responses/test_serving_responses.py` - Run static validation on the changed Python files. - Serve `MiniMaxAI/MiniMax-M3` with the patch and send adaptive and enabled streaming chat-completion requests, including a structured response format. - Confirm that reasoning markers never appear in `delta.content`, reasoning remains in the reasoning field, and the output grammar starts only after generated reasoning ends or advances the first direct adaptive content token. ## Test Result Passed: - MiniMax-M3 reasoning parser: 44 tests passed - reasoning-aware structured output: 21 tests passed - offline chat prompt-state forwarding: 1 test passed - batch prompt-state forwarding: 3 tests passed - Responses serving: 23 tests passed, 1 expected xfail - `ruff check`: all checks passed - `ruff format`: all changed files formatted - Python `compileall` - `git diff --check` The 92 focused tests were run in a CPU-only temporary environment with `VLLM_TARGET_DEVICE=empty`; the repository-wide GPU CI has not run yet. Live MiniMax-M3 evaluation is pending because this workspace has no GPU/model deployment. It should be completed before merge. ## AI assistance OpenAI Codex assisted with implementation and regression-test drafting. The commits include `Assisted-by` trailers. The submitter requested the changes after reviewing the proposed scope.", "url": "https://github.com/vllm-project/vllm/pull/50594", "createdAt": "2026-07-31T16:34:24Z", "updatedAt": "2026-08-13T08:00:12Z", "timestamp": "2026-08-13T08:00:12Z", "metrics": { "reactions": 2, "comments": 21 }, "labels": [ "bug", "structured-output", "frontend", "tool-calling" ], "author": "jinbagi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:da75744f324c2ac696a9", "signalId": "github:vllm-project/vllm:pull_request:52120", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52120", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Fix ReDoS in validation error path sanitizer", "text": "## Summary - Fix CWE-1333 regex denial-of-service in `sanitize_message()` where a 100KB slash-delimited input (`\"/a\" * 50000`) could keep an API worker busy for 3+ seconds per rejected request. - Add a dot-character pre-check to skip the generic path regex entirely when no filename extension is possible, and use an atomic group `(?>...)` to prevent backtracking within repeated slash-segments for messages that do contain dots. - Addresses GHSA-f2g9-pmwr-xwc7. ## Test plan - [x] `pytest tests/entrypoints/serve/utils/test_api_utils.py -v` — 22 tests pass (4 new ReDoS regression tests) - [x] `pytest tests/entrypoints/serve/utils/test_error_sanitization.py -v` — 9 tests pass - [x] `pre-commit run --files ` — all hooks pass - [x] Reproduced advisory PoC shape (`\"/a\" * 50000`): completes in <0.1ms after fix (was 3+ second hang) - [x] Verified legitimate path redaction still works (`/app/server.py`, `/usr/lib/...` etc.) Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/52120", "createdAt": "2026-08-13T07:43:58Z", "updatedAt": "2026-08-13T07:59:25Z", "timestamp": "2026-08-13T07:59:25Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "frontend", "ready" ], "author": "jperezdealgaba", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ec5b7cc3e2c399581a8b", "signalId": "github:vllm-project/vllm:pull_request:45581", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45581", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: drop trailing MTP entry from text_config.layer_types", "text": "## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM v0.23.0. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stack. On `transformers` v5, the inner text-config is instantiated as `Exaone4Config` — the legacy `exaone4_5_text` model type is remapped to `exaone4` by the upstream `Exaone4_5_Config.__post_init__` — and `Exaone4Config` has no notion of the MTP layer. Its `@strict` dataclass init runs `validate_layer_type` and fails before vLLM ever gets to the model loader: ``` ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` Full traceback from `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":3}'` on `vllm==0.23.0` / `transformers==5.12.0`: ``` File \".../transformers/models/exaone4_5/configuration_exaone4_5.py\", line 84, in __post_init__ self.text_config = CONFIG_MAPPING[self.text_config[\\\"model_type\\\"]](**self.text_config) File \".../huggingface_hub/dataclasses.py\", line 276, in init_with_validate cls.validate(self) File \".../huggingface_hub/dataclasses.py\", line 253, in validate raise StrictDataclassClassValidationError(...) from e huggingface_hub.errors.StrictDataclassClassValidationError: Class validation error for validator 'validate_layer_type': ValueError: `num_hidden_layers` (64) must be equal to the number of `layer_types` (65) ``` vLLM v0.23.0's pin `transformers >= 4.56.0, != 5.3.*, ...` rules out the LG-AI `nuxlear` transformers fork pinned at `5.3.0.dev0` (which predates the strict validator and is what the LG-AI repo's deployment recipe still suggests). So the only path forward on mainline is to fix the config on the vLLM side. ## Fix Register a vLLM-side `Exaone4_5Config` (`vllm.transformers_utils.configs.exaone4_5`) that subclasses the upstream config and trims the trailing `layer_types` entry before the inner `Exaone4Config` is built. The trim is gated on `len(layer_types) == num_hidden_layers + 1` exactly; any other shape is left untouched so unrelated config bugs are not silently masked. The MTP layer is consumed separately by the speculative-decode path via `num_nextn_predict_layers` / `_num_mtp_layers` (and the MTP weights themselves come from the `mtp.*` checkpoint keys), so dropping the metadata entry in `layer_types` does not change MTP behavior — it only unblocks the inner `Exaone4Config` validation. 3 files, 57 lines added: - `vllm/transformers_utils/configs/exaone4_5.py` (new, 54 lines) - `vllm/transformers_utils/configs/__init__.py` (+2 — register in `_CLASS_TO_MODULE` and `__all__`) - `vllm/transformers_utils/config.py` (+1 — map `exaone4_5` → `Exaone4_5Config` in `_CONFIG_REGISTRY`) ## Test Manually verified end-to-end on the following stack: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before the patch: `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails at engine init with the validator error above. After the patch the same command starts cleanly and the speculative-decode path is active: ``` INFO [model.py:611] Resolved architecture: Exaone4_5_MTP INFO [llm_base_proposer.py:1347] Detected MTP model. Sharing target model embedding weights with the draft model. INFO [llm_base_proposer.py:1403] Detected MTP model. Sharing target model lm_head weights with the draft model. ``` `vllm bench serve --dataset-name sonnet --dataset-path benchmarks/sonnet.txt --sonnet-input-len 550 --sonnet-output-len 256 --sonnet-prefix-len 200 --num-prompts 64 --max-concurrency 16 --request-rate inf` against `LGAI-EXAONE/EXAONE-4.5-33B-FP8 --enforce-eager --speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":3}'`: ``` ============ Serving Benchmark Result ============ Successful requests: 64 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 24.06 Total input tokens: 34915 Total generated tokens: 16384 Request throughput (req/s): 2.66 Output token throughput (tok/s): 680.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.51 Median TPOT (ms): 20.66 P99 TPOT (ms): 24.84 ---------------Speculative Decoding--------------- Acceptance rate (%): 35.91 Acceptance length: 2.08 Drafts: 7877 Draft tokens: 23631 Accepted tokens: 8486 Per-position acceptance (%): Position 0: 59.64 Position 1: 31.64 Position 2: 16.45 ================================================== ``` The per-position monotonic decay (≈60% / 32% / 16%) matches what `[speculative.py:722]` warns about for `num_speculative_tokens > 1` and is consistent with the published MTP behavior for this checkpoint family on real text. (A control run with `--dataset-name random` reports near-zero acceptance, as expected — random tokens are out-of-distribution for a MTP head trained on text.) ## Notes - This PR only touches the configuration shim path. The vLLM `exaone4_5` model module and the `Exaone4_5_MTP` speculator are not modified. - Separately, on `vllm/model_executor/models/exaone4_5.py:243`, `Exaone4_5_VisionBlock.forward()` still uses `seqlens=` while the Qwen2.5-VL caller it inherits from passes `sequence_lengths=` since the upstream rename in `qwen2_5_vl.py`. That is an unrelated bug and out of scope for this PR; happy to send it as a follow-up if helpful.", "url": "https://github.com/vllm-project/vllm/pull/45581", "createdAt": "2026-06-14T12:48:26Z", "updatedAt": "2026-08-13T07:57:36Z", "timestamp": "2026-08-13T07:57:36Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build", "v1", "cpu" ], "author": "seongyun1104", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:238de264b97bd680d323", "signalId": "github:vllm-project/vllm:pull_request:45583", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45583", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] EXAONE 4.5: rename Exaone4_5_VisionBlock.forward kwarg seqlens -> sequence_lengths", "text": "## Purpose `EXAONE4_5_VisionTransformer` inherits from `Qwen2_5_VisionTransformer`, and the parent class's vision-block loop (`vllm/model_executor/models/qwen2_5_vl.py:1100`) now invokes each block with `sequence_lengths=` rather than the older `seqlens=`: ```python hidden_states = blk( hidden_states, cu_seqlens=cu_seqlens_now, rotary_pos_emb_cos=rotary_pos_emb_cos, rotary_pos_emb_sin=rotary_pos_emb_sin, max_seqlen=max_seqlen_now, sequence_lengths=sequence_lengths_now, ) ``` `Exaone4_5_VisionBlock.forward()` (`vllm/model_executor/models/exaone4_5.py:243`) still declares the kwarg as `seqlens`, so loading any `Gemma4ForConditionalGeneration`-style EXAONE 4.5 checkpoint through the multimodal path raises during multi-modal warmup before the engine is ready to serve: ``` TypeError: Exaone4_5_VisionBlock.forward() got an unexpected keyword argument 'sequence_lengths' File \".../vllm/model_executor/models/qwen2_5_vl.py\", line 1100, in forward hidden_states = blk( ^^^^ ``` ## Fix Rename the EXAONE block's xFormers-only kwarg from `seqlens` to `sequence_lengths` so it matches the Qwen2.5-VL caller's signature. The argument is unused inside the block body — only `cu_seqlens`, `max_seqlen`, and the rotary pos embeddings are forwarded into `self.attn` — so this is purely a keyword-argument name fix with no behavior change. 1 file, 1 line: \\`\\`\\` vllm/model_executor/models/exaone4_5.py | 2 +- \\`\\`\\` ## Test Verified end-to-end on the same stack used in #45581: | | | |---|---| | vllm | 0.23.0 | | torch | 2.11.0+cu130 | | transformers | 5.12.0 | | flashinfer-python | 0.6.12 | | GPU | RTX PRO 6000 Blackwell Workstation Edition (SM12.0, 94 GB) | | Driver / CUDA | 590.48.01 / 13.1.115 | Before this patch: `vllm serve LGAI-EXAONE/EXAONE-4.5-33B-FP8 ...` fails during multi-modal warmup with the TypeError above. After this patch the multimodal warmup completes and the server reaches \\`Application startup complete\\`: \\`\\`\\` INFO ... [base.py:227] Multi-modal warmup completed in 3.886s INFO ... [base.py:227] Readonly multi-modal warmup completed in 0.125s INFO ... [api_server.py:583] Starting vLLM server on http://0.0.0.0:8005 \\`\\`\\` The reproducer / test stack and the bench-serve numbers are identical to the verification block in #45581 — once both patches land, EXAONE 4.5 + MTP loads and serves on mainline vLLM + mainline transformers with no fork dependency. ## Related - #45581: drops the trailing MTP entry from \\`text_config.layer_types\\` so the base config validation passes. This PR addresses the next failure that surfaces once #45581 unblocks engine init.", "url": "https://github.com/vllm-project/vllm/pull/45583", "createdAt": "2026-06-14T12:56:23Z", "updatedAt": "2026-08-13T07:57:33Z", "timestamp": "2026-08-13T07:57:33Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build", "v1", "cpu" ], "author": "seongyun1104", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f296756027e40c57f67c", "signalId": "github:vllm-project/vllm:issue:48495", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:48495", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] Drafter attention backend auto-selection picks FlashInfer on SM90 for a sliding-window drafter, hitting the guard crash", "text": "### Environment Same as #48494 (v0.25.0, single H100 NVL 94GB SM90, Gemma4 MTP drafter with 3 sliding-attention layers + 1 full-attention layer), **without** `--attention-backend`. ### Behavior - Target model backend selection: `Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN']` — fine (heterogeneous head dims exclude others). - Drafter selection: `Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN']` — then engine init crashes: ``` File \".../vllm/v1/attention/backends/flashinfer.py\", line 757, in __init__ raise NotImplementedError( NotImplementedError: FlashInfer backend on SM90 currently crashes with sliding-window attention layers. Use the default attention backend. ``` ### Expected The auto-selector should exclude FlashInfer for a draft model containing sliding-window layers on SM90, instead of selecting it and failing at `create_metadata_builders`. The constraint is already documented in-code — the guard at `flashinfer.py:757` cites `flashinfer-ai/flashinfer#3578`; the selector just doesn't consult it. This is not Gemma-specific: any drafter with sliding-window layers on SM90 will hit the same path. ### Workaround `--attention-backend TRITON_ATTN` (forces both target and drafter). Note for migrators: the `VLLM_ATTENTION_BACKEND` env var is removed in 0.25 (logged as *Unknown vLLM environment variable*) — the CLI flag is the only path now.", "url": "https://github.com/vllm-project/vllm/issues/48495", "createdAt": "2026-07-13T13:32:33Z", "updatedAt": "2026-08-13T07:57:31Z", "timestamp": "2026-08-13T07:57:31Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f487338bc6b1902bb73f", "signalId": "github:vllm-project/vllm:issue:48494", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:48494", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] num_speculative_tokens_per_batch_size + MTP speculator fails full CUDA graph decode capture (InputBatch.make_dummy assert)", "text": "### Environment - vLLM v0.25.0 (official docker image), single H100 NVL 94GB (SM90) - Target: `prithivMLmods/gemma-4-31B-it-qat-FP8` (compressed-tensors FP8) - Drafter: `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` (Gemma4 MTP, 4 layers: 3 sliding + 1 full attention) - `--kv-cache-dtype fp8 --async-scheduling --attention-backend TRITON_ATTN` (backend forced due to a separate SM90/SWA selection issue, filed separately) - `VLLM_USE_V2_MODEL_RUNNER=1` (per the new hint in the downgrade warning, to enable DSD × full CUDA graphs from #45953) ### Repro ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /models \\ --speculative-config '{\"model\":\"/drafter\",\"method\":\"mtp\",\"num_speculative_tokens\":3, \"num_speculative_tokens_per_batch_size\":[[1,64,3],[65,128,1],[129,512,0]]}' \\ --kv-cache-dtype fp8 --async-scheduling --attention-backend TRITON_ATTN ``` The env opt-in follows the hint added in 0.25's downgrade warning: *\"Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs.\"* Without it, DSD falls back to PIECEWISE and does not reach this capture path. Startup proceeds through weight load, torch.compile, target-model graph capture (`Capturing CUDA graphs (FULL): 150/150` OK), speculator prefill capture OK, then crashes at speculator **decode** FULL capture: ``` File \".../vllm/v1/worker/gpu/spec_decode/autoregressive/cudagraph_utils.py\", line 83, in create_forward_fn attn_state = prepare_inputs_to_capture( File \".../vllm/v1/worker/gpu/cudagraph_utils.py\", line 604, in prepare_inputs_to_capture input_batch = InputBatch.make_dummy(num_reqs, num_tokens, input_buffers) File \".../vllm/v1/worker/gpu/input_batch.py\", line 109, in make_dummy assert 0 < num_reqs <= num_tokens AssertionError ``` ### Isolation matrix (3 configurations, same stack otherwise) | speculative-config | Result | |---|---| | DSD table `[[1,64,3],[65,128,1],[129,512,0]]` | crash at ~1% of speculator decode capture | | DSD table `[[1,64,3],[65,512,1]]` (no K=0 tier) | crash immediately at speculator decode capture | | static `num_speculative_tokens: 3` (no table) | **capture completes** (`Graph capturing finished`) | → The **presence of the batch-size table** is the trigger; K=0 tiers are not required. ### Candidate root cause (hypothesis, file:line trail) `vllm/v1/worker/gpu/cudagraph_utils.py:205-225` reconstructs per-descriptor decode query lens as `num_new_sampled_tokens_per_step = decode_query_len − num_speculative_tokens`, an invariant written for the **target** manager (`decode_query_len = K + newly sampled`). The autoregressive speculator's capture manager (`spec_decode/autoregressive/cudagraph_utils.py:59-105`) inherits this base, but MTP drafting has different decode-query semantics (1 token/seq/step), so the derived `num_tokens` collapses to ≈K while `num_reqs` is the capture batch size — matching the observed failure boundary (K=3 table fails from bs=4, K=1 table from bs=2: `num_reqs > num_tokens`). EAGLE presumably passes because its execution shape satisfies the invariant. ### Notes - The DSD docs state the feature is \"only tested with Eagle and Eagle-3\"; this report adds the MTP data point. MTP + DSD works on v0.24.0 (runtime K switching across tiers verified with spec-decode counters), but v0.24.0 forces `cudagraph_mode=PIECEWISE` for DSD, so #45953's full-CG path is the reason to move to 0.25 — and this capture bug currently blocks that move for MTP. - Perf context (v0.24.0, informational, related to #47277): at client concurrency 256 we see DSD mode incur a 12–25% throughput penalty vs no-spec (oscillating run-to-run) with TTFT p50 inflating ~6×, **even with an all-K=0 table producing near-zero draft tokens**. We isolated it away from the cudagraph downgrade (forcing `cudagraph_mode=PIECEWISE` on a no-spec config costs ≈0% at the same concurrency) — the overhead appears to live in the DSD/spec-decode path itself at high batch, matching the \"per-step overhead erases the benefit\" pattern reported in #47277 on a different stack (Qwen3.5 native MTP / L20X). We can post the isolation trace on that thread once this capture blocker is resolved and we can re-run the comparison on 0.25. - Workaround today: static `num_speculative_tokens` (on 0.25 the full-CG capture passes; note startup then needs extra gpu-memory-utilization margin — we hit a warmup OOM at 0.95 on a 94GB H100), or DSD on 0.24 with the PIECEWISE cost.", "url": "https://github.com/vllm-project/vllm/issues/48494", "createdAt": "2026-07-13T13:32:16Z", "updatedAt": "2026-08-13T07:57:29Z", "timestamp": "2026-08-13T07:57:29Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a9b39190746b53bad757", "signalId": "github:vllm-project/vllm:pull_request:46064", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46064", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for _fwd_kernel_ep_scatter_1 and _fwd_kernel_ep_…", "text": "## Purpose Adds a unit test for the `_fwd_kernel_ep_scatter_1` and `_fwd_kernel_ep_scatter_2` Triton kernels (DeepGemm group) in `tests/kernels/moe/test_ep_scatter.py`. Both kernels run inside the `ep_scatter` wrapper and had no upstream coverage: - `_fwd_kernel_ep_scatter_1` computes per-expert region offsets and fills `m_indices` (expert label per output row, -1 for padding), counts aligned to BLOCK_E=128. - `_fwd_kernel_ep_scatter_2` copies each token's activation and per-group scale into the row of its (optionally remapped) expert, recording the destination in `output_index`. Phase 1 is checked by comparing `m_indices` against a reference region layout. Phase 2 is checked by reading the destination back out of the kernel: each token's `output_index` must point to a row labelled with its expert in `m_indices`, and that row must hold both the token's activation and its per-group scale. A non-trivial `expert_map` case is included. Device-agnostic (`current_platform.device_type`); runs on CUDA and XPU. ## Test Plan `python -m pytest tests/kernels/moe/test_ep_scatter.py -v` ## Test Result All 15 cases pass.", "url": "https://github.com/vllm-project/vllm/pull/46064", "createdAt": "2026-06-18T15:40:22Z", "updatedAt": "2026-08-13T07:57:27Z", "timestamp": "2026-08-13T07:57:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:461371a0ef48ead08187", "signalId": "github:vllm-project/vllm:issue:49986", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:49986", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Perf] DSD arms pay a large baseline tax vs no-spec under production defaults; PIECEWISE override identified as one factor", "text": "## Motivation While benchmarking `num_speculative_tokens_per_batch_size` (Dynamic SD) schedules on H100 NVL 94GB (`prithivMLmods/gemma-4-31B-it-qat-FP8` target + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` draft, vLLM `c5d967c23`) I found that **every arm using a `speculative_config` pays a substantial throughput tax vs no-spec under production defaults**, and the gap is large enough at short ctx that it dominates the aggregate spec-bench signal for schedules that lower K on the short-ctx majority. Position-balanced 2-trial (T1 order: C→N→S→A, T2 reverse: A→S→N→C), cache-wipe + cold-start-burn methodology, `prefix_repetition` c=256, 3 warmup + 3 measure per cell. | ctx | no-spec (K=0) | static K=3 | 3-item batch schedule (A′) | 2D 6-cell schedule (C′) | |---:|---:|---:|---:|---:| | 400 | **2711.7** | 2139.8 (−21%) | 1875.6 (**−31%**) | 1890.7 (−30%) | | 900 | 1987.1 | 1838.9 (−7%) | 1453.5 (−27%) | 1874.6 (−6%) | | 1900 | 1815.2 | 1822.4 (~0%) | 1416.6 (−22%) | 1848.2 (+2%) | | 4000 | 1535.9 | 1692.8 (+10%) | 1232.8 (−20%) | 1680.4 (+9%) | Two observations: 1. **The gap is largest at short ctx**, where a K=0 tier should be nearly free (drafter is loaded but the verifier is not asked to check any drafted tokens). At ctx=400, the DSD-mode arms are 20-31% slower than no-spec. 2. **The gap narrows as ctx grows**, and the 2D schedule overtakes no-spec at ctx ≥ 1900. The mechanism is memory-bound decode amortization — but the crossover would happen much earlier if the baseline tax at short ctx were smaller. ## Identified factor: PIECEWISE override Server logs confirm the DSD arms are downgraded from `FULL_AND_PIECEWISE` to `PIECEWISE`: ``` WARNING vllm.py:829 Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` The trigger is in `vllm/config/vllm.py::_maybe_override_dynamic_sd_cudagraph_mode` (lines 824-841): ```python def _maybe_override_dynamic_sd_cudagraph_mode(self) -> None: if ( speculative_config is None or not speculative_config.uses_dynamic_speculative_decoding() or not self.compilation_config.cudagraph_mode.has_full_cudagraphs() or self.use_v2_model_runner ): return self.compilation_config.cudagraph_mode = CUDAGraphMode.PIECEWISE ``` So all DSD schedules (including trivial `[[1, 512, 3]]` static schedules) lose full CUDA graphs unless `VLLM_USE_V2_MODEL_RUNNER=1` is set. The no-spec baseline retains full graphs, which likely explains part of the −31% gap at short ctx. **But cudagraph is not the only factor.** In an eager-mode control (`enforce_eager=True`, both arms lose graphs equally) at ctx=1900, static K=3 was still 6% slower than no-spec (0.94×), not faster as the mechanism would predict. So there is a residual DSD-mode overhead beyond the cudagraph-mode difference — probably drafter forward on K=0 steps, spec bookkeeping, admission cost — that needs to be decomposed. ## Proposed decomposition + fix path 1. **Decompose the tax** into: (a) PIECEWISE downgrade, (b) drafter forward on K=0 steps, (c) spec scheduling bookkeeping, (d) tier-boundary ramp effects (a small fraction of steps at c≈192-256 fire lower tiers than the aggregate suggests, per the ramp-leak observation in RFC #48627 §Motivation). 2. **True K=0 fast path**: when a DSD schedule selects K=0 for a given (bs, ctx), route through the no-spec path entirely for that step (skip drafter forward, skip spec accounting). This should recover most of the short-ctx gap without touching cudagraph capture. 3. **K-keyed FULL cudagraph capture**: since a declarative schedule enumerates the K set at startup (unlike learned per-step K predictors), the (batch bucket × K)-shape graphs are statically enumerable. Capturing them removes the need for the PIECEWISE downgrade in the DSD path — the \"resource-neutral\" property claimed for declarative schedules becomes literal once graphs are keyed on K. ## Repro Config: `prithivMLmods/gemma-4-31B-it-qat-FP8` + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`, H100 NVL 94GB, driver 580, CUDA 13.0, vLLM `c5d967c23` (branch `feat/dsd-2d-ctx-schedule` of #48944, but the tax is present with any DSD schedule including the merged `num_speculative_tokens_per_batch_size` from #32374). ```bash # static K=3 arm (reproduces the DSD-mode override): vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 \\ --speculative-config '{\"model\":\"\",\"num_speculative_tokens\":4,\"num_speculative_tokens_per_batch_size\":[[1,512,3]]}' # no-spec baseline: vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 # Bench (either arm): vllm bench serve --model --port 8000 \\ --dataset-name prefix_repetition --prefix-repetition-prefix-len 400 \\ --prefix-repetition-suffix-len 96 --prefix-repetition-num-prefixes 1 \\ --prefix-repetition-output-len 100 \\ --num-prompts 256 --max-concurrency 256 --ignore-eos \\ --percentile-metrics ttft,tpot,itl ``` Full raw data (server logs, per-cell measurements, snapshot deltas) available on request; happy to share the tarball or push to a public repo if that's easier for review. ## Context This came out of the measurement decomposition on #48944 — full 6-arm, 4-ctx table and methodology are in that PR's comment thread. The 2D ctx-axis schedule in #48944 is one specific instance of a DSD schedule; the baseline tax observed here applies to all DSD-mode arms (including the pre-existing `num_speculative_tokens_per_batch_size` API from #32374), so this issue is orthogonal to that PR and worth tracking separately. Happy to prototype the K=0 fast path as a follow-up PR if this direction sounds right — would appreciate a sanity check from @ekagra-ranjan (as the DSD API author) and @benchislett before I start.", "url": "https://github.com/vllm-project/vllm/issues/49986", "createdAt": "2026-07-27T13:04:39Z", "updatedAt": "2026-08-13T07:54:54Z", "timestamp": "2026-08-13T07:54:54Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b3180fd4644071cef99b", "signalId": "github:vllm-project/vllm:issue:48627", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:48627", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Context-length-aware speculative token scheduling — extending num_speculative_tokens_per_batch_size with a context-length axis", "text": "## Summary `num_speculative_tokens_per_batch_size` schedules the speculative depth K by batch size. We propose extending each entry with an optional, backward-compatible **context-length range**, so the runtime picks K from a `(batch, ctx)` table instead of a batch-only table: ```jsonc // today (unchanged, still valid): \"num_speculative_tokens_per_batch_size\": [[1,64,3],[65,128,1],[129,512,0]] // proposed (5-tuple entries; 3-tuple entries mean \"all context lengths\"): \"num_speculative_tokens_per_batch_size\": [ [1, 64, 0, 32768, 3], [65, 256, 0, 768, 0], // short-context high batch: spec off [65, 256, 769, 32768, 3], // long-context high batch: spec back on [257, 512, 0, 32768, 0] ] ``` *(Both tables are illustrative of the API shape, not prescriptive; K values are deployment-specific — see §Evidence for the measured Gemma-4-31B / H100 numbers this shape is drawn from.)* The scheduler change is one lookup dimension; per-K CUDA graphs and buffers are unaffected (they are keyed by K, not by table shape), so the extension is resource-neutral. ## Motivation **The batch-only rule \"speculation stops paying at high batch\" is a short-context artifact, not a general property.** We measured this on Gemma-4-31B (FP8, hybrid sliding+global attention) with its MTP drafter on a single H100 NVL 94GB, vLLM 0.23/0.24, greedy, fixed output length, 120s steady-state windows with Prometheus counter cross-checks. **1. Short-context crossover (the motivation for today's batch table) — reproduced:** | client concurrency | K=0 (tok/s) | best K (tok/s) | gain | |---|---|---|---| | 30 | 1,400 | 2,500 (K=3) | 1.79× | | 60 | 2,200 | 3,000 (K=3) | 1.36× | | 128 | 3,413 | 3,413 | 1.00× (converged) | **2. Same high batch, growing context (prefix-cache-hit regime, ~98% APC hit, prefill amortized) — the gain returns and grows:** | decode-time ctx (tok) | K=0 (tok/s) | K=3 (tok/s) | gain | K=0 TPOT | K=3 TPOT | |---|---|---|---|---|---| | ~460 | 3,107 | 3,640 | **1.17×** | 81.0 ms | 62.9 ms | | ~970 | 2,320 | 2,897 | **1.25×** | 109.3 ms | 77.8 ms | | ~1,990 | 2,110 | 2,915 | **1.38×** | 119.9 ms | 78.0 ms | | ~4,096 | 1,768 | 2,397 | 1.36× [1] | 137.8 ms | 96.9 ms [2] | (client concurrency 256 fixed; at concurrency 192 the ctx≈2k cell gives 1.45×.) [1] The 4k throughput ratio is a 3rd-run warm value; APC cache accumulation nudged it 1.28→1.32→1.36 run-to-run, so 1.36× is a conservative floor. The TPOT ratio (~1.4×, stable across 3 runs) is the more robust estimate. [2] 4k is a single point; we characterize it as *onset of decline*, not a decline curve — mapping ctx > 2k needs more points. The mechanism is visible in the TPOT column: as ctx doubles (970 → 1,990), **K=3 TPOT stays flat (77.8 → 78.0 ms) while K=0 TPOT keeps climbing (109 → 120 ms)**. Long-context decode is memory-bandwidth-bound on the per-step KV read; verifying K drafted tokens amortizes that read across K+1 tokens. The saving grows with ctx — but not without bound. **The gain peaks around ctx ≈ 1.5–2k (1.38× throughput / 1.54× TPOT) and begins to recede by 4k (1.36× / ~1.42×), as K=3's own per-step cost starts rising past 2k (its flat 78 ms breaks to 97 ms), narrowing the gap to K=0.** (We did not decompose that rise; candidates include the drafter's own long-context decode cost — SWA window growth, draft-layer FLOPs — and target-side KV growth.) The practical reading: speculation's sweet spot is the mid-to-long band (a few hundred to ~2k tokens); in the very-long regime the amortization gain converges to a ceiling rather than growing indefinitely. This is a stronger claim than monotone growth — it is bounded, mechanistic, and does not invite the \"then why not always speculate at long ctx\" objection. **Consequence: optimal K is a function of `(batch, ctx)` and is not separable.** A batch-only table forces one K per batch tier: - If the tier says K=0 (tuned for short-context saturation), long-context traffic at that batch loses a measured 1.2–1.45×. - If the tier says K>0 (tuned for long-context), short-context traffic at that batch pays verify overhead for no gain (and TTFT tail inflation). This matters most for exactly the workloads the ecosystem is optimizing for — agentic / RAG / multi-turn traffic with long, prefix-shared contexts at high concurrency. Related discussion where a member raised the \"MTP may only help for small batch size\" rule and we posted this datapoint: #47277. A secondary observation reinforces that even the *batch* axis is currently scheduled coarsely: on the same stack, short-context K=3 stays optimal further up the batch range than a typical hand-tuned table assumes — at 110 scheduled requests K=3 (3,300 tok/s) still beats K=1 (3,185) with a healthy TTFT p99 (796 ms), only converging to K=0 at the ~128 crossover. A static per-batch table with a conservative middle tier (e.g. dropping to K=1 at 65) leaves throughput on the table. This is orthogonal to the ctx axis but points the same way: the optimal-K surface is finer than a coarse batch-only table captures, along both axes. ## Proposed Change **1. Schema (`vllm/config/speculative.py`, `v1/spec_decode/dynamic/utils.py`):** accept 5-tuple entries `(bs_lo, bs_hi, ctx_lo, ctx_hi, K)` alongside today's 3-tuples (interpreted as `ctx_lo=0, ctx_hi=max_model_len`). Validation extends the existing rules: inclusive ranges, bs coverage from 1, non-overlapping, and per-bs-range full ctx coverage (rectangular grid). **2. Runtime lookup (`v1/core/sched/scheduler.py`):** today the scheduler does `dynamic_sd_lookup[len(num_scheduled_tokens)]`. We extend the dense lookup to two dimensions: `dense[B][ctx_bucket]`, where the batch's context representative is the **p50 of decode-time sequence lengths of the scheduled requests** — information the scheduler already holds, so the new signal costs nothing. `SchedulerOutput.num_spec_tokens_to_schedule` is unchanged (still a scalar per step). **3. Resource neutrality:** CUDA graphs and runtime buffers are keyed by the K values appearing in the schedule, not by the number of table cells (this is already how the per-K capture works). A 2D table with palette {0,1,3} captures exactly the same graphs as a 1D table with the same palette. **The context axis is resource-free.** **4. Docs:** two clarifications we found necessary in practice: - The table index is the **per-step scheduled request count**, not client concurrency; it fluctuates through admission ramps, so tier boundaries near a workload's steady running level leak (~2% draft volume in our measurement). Guidance: place boundaries outside the running-distribution tail. - Because the amortization gain saturates around ~2k (see Motivation), ctx buckets need not be fine-grained in the long tail: a single bucket covering \"≳2k\" is sufficient in our data, keeping the table small. - K=0 tiers do not currently make speculation free (drafter obligations persist; see the overhead reports in #47277 and our isolation summary there). In our measurement the penalty concentrates in TTFT (prefill/scheduling: p50 0.28s → 2.0s) rather than TPOT (+6.7%), consistent with the input-preparation hotspots reported in #47277. That is an orthogonal issue, but users sizing K=0 tiers for high-batch production should know it. ## Evidence & reproducibility Full tables, the measurement protocol (120s windows / 30s warmup / generated-token deltas / spec-counter cross-checks), harness, and an engine-agnostic reference controller (declarative `(B, ctx)` table + acceptance-rate correction + overload override) are public: **https://github.com/seongyun1104/depthchart**. MTP × DSD runtime tier switching on 0.24.0 was verified with spec-decode counters (c30 → K=3 with drafts/step ≈ 3.0; c400 → K=0 with zero drafts); the DSD docs currently note testing with Eagle/E3 only, so this doubles as an MTP datapoint (capture-path bug on 0.25 filed as #48494, backend-selection issue as #48495). ## Alternatives considered - **Acceptance-driven adaptation only** (SGLang `--speculative-adaptive` style): reacts to drafter quality but not to the regime. Two structural limits: at K=0 the acceptance signal vanishes and requires periodic probing to escape; and acceptance does not encode the ctx-dependent verify economics at all (our AR was flat 86/66/50 per position across c=30→128 while the gain moved from 1.79× to 1.00×). A `(B, ctx)` table's inputs never vanish; acceptance works better as a *correction* layer on top. - **Per-sequence K**: strictly more expressive, but requires variable-K verify batching and straggler control (cf. DSDE's per-sequence SL with a cap). The per-batch 2D table is the minimal change that captures the measured effect; per-sequence can layer on later. - **Entropy/complexity signals** (HeteroSpec-style): orthogonal — they estimate *acceptance*, not the *verify-side KV-read economics* that the ctx axis captures. (Note: HeteroSpec's *contextual* refers to language / semantic complexity of the input, not context length; the two signals target different phenomena and are compatible layers rather than substitutes.) ## Composition with concurrent adaptive-verification work An adaptive-verification design is being iterated in the DSpark track — #48692 (open, actively iterated; per its author on 2026-07-15, *\"Iterated design should land somewhere between this PR and #47808\"*). Its body is explicit that this layer does not schedule its own verification budget: it consumes the user-provided `num_speculative_tokens_per_batch_size` as the per-batch-size pool of draft tokens to verify. This RFC extends exactly that surface — existing 3-tuple entries normalize to a single full-range ctx bucket, so consumers that read the schedule (including `SpeculativeConfig.uses_dynamic_speculative_decoding()` gating already in main) see no interface change. The two designs are then strictly compositional layers: the DSpark track varies **per-request K within a budget cell** (spend more of the budget where confidence is high); this RFC defines **the per-(bs, ctx) budget cell itself**. ## Policy ladder The `(bs, ctx)` table this RFC adds is the base rung of a three-rung policy ladder for choosing K; the higher rungs remain open surface for follow-up work and other tracks. 1. **Declarative table (this RFC)** — a zero-lag prior over K derived from deployment characterization. Tier-boundary transitions incur no learning delay because the lookup is a pure function of `(bs, ctx)`; the moment the batch composition crosses a boundary, the new K applies. This is the adaptation-latency argument for keeping a table as the base layer even under adaptive strategies: online estimators (per-bucket AR, hybrid cost/acceptance models) have inherent settling time on regime shifts, whereas the table's response is O(1) lookup. 2. **Per-bucket acceptance correction** — table K modified by the running acceptance rate observed in each `(bs, ctx)` cell. Handles drafter quality drift that a static table cannot; DSDE and SGLang `--speculative-adaptive` are single-axis instances of this rung. 3. **Hybrid cost-model × acceptance** — a full verify-cost model composed with per-bucket acceptance to select K online (cf. the DSpark confidence-scheduled verification track in #47808). Requires the base table as its bootstrap prior — the cost model has to know where to look for signal before it can improve on the prior. This RFC defines rung 1. #48692 / #47808 iterate rung 3 and (as noted in the Composition section above) consume the schedule from rung 1 as the per-batch-size budget pool for their per-request budget scheduling. Rung 2 is the natural intermediate; a first-class implementation is out of scope here but the schedule surface accommodates it (`num_speculative_tokens` override at request time on top of the table lookup). ## Prior art & scope honesty Batch-size-conditioned K is established (this feature; SGLang ships batch-tiered candidate sets by default). \"Context-aware speculative decoding\" (CASD, 2024) names a different technique — drafting by retrieval *from* the context. ~~We have not found prior art that uses **context length as a first-class scheduling axis for K**, nor measurements of the `(batch, ctx)` interaction; we would welcome pointers if they exist.~~ **Update (2026-07-27):** The core observation — that optimal K should depend on sequence length under memory-bound decode — was formalized by **MagicDec (Sadhukhan et al., arXiv:2408.11049, 2024)**: *\"Leveraging our observation that there is a sequence length threshold above which decoding becomes memory bound, we propose choosing the amount of speculation as a function of the sequence length (longer sequence length → more speculated tokens).\"* MagicDec's instantiation was self-speculation with StreamingLLM sparse KV in a batch-centric framing; this RFC extends the general draft-target DSD lookup surface with an explicit ctx-length axis. See the [MagicDec anchor comment below](#issuecomment-5091701498) for the framing detail. **TurboSpec / SmartSpec lineage:** The dynamic-schedule surface this RFC extends was formalized as SmartSpec (Liu et al., [arXiv:2406.14066](https://arxiv.org/abs/2406.14066), later retitled TurboSpec) — the batch-adaptive predecessor of the `num_speculative_tokens_per_batch_size` API landed via #5886 / #32374. TurboSpec's evaluation focuses on the ≤1k-context regime where the crossover we observe at ctx≈2k sits to their right; their reported gains and the 1.29-1.36× we measure at ctx 900/1900/4000 are adjacent regions of the same K-vs-(B, ctx) surface rather than competing claims. The ctx-length axis this RFC adds makes both regions addressable from one declarative schedule. Known limitations of our data: one model/drafter pair (Gemma-4-31B FP8 + QAT-matched MTP head — drafter lineage alone moved acceptance 51.6% → 67.7% in our A/B, so table *values* are deployment-specific and need calibration; the *shape* of the surface is what this RFC relies on); the ctx sweep is from a prefix-cache-hit regime (miss-heavy traffic unmeasured); single GPU, TP=1. ## Implementation We are happy to contribute the patch: the change is contained (schema validation + dense-builder in `dynamic/utils.py`, one lookup site in `scheduler.py`, docs), backward compatible, and covered by the validation rules above. A second-engine replication of the mechanism (SGLang `_route` extension) is in progress on our side and can inform the design review. ## Feedback Period 2 weeks, or as maintainers prefer. ## CC @ekagra-ranjan @benchislett @luccafong @MatthewBonanni @JumpingRain", "url": "https://github.com/vllm-project/vllm/issues/48627", "createdAt": "2026-07-14T15:28:22Z", "updatedAt": "2026-08-13T07:54:53Z", "timestamp": "2026-08-13T07:54:53Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8572ce5e9497d3b48725", "signalId": "github:vllm-project/vllm:issue:47277", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:47277", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Qwen3.5 native MTP can be slower than no-MTP CUDA graph baseline despite good acceptance", "text": "## Summary We are benchmarking a Qwen3.5 OCR checkpoint with native MTP support in vLLM. The model has a lightweight native MTP branch: - `mtp_num_hidden_layers=1` - `mtp_use_dedicated_embeddings=false` In our OCR workload, MTP acceptance is not terrible. MTP1 reaches around `82%-88%` acceptance, and MTP2 can reach mean accepted length above `2.3`. However, end-to-end throughput in graph mode is still below the best no-MTP vLLM baseline. The main issue seems to be that the vLLM spec decode execution path adds enough per-step overhead to erase the theoretical benefit from fewer decode steps. This is particularly visible with `num_speculative_tokens=2`, where rejection sampling, metadata preparation, slot mapping, and `num_computed_tokens` updates become large overheads. We would like to understand whether this is expected for native Qwen MTP today, whether `num_speculative_tokens > 1` currently requires repeated forward/processing through the same MTP layer, and whether there are planned optimizations for the graph-mode spec decode path. ## Environment - vLLM: `0.23.0` - PyTorch: `2.11.0+cu130` - GPU: NVIDIA L20X - Tensor parallel: `1` - CUDA graph mode: enabled, `enforce_eager=false` - Workload: multimodal OCR / document parsing, long outputs - Decode temperature: `0` Main generation config: ```text max_new_tokens=16384 max_model_len=32768 max_num_batched_tokens=32768 gpu_memory_utilization=0.8 max_num_seqs=None ``` We intentionally did not give MTP a larger `max_num_seqs` or give baseline a smaller one. Both paths use the same input subset and batching-related configuration. ## Speculative configs tested MTP1: ```json {\"method\":\"mtp\",\"num_speculative_tokens\":1,\"use_local_argmax_reduction\":true} ``` MTP2: ```json {\"method\":\"mtp\",\"num_speculative_tokens\":2,\"use_local_argmax_reduction\":true} ``` ## Results ### n64 graph-mode clean benchmark | mode | tok/s | req/s | bench sec | output tokens | finish | acceptance | mean accept length | per-position acceptance | | --- | ---: | ---: | ---: | ---: | --- | ---: | ---: | --- | | no-MTP baseline | 2246.73 | 1.381 | 46.35 | 104138 | `stop:64` | - | - | - | | MTP1 | 1926.41 | 1.181 | 54.21 | 104432 | `stop:64` | 82.83% | 1.828 | 82.83% | | MTP2 | 1821.76 | 1.119 | 57.17 | 104152 | `stop:64` | 66.77% | 2.335 | 78.21%, 55.34% | MTP1 is about `0.86x` of the no-MTP graph baseline, not a speedup. MTP2 has a higher mean accepted length, but still does not beat the best no-MTP graph baseline. The second speculative token acceptance is also much lower than the first token. ### n32 timing benchmark after local instrumentation | mode | tok/s | output tokens | finish | acceptance | mean accept length | | --- | ---: | ---: | --- | ---: | ---: | | no-MTP baseline | 1898.19 | 29017 | `stop:32` | - | - | | MTP1 before local Triton patch | 1824.31 | 29024 | `stop:32` | 88.06% | 1.881 | | MTP1 after local Triton patch | 1980.96 | 29016 | `stop:32` | 88.08% | 1.881 | | MTP2 after local Triton patch | 2172.36 | 28972 | `stop:32` | 76.03% | 2.521 | The n32 timing case is not our final speed claim; it is mostly for profiling. It suggests that some overhead can be reduced, but the larger n64 graph-mode benchmark still does not show a stable MTP speedup over the best no-MTP baseline. ## Timing hotspots With local timing instrumentation, MTP2 n32 shows the following cumulative costs: | item | cumulative time | | --- | ---: | | `gpu_draft_total` | 3017.8 ms | | `runner_prepare_inputs` | 2276.5 ms | | `reject_sample_kernel` | 1702.8 ms | | `gpu_target_forward` | 1357.8 ms | | `prepare_num_computed_gpu` | 1033.5 ms | | `prepare_num_computed_update_kernel` | 1016.9 ms | | `step35_loop_update_positions` | 668.5 ms | | `prepare_compute_slot_mapping` | 603.1 ms | Before our local patch, `prepare_num_computed_gpu` was one of the clearest overheads. In MTP1 n32, we observed: | item | baseline | MTP1 before patch | | --- | ---: | ---: | | `prepare_num_computed_gpu` | 6.4 ms | 626.7 ms | | `runner_prepare_inputs` | 529.8 ms | 1346.0 ms | | `gpu_target_forward` | 858.0 ms | 1243.5 ms | | `gpu_draft_total` | - | 531.5 ms | We locally replaced a `torch.compile(dynamic=True)` style `update_num_computed_tokens_for_batch_change` path with a small fused Triton kernel. This improved MTP1 n32 throughput from `1824.31 tok/s` to `1980.96 tok/s`, but it did not solve the full graph-mode overhead problem, especially for MTP2. ## Local experiments that were not safe enough to use We also tried more aggressive local fast paths: - a greedy rejection fast path for `temperature=0` - packed/allfast variants - faster attention metadata experiments - GPU token cache / active request gating variants Some of these either regressed performance or produced non-clean outputs such as `length:1` in offset tests, so we did not count them as valid benchmark results. ## Questions 1. Is it expected that Qwen native MTP in vLLM graph mode can be slower than no-MTP graph baseline even with MTP1 acceptance around `82%-88%`? 2. For `num_speculative_tokens > 1`, does vLLM currently run the same Qwen MTP layer multiple times / perform repeated per-token processing rather than producing multiple draft tokens in one low-overhead pass? 3. Are there known bottlenecks or planned optimizations for: - `reject_sample_kernel` - spec decode metadata preparation - slot mapping updates - `num_computed_tokens` correction/update - target/draft metadata reuse under CUDA graph 4. Is there a recommended benchmark setup for fairly comparing no-MTP graph baseline vs native MTP graph mode for long-output multimodal workloads? 5. Would the vLLM team be interested in a minimal repro using a public Qwen3.5 native-MTP checkpoint, if available, or local profiling traces from our private OCR checkpoint? ## Expected behavior For a native MTP model with `80%+` MTP1 acceptance and graph mode enabled, we would expect MTP1 to at least be competitive with no-MTP graph baseline, or for documentation to clarify that MTP may be slower in graph mode unless the acceptance rate and sequence/batch shape meet specific thresholds. For MTP2/MTP3, we would expect either: - a low-overhead multi-token draft path, or - guidance that `num_speculative_tokens > 1` may be slower for Qwen native MTP because it requires repeated MTP forward/processing. ## Actual behavior On this workload: - no-MTP graph baseline is still the fastest stable path. - MTP1 has good acceptance but is slower end to end. - MTP2 increases mean accepted length but introduces enough extra overhead that it still does not beat the best no-MTP graph baseline. - Local profiling points to rejection sampling, metadata preparation, slot mapping, and `num_computed_tokens` updates as major overheads. ## Additional context This benchmark is from a private OCR/document parsing workload, so some exact data/model paths are not public. We can prepare a smaller reproducer if a public Qwen native-MTP checkpoint is suitable. The most relevant local observation is that the bottleneck is not just draft quality. Even when acceptance is usable, the spec decode path can be too expensive relative to a strong CUDA graph no-MTP baseline.", "url": "https://github.com/vllm-project/vllm/issues/47277", "createdAt": "2026-07-01T09:23:25Z", "updatedAt": "2026-08-13T07:54:47Z", "timestamp": "2026-08-13T07:54:47Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [], "author": "JumpingRain", "state": "open", "assignees": [ "andakai" ], "change": "updated" } }, { "id": "event:9d1b4f935d00cdcf0c9b", "signalId": "github:vllm-project/vllm:pull_request:46134", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for merge_attn_states kernel", "text": "## Purpose Adds a device-agnostic unit test for the `merge_attn_states` Triton kernel (Attention group) in `tests/kernels/attention/test_merge_attn_states_torch_ref.py`. The existing `test_merge_attn_states.py` compares the Triton kernel against a custom CUDA op (`merge_attn_states_cuda`) and skips on non-CUDA platforms, so it cannot run on XPU. This test instead uses a naive PyTorch reference as the oracle (Section 2.2 of arxiv 2501.01005 — the softmax-weighted combination of two split-KV partial attention results), making it runnable on both CUDA and XPU (`current_platform.device_type`, no skip guard). Kept as a separate file so the existing CUDA-parity test is preserved. Covers output and output_lse across token counts, head counts, head sizes, and fp16/bf16, including ~10% +inf lse entries (a split that attended to no keys). ## Test Plan `python -m pytest tests/kernels/attention/test_merge_attn_states_torch_ref.py -v` ## Test Result All 36 cases pass.", "url": "https://github.com/vllm-project/vllm/pull/46134", "createdAt": "2026-06-19T06:54:28Z", "updatedAt": "2026-08-13T07:54:02Z", "timestamp": "2026-08-13T07:54:02Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:2ed5ad5e228d1f058f61", "signalId": "github:vllm-project/vllm:pull_request:50535", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50535", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Use AITER tuned GEMM for the MoE router gate", "text": "## Purpose The MoE gate linear layer was set to use a bf16×bf16→fp32 GEMM on tier 5 for faster performance, but it is conditioned on Hopper and Blackwell. This enables it for AITER when the underlying tuned GEMM shapes exist. ### Trace comparison \"pr_fig1_cpu_stack\" On the CPU side, the gate previously performed two operations: a conversion of its input to fp32, followed by the multiply. AITER was already invoked, but with fp32 operands it found no tuned kernel and fell through to `aten::linear` → `aten::matmul` → `aten::mm`. The input now remains in bf16, AITER dispatches a tuned kernel directly, and the cast is moved after the GEMM, where it applies only to the small output. \"pr_fig2_gpu_kernels\" The GPU stream shows the same change. The `add_rmsnorm_quant` kernel on the left is unaffected by this PR and serves as a scale reference between the two panels. ## Test Plan `tests/kernels/test_rocm_aiter_router_gemm.py`, skipped unless ROCm and AITER tuned GEMM are available. It verifies agreement with an fp64 reference, top-k expert selection, and equivalence with the `F.linear` fallback it replaces, across three router shapes and M from 1 to 128. End-to-end throughput, latency and accuracy were measured on MI355X against an otherwise identical build. ## Test Result 81/81 kernel tests pass on gfx950 with `VLLM_ROCM_USE_AITER=1`. MI355X, TP4, geomean over a 16-point concurrency sweep (ISL 1024/8192, OSL 1024): **TPOT -3.05%, output throughput +3.10%**. Accuracy is unchanged within one standard error: gsm8k strict-match 0.9515 against 0.9500 baseline, RULER niah_single_2 0.876 against 0.860. `ruff check` and `ruff format` are clean. ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 86.0 | 90.0 | **+4.6%** | 381 | 384 | 11.17 | 10.66 | -4.6% | | 1024 | 4 | 32 | 315.1 | 323.0 | **+2.5%** | 329 | 326 | 12.01 | 11.71 | -2.5% | | 1024 | 8 | 64 | 497.3 | 510.3 | **+2.6%** | 144 | 141 | 14.90 | 14.54 | -2.4% | | 1024 | 16 | 96 | 797.9 | 826.9 | **+3.6%** | 221 | 213 | 18.08 | 17.44 | -3.5% | | 1024 | 32 | 128 | 1180.4 | 1209.6 | **+2.5%** | 364 | 350 | 23.35 | 22.78 | -2.4% | | 1024 | 64 | 256 | 1786.7 | 1834.7 | **+2.7%** | 475 | 454 | 30.59 | 29.81 | -2.5% | | 1024 | 128 | 384 | 2577.1 | 2633.2 | **+2.2%** | 1079 | 1017 | 41.06 | 40.17 | -2.2% | | 1024 | 256 | 768 | 3850.1 | 3944.5 | **+2.5%** | 1819 | 1667 | 56.98 | 55.72 | -2.2% | | 8192 | 1 | 16 | 82.0 | 85.6 | **+4.3%** | 360 | 345 | 11.84 | 11.35 | -4.1% | | 8192 | 4 | 32 | 292.9 | 300.2 | **+2.5%** | 476 | 468 | 12.83 | 12.51 | -2.5% | | 8192 | 8 | 64 | 420.0 | 432.1 | **+2.9%** | 697 | 673 | 17.25 | 16.70 | -3.2% | | 8192 | 16 | 96 | 605.9 | 627.0 | **+3.5%** | 891 | 854 | 23.64 | 22.75 | -3.8% | | 8192 | 32 | 128 | 848.5 | 873.9 | **+3.0%** | 1911 | 1851 | 33.13 | 32.15 | -3.0% | | 8192 | 64 | 256 | 1136.1 | 1174.3 | **+3.4%** | 3427 | 3250 | 49.81 | 48.15 | -3.3% | | 8192 | 128 | 384 | 1452.9 | 1502.4 | **+3.4%** | 7128 | 6696 | 77.31 | 74.99 | -3.0% | | 8192 | 256 | 768 | 1731.1 | 1792.3 | **+3.5%** | 15375 | 14468 | 128.75 | 124.28 | -3.5% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+2.9%** | -3.6% | -2.8% | | ISL 8192 | **+3.3%** | -4.2% | -3.3% | | **all 16** | **+3.1%** | -3.9% | -3.0% | Per-case expansion of the geomean quoted above. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9500 | 0.9515 | | gsm8k flexible-extract | 0.9538 | 0.9553 | | RULER niah_single_2 @16k | 0.860 | 0.876 |", "url": "https://github.com/vllm-project/vllm/pull/50535", "createdAt": "2026-07-31T06:14:30Z", "updatedAt": "2026-08-13T07:53:19Z", "timestamp": "2026-08-13T07:53:19Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:49942e19ad4a255c2478", "signalId": "github:vllm-project/vllm:pull_request:51309", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51309", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Skip redundant sparse index remap on non-indexer layers", "text": "## Summary With cross-layer index sharing, only some attention layers update the shared top-k buffer. The old path still remapped that buffer on every layer. This PR runs the remap only when the layer owns an indexer. For GLM-5.2 with `index_topk_freq = 4`, this reduces remaps from 78 to 21 per forward pass. ## Safety - Layer 0 always owns an indexer, so the shared output is initialized before reuse. - Non-indexer layers do not change the inputs to the remap. - The default `index_topk_freq = 1` path is unchanged. - Speculative decoding still remaps on every layer. ## Results MI355X, TP=4, rank 0, same serving workload: | remap kernel | before | after | change | |---|---:|---:|---:| | launches | 139,378 | 37,254 | **-73.3%** | | GPU time | 2,392.1 ms | 645.7 ms | **-73.0%** | | launches per layer | 1.000 | 0.269 | matches 21/78 | The red box below is a redundant remap on a layer with no indexer. In the second capture, the same layer enters the MLA kernel directly. ![Before: redundant remap](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr2_before.png) ![After: remap removed](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr2_after.png) ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 81.8 | **+0.7%** | 1392 | 1385 | 10.80 | 10.73 | -0.6% | | 1024 | 4 | 32 | 330.5 | 332.4 | **+0.6%** | 131 | 130 | 11.66 | 11.60 | -0.5% | | 1024 | 8 | 64 | 562.4 | 567.0 | **+0.8%** | 142 | 141 | 13.08 | 12.96 | -0.9% | | 1024 | 16 | 96 | 875.0 | 879.9 | **+0.6%** | 211 | 210 | 16.44 | 16.37 | -0.4% | | 1024 | 32 | 128 | 1333.8 | 1329.9 | **-0.3%** | 333 | 337 | 20.58 | 20.63 | +0.2% | | 1024 | 64 | 256 | 2026.2 | 2057.8 | **+1.6%** | 442 | 437 | 26.96 | 26.45 | -1.9% | | 1024 | 128 | 384 | 2987.2 | 2999.1 | **+0.4%** | 986 | 973 | 35.19 | 35.01 | -0.5% | | 1024 | 256 | 768 | 4338.7 | 4378.1 | **+0.9%** | 1630 | 1633 | 50.59 | 50.17 | -0.8% | | 8192 | 1 | 16 | 84.5 | 85.1 | **+0.7%** | 339 | 338 | 11.50 | 11.42 | -0.7% | | 8192 | 4 | 32 | 302.2 | 304.3 | **+0.7%** | 447 | 442 | 12.44 | 12.36 | -0.6% | | 8192 | 8 | 64 | 473.4 | 474.2 | **+0.2%** | 666 | 640 | 15.10 | 15.16 | +0.4% | | 8192 | 16 | 96 | 665.3 | 668.3 | **+0.4%** | 825 | 824 | 21.44 | 21.32 | -0.6% | | 8192 | 32 | 128 | 949.8 | 949.6 | **-0.0%** | 1861 | 1749 | 29.40 | 29.58 | +0.6% | | 8192 | 64 | 256 | 1269.4 | 1276.5 | **+0.6%** | 2991 | 3060 | 44.80 | 44.31 | -1.1% | | 8192 | 128 | 384 | 1634.2 | 1649.1 | **+0.9%** | 6450 | 6453 | 68.91 | 68.02 | -1.3% | | 8192 | 256 | 768 | 1909.0 | 1927.7 | **+1.0%** | 14096 | 13901 | 116.63 | 115.29 | -1.1% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.6%** | -0.5% | -0.7% | | ISL 8192 | **+0.6%** | -1.3% | -0.6% | | **all 16** | **+0.6%** | -0.9% | -0.6% | Throughput improves on 14 of 16 cases; the two exceptions are both at concurrency 32, at -0.3% and -0.0%. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9447 | | gsm8k flexible-extract | 0.9507 | 0.9469 | | RULER niah_single_2 @16k | 0.870 | 0.884 | Unchanged within run-to-run variation. The skipped remap rewrites `paged_kv_indices` with bit-identical values, so no output change is expected. ## Tests ```bash pytest tests/kernels/attention/test_rocm_aiter_mla_sparse_index_remap.py ``` The tests cover the indexer/speculative-decoding gate and verify whether the remap is called.", "url": "https://github.com/vllm-project/vllm/pull/51309", "createdAt": "2026-08-06T21:10:14Z", "updatedAt": "2026-08-13T07:53:17Z", "timestamp": "2026-08-13T07:53:17Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6f66e8518cb713655b70", "signalId": "github:vllm-project/vllm:pull_request:51674", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51674", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN", "text": "## Purpose Speed up Qwen3.5 (GDN linear attention) MTP speculative decode on Blackwell. During MTP decode, the Triton path launches a chain of small kernels per step (gating, delta-rule recurrence, state rewind/update, gated RMSNorm), which leaves the GPU latency-bound at decode batch sizes. This PR adds a single fused CUDA kernel, `fused_gdn_decode_post_conv_mtp`, that consumes the post-convolution `mixed_qkv` and performs the entire GDN MTP decode step in one launch: - SiLU activation on q/k/v and gating (`softplus`/`sigmoid` on `a`, `b`, `A_log`, `dt_bias`) - the gated delta-rule recurrence over up to 8 speculative tokens per request, with rewind to the last accepted token via `num_accepted_tokens` - recurrent state update in place (BF16 or FP32 state) - gated RMSNorm epilogue, with output precision matched to the Triton path ## Test Plan New/extended tests (all require SM100): ```bash pytest tests/kernels/mamba/test_gdn_fused_mtp.py -v pytest tests/kernels/test_fused_gdn_post_conv.py -v pytest tests/v1/attention/test_gdn_metadata_builder.py -v pytest tests/test_envs.py -k \"gdn\" pytest tests/compile/test_config.py ``` `test_gdn_fused_mtp.py` covers the model-path dispatch for pure spec-decode, mixed, prefill, and regular-decode batches against the Triton reference. `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` covers the kernel directly for BF16/FP32 state, ragged acceptance patterns, and TP head counts (tp4/tp16). ## Test Result On 1x B200 (SM100), CUDA 13.1, torch 2.13.0+cu130: - `tests/kernels/mamba/test_gdn_fused_mtp.py`: **6 passed** - `tests/kernels/test_fused_gdn_post_conv.py`: **70 passed** - `tests/v1/attention/test_gdn_metadata_builder.py`: **10 passed** - `tests/test_envs.py -k gdn`: **1 passed** - `tests/compile/test_config.py`: **44 passed** ## Microbenchmark 1x B200, CUDA 13.1, torch 2.13.0+cu130. Both variants measured over the **same kernel boundary**: post-conv `mixed_qkv` in, gated+normed `core_attn_out` out, recurrent state updated in place. - Triton path = `fused_sigmoid_gating_delta_rule_update` + gated RMSNorm (`rmsnorm_fn`), exactly as dispatched by `_forward_core_fused_norm` today. - Fused path = single `fused_gdn_decode_post_conv_mtp` launch. - Timed by capturing 20 iterations in a CUDA graph and replaying 50x with CUDA events (production MTP decode runs under full CUDA graphs, so graph-replay GPU time is the relevant metric; CPU launch overhead excluded for both). - **cold** rotates input/state buffer sets so the working set exceeds 4x L2 (126 MB), i.e. state comes from HBM as in real serving where other layers evict L2 between calls; **warm** reuses one buffer set (L2-hot upper bound). - K=V=128, 4 MTP tokens/request, `num_accepted_tokens=2`. Context length does not affect either variant (fixed-size recurrent state), so results apply to any seqlen (e.g. 8192). Cold-cache speedup is **1.17x-2.20x**, largest in the latency-sensitive BS=4-32 decode regime. **TP=4 (H=4, HV=32), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 20.31 | 12.19 | 1.67x | 16.49 | 11.57 | 1.42x | | 4 | 27.91 | 12.99 | 2.15x | 20.65 | 11.73 | 1.76x | | 8 | 35.69 | 17.89 | 2.00x | 27.26 | 16.59 | 1.64x | | 16 | 46.66 | 32.32 | 1.44x | 41.44 | 30.60 | 1.35x | | 32 | 71.85 | 51.75 | 1.39x | 71.99 | 51.62 | 1.39x | | 64 | 120.65 | 97.12 | 1.24x | 120.61 | 96.93 | 1.24x | | 128 | 221.13 | 188.16 | 1.18x | 221.11 | 188.02 | 1.18x | **TP=4 (H=4, HV=32), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 23.50 | 13.73 | 1.71x | 17.74 | 12.70 | 1.40x | | 4 | 32.04 | 14.57 | 2.20x | 24.12 | 12.82 | 1.88x | | 8 | 42.69 | 20.78 | 2.05x | 33.78 | 18.68 | 1.81x | | 16 | 60.27 | 38.14 | 1.58x | 60.01 | 37.70 | 1.59x | | 32 | 96.82 | 65.27 | 1.48x | 97.08 | 65.15 | 1.49x | | 64 | 167.63 | 130.97 | 1.28x | 167.27 | 126.25 | 1.32x | | 128 | 311.52 | 267.23 | 1.17x | 314.21 | 266.75 | 1.18x | **TP=16 (H=1, HV=8), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 15.87 | 11.52 | 1.38x | 15.92 | 11.57 | 1.38x | | 4 | 19.64 | 12.32 | 1.59x | 16.46 | 11.57 | 1.42x | | 8 | 23.52 | 12.67 | 1.86x | 17.92 | 11.61 | 1.54x | | 16 | 27.56 | 12.92 | 2.13x | 20.64 | 11.69 | 1.77x | | 32 | 35.39 | 17.82 | 1.99x | 27.16 | 16.60 | 1.64x | | 64 | 46.45 | 32.27 | 1.44x | 40.72 | 30.57 | 1.33x | | 128 | 73.57 | 51.77 | 1.42x | 73.69 | 51.61 | 1.43x | **TP=16 (H=1, HV=8), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 17.44 | 12.59 | 1.38x | 17.71 | 12.61 | 1.41x | | 4 | 23.30 | 13.71 | 1.70x | 17.92 | 12.68 | 1.41x | | 8 | 26.18 | 13.93 | 1.88x | 20.08 | 12.71 | 1.58x | | 16 | 31.77 | 14.50 | 2.19x | 23.99 | 12.81 | 1.87x | | 32 | 42.81 | 20.84 | 2.05x | 34.81 | 19.05 | 1.83x | | 64 | 61.15 | 38.45 | 1.59x | 60.99 | 37.89 | 1.61x | | 128 | 98.89 | 66.26 | 1.49x | 99.21 | 66.21 | 1.50x | --- ## Model Evaluation An lm_eval score comparison was not run: the fused path requires the `num_v_heads == 8 * num_k_heads` GDN head layout, and no public Qwen3.5 checkpoint ships that config (0.8B is 16/16, 35B-A3B is 16/32, 397B-A17B is 16/64) — on those models the dispatch check never selects the fused kernel, so default behavior is unchanged. Correctness of the fused kernel against the Triton path is covered by the numerical-equivalence tests above: `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` checks kernel outputs and updated recurrent state against `fused_sigmoid_gating_delta_rule_update` + `rmsnorm_fn` across BF16/FP32 state, ragged acceptance patterns, and TP head counts, and `test_gdn_fused_mtp.py` checks the model-path dispatch (pure / mixed / prefill / decode batches) end-to-end at the layer level.
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51674", "createdAt": "2026-08-10T12:49:25Z", "updatedAt": "2026-08-13T07:53:15Z", "timestamp": "2026-08-13T07:53:15Z", "metrics": { "reactions": 5, "comments": 7 }, "labels": [ "ready", "ci/build", "qwen", "nvidia" ], "author": "Jie-Fang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e402683defcd7fdab184", "signalId": "github:vllm-project/vllm:pull_request:51314", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51314", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Skip cleaning sparse prefill MQA logits", "text": "## Summary The sparse prefill path creates the MQA-logits buffer with `torch.full(..., -inf)`. This launches an FP32 fill before every logits kernel. This PR passes `clean_logits=False` to AITER, so the buffer uses `torch.empty` and the fill is removed. ## Safety The only consumer is `top_k_per_row_prefill`. It reads each row only inside `[cu_seqlen_ks, cu_seqlen_ke)`. Its vectorized path also guards the final partial load, so padding values are never read. The top-k result is unchanged. ## Results MI355X, TP=4, rank 0, comparable before/after captures: | metric | before | after | |---|---:|---:| | FP32 fill launches | 18,162 | **0** | | FP32 fill GPU time | 1,250.2 ms | **0 ms** | | MQA-logits launches | 18,108 | 17,930 | The removed fill was **0.89% of total GPU kernel time**. In the sample below it takes 74.6 µs and is fully serialized before the logits kernel. ![Before: FP32 fill before MQA logits](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr4_before.png) ![After: fill removed](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr4_after.png) ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 81.2 | **-0.1%** | 1392 | 1387 | 10.80 | 10.82 | +0.2% | | 1024 | 4 | 32 | 330.5 | 330.5 | **+0.0%** | 131 | 131 | 11.66 | 11.66 | +0.0% | | 1024 | 8 | 64 | 562.4 | 561.0 | **-0.3%** | 142 | 142 | 13.08 | 13.10 | +0.2% | | 1024 | 16 | 96 | 875.0 | 880.4 | **+0.6%** | 211 | 211 | 16.44 | 16.35 | -0.5% | | 1024 | 32 | 128 | 1333.8 | 1333.5 | **-0.0%** | 333 | 333 | 20.58 | 20.56 | -0.1% | | 1024 | 64 | 256 | 2026.2 | 2042.7 | **+0.8%** | 442 | 438 | 26.96 | 26.66 | -1.1% | | 1024 | 128 | 384 | 2987.2 | 2985.9 | **-0.0%** | 986 | 979 | 35.19 | 35.17 | -0.1% | | 1024 | 256 | 768 | 4338.7 | 4337.8 | **-0.0%** | 1630 | 1628 | 50.59 | 50.55 | -0.1% | | 8192 | 1 | 16 | 84.5 | 84.3 | **-0.3%** | 339 | 336 | 11.50 | 11.53 | +0.3% | | 8192 | 4 | 32 | 302.2 | 302.4 | **+0.0%** | 447 | 444 | 12.44 | 12.44 | +0.0% | | 8192 | 8 | 64 | 473.4 | 471.4 | **-0.4%** | 666 | 636 | 15.10 | 15.27 | +1.1% | | 8192 | 16 | 96 | 665.3 | 668.7 | **+0.5%** | 825 | 826 | 21.44 | 21.33 | -0.5% | | 8192 | 32 | 128 | 949.8 | 947.2 | **-0.3%** | 1861 | 1770 | 29.40 | 29.59 | +0.6% | | 8192 | 64 | 256 | 1269.4 | 1270.4 | **+0.1%** | 2991 | 3077 | 44.80 | 44.52 | -0.6% | | 8192 | 128 | 384 | 1634.2 | 1641.3 | **+0.4%** | 6450 | 6423 | 68.91 | 68.55 | -0.5% | | 8192 | 256 | 768 | 1909.0 | 1917.4 | **+0.4%** | 14096 | 14072 | 116.63 | 115.82 | -0.7% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.1%** | -0.3% | -0.2% | | ISL 8192 | **+0.1%** | -1.1% | -0.0% | | **all 16** | **+0.1%** | -0.7% | -0.1% | The removed fill is on the prefill path, so the signal lands in TTFT rather than throughput: **-1.1% geomean at ISL 8192**, reaching -4.9% at concurrency 32 (1861 -> 1770 ms). Throughput and TPOT are flat, as expected for a change worth 0.89% of GPU kernel time in a decode-bound sweep; every case sits within +/-0.8% on throughput, i.e. inside run-to-run spread. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9575 | | gsm8k flexible-extract | 0.9507 | 0.9621 | | RULER niah_single_2 @16k | 0.870 | 0.838 | Unchanged. Both tasks run at the checkpoint's default sampling, so these deltas are inside run-to-run variation in both directions. ## Tests ```bash pytest tests/v1/attention/test_rocm_aiter_mla_sparse.py ``` The new test verifies that vLLM forwards `clean_logits=False` to AITER.", "url": "https://github.com/vllm-project/vllm/pull/51314", "createdAt": "2026-08-06T21:41:17Z", "updatedAt": "2026-08-13T07:53:15Z", "timestamp": "2026-08-13T07:53:15Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm" ], "author": "amd-sriram", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:79fac634b401ba150252", "signalId": "github:vllm-project/vllm:pull_request:51315", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51315", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fuse the DSA indexer prologue with AITER", "text": "## Summary The DSA indexer prologue uses four kernels for K normalization, Q/K RoPE, FP8 quantization, and the K-cache write. This PR uses AITER's fused `indexer_qk_rope_quant_and_cache` kernel instead. The fused path is enabled only for its supported BF16/FP8 shape; other configurations keep the existing path. ## Results MI355X, TP=4, rank 0: | indexer stage | before | after | change | |---|---:|---:|---:| | launches | 4 | 1 | **-75%** | | decode time | 17.4 µs | 5.9 µs | **-66%** | | prefill time | 163.9 µs | 270.6 µs | **+65%** | The fused kernel is faster for decode but slower for prefill. Across the full trace, prologue GPU time changes from 1,093.6 ms to 1,041.4 ms (**-4.8%**). One decode indexer stage at the same zoom: ![Before: four prologue kernels](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr3_before.png) ![After: one fused prologue kernel](https://raw.githubusercontent.com/amd-sriram/vllm/ce9490285bbd8099e2d37c5aec8b3d24b7a20785/figures/fig_pr3_after.png) End-to-end geomean: | metric | before | after | change | |---|---:|---:|---:| | throughput | 811.75 | 817.58 | **+0.72%** | | TPOT | 24.12 ms | 23.96 ms | **-0.66%** | | ITL | 23.85 ms | 23.69 ms | **-0.67%** | ## Model-level e2e results ### Test conditions | | | |---|---| | Hardware | MI355X (gfx950), TP=4 | | Model | `amd/GLM-5.2-MXFP4`, fp8 KV cache | | Stack | vLLM `69715823`, AITER `4a1cc77`, flydsl `0.3.0` | | Server | `--async-scheduling --max-num-seqs 256 --max-num-batched-tokens 16384 --block-size 64 --no-enable-prefix-caching`, `index_topk_freq=4` | | Env | `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_SPARSE_INDEXER=1`, `AITER_USE_FLYDSL_MOE_SORTING=1`, `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` | | Perf | `vllm bench serve`, random dataset, `--ignore-eos --request-rate inf`, ISL 1024 and 8192, OSL 1024, concurrency 1-256 (16 points) | | Accuracy | `lm_eval` gsm8k 5-shot chat template with thinking enabled, `max_gen_toks=8192`; RULER `niah_single_2` at 16k | | Sampling | checkpoint default (temperature 1.0, top_p 0.95), so scores carry roughly +/-0.006 gsm8k and +/-0.02 niah run-to-run | | Baseline | the same image with only this PR reverted | | tok/s | aggregate output tokens/s across the 4 GPUs; geomean rows are over the sweep | ### Serving sweep | ISL | conc | prompts | base tok/s | PR tok/s | Δ tput | base TTFT ms | PR TTFT ms | base TPOT ms | PR TPOT ms | Δ TPOT | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1024 | 1 | 16 | 81.3 | 82.2 | **+1.1%** | 1392 | 1401 | 10.80 | 10.66 | -1.3% | | 1024 | 4 | 32 | 330.5 | 334.0 | **+1.0%** | 131 | 132 | 11.66 | 11.54 | -1.0% | | 1024 | 8 | 64 | 562.4 | 570.2 | **+1.4%** | 142 | 143 | 13.08 | 12.90 | -1.4% | | 1024 | 16 | 96 | 875.0 | 884.8 | **+1.1%** | 211 | 211 | 16.44 | 16.29 | -0.9% | | 1024 | 32 | 128 | 1333.8 | 1345.3 | **+0.9%** | 333 | 340 | 20.58 | 20.41 | -0.8% | | 1024 | 64 | 256 | 2026.2 | 2037.0 | **+0.5%** | 442 | 443 | 26.96 | 26.83 | -0.5% | | 1024 | 128 | 384 | 2987.2 | 3011.6 | **+0.8%** | 986 | 989 | 35.19 | 34.90 | -0.8% | | 1024 | 256 | 768 | 4338.7 | 4353.5 | **+0.3%** | 1630 | 1628 | 50.59 | 50.52 | -0.1% | | 8192 | 1 | 16 | 84.5 | 85.3 | **+0.9%** | 339 | 338 | 11.50 | 11.39 | -1.0% | | 8192 | 4 | 32 | 302.2 | 304.6 | **+0.8%** | 447 | 445 | 12.44 | 12.34 | -0.8% | | 8192 | 8 | 64 | 473.4 | 478.1 | **+1.0%** | 666 | 666 | 15.10 | 14.95 | -1.0% | | 8192 | 16 | 96 | 665.3 | 669.9 | **+0.7%** | 825 | 824 | 21.44 | 21.34 | -0.5% | | 8192 | 32 | 128 | 949.8 | 951.5 | **+0.2%** | 1861 | 1798 | 29.40 | 29.42 | +0.1% | | 8192 | 64 | 256 | 1269.4 | 1274.2 | **+0.4%** | 2991 | 3148 | 44.80 | 44.47 | -0.7% | | 8192 | 128 | 384 | 1634.2 | 1640.1 | **+0.4%** | 6450 | 6414 | 68.91 | 68.69 | -0.3% | | 8192 | 256 | 768 | 1909.0 | 1908.9 | **-0.0%** | 14096 | 14005 | 116.63 | 116.91 | +0.2% | | geomean | tput | TTFT | TPOT | |---|---:|---:|---:| | ISL 1024 | **+0.9%** | +0.5% | -0.9% | | ISL 8192 | **+0.5%** | -0.0% | -0.5% | | **all 16** | **+0.7%** | +0.2% | -0.7% | Per-case expansion of the geomean above. The gain is larger at ISL 1024 (+0.9%) than at ISL 8192 (+0.5%), consistent with the fused kernel being faster for decode and slower for prefill. ### Accuracy | task | baseline | with PR | |---|---:|---:| | gsm8k strict-match | 0.9462 | 0.9530 | | gsm8k flexible-extract | 0.9507 | 0.9545 | | RULER niah_single_2 @16k | 0.870 | 0.852 | Unchanged within run-to-run variation. ## Tests ```bash pytest tests/kernels/attention/test_rocm_indexer_qk_rope_quant_cache.py ``` The test compares the fused and existing paths across both RoPE layouts, block sizes 16/64, and token counts 1/37/256.", "url": "https://github.com/vllm-project/vllm/pull/51315", "createdAt": "2026-08-06T21:41:29Z", "updatedAt": "2026-08-13T07:53:11Z", "timestamp": "2026-08-13T07:53:11Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "deepseek" ], "author": "amd-sriram", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e6aef4ea95976b55e13b", "signalId": "github:vllm-project/vllm:pull_request:48944", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48944", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Context-length-aware K in DSD (RFC #48627): extend num_speculative_tokens_per_batch_size with a ctx axis", "text": "## What this PR does Extends `num_speculative_tokens_per_batch_size` (Dynamic SD, #32374) with an optional ctx-length axis, so that a schedule can select K based on both batch size and per-request context length instead of batch size alone. The change is a backward-compatible schema extension — a 3-item entry `[bs_lo, bs_hi, K]` parses and behaves exactly as before, a 5-item entry `[bs_lo, bs_hi, ctx_lo, ctx_hi, K]` opts in to the ctx axis. ## Why Under memory-bound decode with a shared prefix (RAG serving, agentic sessions, batched summarization), the optimal K depends on ctx as well as batch. A batch-only lookup has to pick one K per BS, either leaving ctx-amortization on the table for long sequences or over-drafting short ones. This PR gives operators a per-(bs, ctx) K table that opts in to this axis when it's useful and stays inert (3-item form) otherwise. The concept was previously formalized by **MagicDec (Sadhukhan et al., arXiv:2408.11049, 2024)** — *\"Leveraging our observation that there is a sequence length threshold above which decoding becomes memory bound, we propose choosing the amount of speculation as a function of the sequence length (longer sequence length → more speculated tokens).\"* MagicDec's instantiation was self-speculation with StreamingLLM sparse KV in a batch-centric framing; this PR is the general draft-target integration into vLLM's DSD lookup surface. See RFC #48627 for the design discussion. ## Evidence (2026-07-27 measurement) Position-balanced 2-trial, `prefix_repetition` c=256, H100 NVL 94GB, `prithivMLmods/gemma-4-31B-it-qat-FP8` + assistant draft, vLLM `c5d967c23`. Full methodology + per-cell data in [decomposition comment](https://github.com/vllm-project/vllm/pull/48944#issuecomment-5091663057). Per-cell stdev <2%, order-bias max 1.72% (signal 29-36× larger). **Primary contrast — C′ (6-cell 2D schedule) vs A′ (3-item batch-only), same DSD-mode cost:** | ctx | A′ | C′ | C′/A′ | |-----:|---:|---:|:-:| | 400 | 1875.6 | 1890.7 | 1.01× (K=0 tier, tie by design) | | 900 | 1453.5 | 1874.6 | **1.29×** | | 1900 | 1416.6 | 1848.2 | **1.30×** | | 4000 | 1232.8 | 1680.4 | **1.36×** | **Zero cost when unused:** the original A/B comparison in the PR body (retracted headline, but this specific check reproduces) — 3-item vs 5-item-same-K on spec-bench aggregate: 2627.2 vs 2642.3 (+0.6%, within noise). Users who don't opt in to the ctx axis pay nothing. **Absolute crossover vs no-spec (measured, not projected):** above ctx ~2k the 2D schedule pays the full DSD-mode tax and still beats no-speculation in absolute throughput (ctx 1900: 1.02×, ctx 4000: 1.09× vs no_spec). This is the direct mechanism measurement. ## Regime guardrail (documented usage) **The ctx axis is intended to raise K for long-ctx buckets, not to lower K for short-ctx buckets in mixed traffic.** Using it in the latter direction (as an earlier iteration of my C schedule did) can regress aggregate throughput — the previous −4.0% spec-bench result was an anti-pattern usage plus the DSD-mode baseline tax at short ctx compounding. The docstring for the 5-item schema states this explicitly. ## Changes - `vllm/config/speculative.py` — 3/5-tuple type union widening, `ctx_agg: Literal[\"median\",\"mean\",\"max\"] = \"mean\"` field - `vllm/v1/spec_decode/dynamic/utils.py` — `DynamicSDLookup(dense, ctx_boundaries)` NamedTuple, rectangular grid validation, mixed-arity rejection, 1-indexed ctx range convention - `vllm/v1/core/sched/scheduler.py` — 2D dispatch (`dense[bs][ctx_bucket]`), decode-only pool filter for ctx representative (`num_computed_tokens >= num_prompt_tokens` with fallback to full pool), aggregator dispatch (median/mean/max) - `tests/v1/spec_decode/test_dynamic_sd.py` — 10 new tests for 5-item parsing, 2D routing, rectangular validation, capture-set K-invariance ## Known limitations - **DSD-mode baseline tax**: all DSD-mode arms (including this PR's schedule) pay a substantial throughput tax vs no-spec at short ctx (up to −31% at ctx=400 in the measurements above). The `PIECEWISE` cudagraph downgrade for DSD is one identified factor; a full decomposition (K=0 fast-path opportunity, drafter forward on K=0, admission cost, tier-boundary ramp) is filed as #49986. This is orthogonal to this PR — it applies to the original 3-item DSD API from #32374 too — but is worth noting because the ctx=400 tie in the C′/A′ table is a consequence of it (both arms fire K=0 at that cell, so both eat the same tax). - **Evidence range**: measured for ctx ∈ {400, 900, 1900, 4000}. Extension to 8k+ requires re-designing the batch tier to fit within achievable concurrency at longer ctx (per-request KV budget scales with ctx) — a separate measurement. - **Single model/drafter pair**: measured on gemma-4-31B-it-qat-FP8 + assistant. Generalization to other pairs (Llama-3.1-8B + EAGLE-3, Qwen, etc.) is a natural next validation. - **APC-high regime**: the measurement uses `prefix_repetition` (num_prefixes=1) which represents high-shared-prefix serving (RAG, agents, batched summarization at ~98% APC hit). Under low-APC-hit workloads the story is different — the gain is bounded by how much of the KV read the amortization can save. - **Aggregator choice**: `mean` is a defensible default (mean × B ≈ Σ ctx_i ≈ verify KV read cost — a cost signal). `median` is the majority-amortization signal and wins under specific batch-shape skew (documented in the `ctx_agg` field). Per-sequence K is out of scope for this PR (would require gathered-K verify batching + straggler control) and noted as a longer-term direction in RFC #48627 Alternatives. ## Roadmap (why this PR is worth landing as the schema base) Filed as #49986: 1. **K=0 fast-path** — when the schedule selects K=0 for a given (bs, ctx), route through the no-spec path entirely for that step. Target: recover C′ ctx=400 from 1890.7 → 2711.7 tok/s (+43%), which would make C′ strictly dominant across all four cells (tie-to-win vs no-spec, +26% over static-K3, +29-35% over batch-only). Separate PR, 2-3 weeks. 2. **K-keyed FULL cudagraph capture** — since the schedule enumerates the K set at startup, (batch bucket × K)-shape graphs are statically capturable, and the DSD-mode PIECEWISE downgrade can be lifted for declarative schedules. Follow-on to the fast-path. 3. **8k+ evidence extension**, **second model/drafter pair validation**, **per-bucket AR reporting** — natural follow-ups. ## Tests 32 unit + integration tests pass on H100 (driver 570, CUDA 12.8): 28 in `test_dynamic_sd.py` + 4 CUDA-graph regression. E2E verified on H100 (driver 580, CUDA 13.0) via the measurement setup above. ## References - vLLM RFC #48627 — Context-length-aware K in DSD (design + Prior Art including MagicDec anchor) - vLLM PR #32374 — original DSD API (ekagra-ranjan) - MagicDec (Sadhukhan et al., arXiv:2408.11049) - Follow-up: #49986 — DSD baseline tax decomposition - Companion: SGLang PR sgl-project/sglang#31716 (sibling 2D routing implementation) --- ## AI assistance This change was drafted with the help of an AI coding assistant (Anthropic Claude) and reviewed line-by-line before this PR was opened. All code, tests, commit message, and PR text were reviewed by me; the test commands above were run by me on this branch. Filing this disclosure per vLLM's `AGENTS.md` requirement for AI-assisted contributions.", "url": "https://github.com/vllm-project/vllm/pull/48944", "createdAt": "2026-07-17T11:40:09Z", "updatedAt": "2026-08-13T07:54:50Z", "timestamp": "2026-08-13T07:54:50Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "documentation", "speculative-decoding", "v1", "nvidia", "mrv2" ], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c6aa9e551582313dca85", "signalId": "github:vllm-project/vllm:pull_request:49585", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49585", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] Added Build Connector Worker Meta for EC Connector", "text": "## Purpose **Why this is still needed after #38390.** The PR implemented the V2 model runner EC Connector, but EC still has no worker -> scheduler metadata channel. KV connectors have a complete one: `build_connector_worker_meta()` -> `KVConnectorOutput.kv_connector_worker_meta` -> `KVOutputAggregator` -> scheduler-side KV connector. EC had none of those three pieces, so a worker-side`ECConnector` has no way to report per-step state — e.g. \"this `mm_hash` is now durably stored\" — back to its scheduler-side EC connector. That blocks the CPU-offload based on CUDA events and NIXL P2P EC connectors, which need exactly this report (We are retiring the StepTracker in the CPU EC Connector, because using CUDA events is the right way to handle \"copy done\" notifications, instead of waiting until the batch is done). This PR adds that channel, and fixes the exit paths where the worker's `ec_connector_output` was computed and then dropped before reaching the scheduler. ### The channel - `ec_connector/base.py`: `ECConnectorWorkerMetadata` ABC with an abstract `aggregate()`, plus `ECConnectorBase.build_connector_worker_meta()`. - `v1/outputs.py`: `ec_connector_worker_meta` field on `ECConnectorOutput`. - `ec_connector/utils.py`: `ECOutputAggregator`, mirroring `KVOutputAggregator` - merges the EC output of the rank that ran the connector onto the rank whose `ModelRunnerOutput` reaches the scheduler. - `engine/core.py`, `executor/abstract.py`, `executor/multiproc_executor.py`: wiring so the aggregator runs when an EC connector is configured. - Both runners (MRv1, MRv2) populate the field via `build_connector_worker_meta()`. Default behavior is unchanged for existing connectors: nothing overrides `build_connector_worker_meta()` yet, so it returns `None`. ## Test Plan ```bash # Existing EC connector unit tests (unaffected areas) .venv/bin/python -m pytest tests/v1/ec_connector/unit/test_metadata.py \\ tests/v1/ec_connector/unit/test_ec_example_connector.py \\ tests/v1/ec_connector/unit/cpu/test_connector.py -v ## Test Result ``` tests/v1/ec_connector/unit/test_metadata.py ..................... PASSED tests/v1/ec_connector/unit/test_ec_example_connector.py ......... PASSED tests/v1/ec_connector/unit/cpu/test_connector.py ................ PASSED 36 passed, 2 warnings pre-commit: ruff check / ruff format / mypy / SPDX headers / etc. — all Passed ```", "url": "https://github.com/vllm-project/vllm/pull/49585", "createdAt": "2026-07-23T12:33:50Z", "updatedAt": "2026-08-13T07:51:27Z", "timestamp": "2026-08-13T07:51:27Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready", "ci/build", "v1", "kv-connector", "mrv2" ], "author": "omerpaz95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b40446137246cff4a641", "signalId": "github:vllm-project/vllm:pull_request:50932", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "buffer size insuffient Dspark sd for FlashInfer MNNVL allreduce", "text": "fixes [#50877](https://github.com/vllm-project/vllm/issues/50877) ## Purpose FlashInferAllReduce.should_use_fi_ar gates on: self.max_num_tokens = max_workspace_size // (hidden_dim * element_size) max_workspace_size is the size of the whole MNNVL allocation (2 MB for TP8). But the MNNVL backend is Lamport-based and rotates through NUM_LAMPORT_BUFFERS=3 buffers, so only ~1/3 of the budget backs any single all-reduce: budget 2,097,152 bytes → gate says 146 tokens per-buffer 698,368 bytes → real capacity is 48 tokens Any tensor between 49–146 tokens passes the gate and then aborts inside the kernel. The drafter produces B × (1 + num_speculative_tokens) tokens per step — e.g. 14 requests × 4 = 56 tokens — landing exactly in this window. Without a drafter, decode batches produce B tokens and stay below 48. The same defect exists in fused_allreduce_gemma_rms_norm._can_use_flashinfer, which its docstring notes \"mirrors FlashInferAllReduce\". ## Fix After the existing budget bound (kept as a cheap pre-filter), call the workspace's own is_buffer_size_sufficient(tp_size, num_tokens, hidden_dim, dtype) method rather than reimplementing the Lamport arithmetic. This is on the abstract base class (flashinfer-python==0.6.15.post1, the pinned version), correct for both mnnvl and trtllm, and version-proof. ## What this does NOT fix AllReduceFusionPass has the same too-permissive bound but self.max_token_num there also drives the workspace allocation request — fixing it requires separating those two concerns and verifying the one-shot/two-shot split, which needs GB200 hardware. Leaving that for a follow-up. ## Workaround (no code change needed) VLLM_ALLREDUCE_USE_FLASHINFER=0 Disables the eager FlashInferAllReduce path; the compiled fusion pass is unaffected. ## Test Plan I have added test file and you can see output below, every assertion in the capacity-accounting logic changed ran and succeeded. pytest tests/compile/passes/distributed/test_fusion_all_reduce.py -v , already existing test. On GB200 python -m vllm.entrypoints.openai.api_server \\ --model --speculative-config '{\"model\": \"\"}' --tensor-parallel-size 8 if reporter can test on GB200 that would be perfect. ## Test Result Cannot test as I do not have GB200, if @ilmarkov can test it, that would be best. ## AI assistance This fix was developed with Claude (claude-opus-4-6) assistance. I have reviewed every changed line and understand the root cause and the design. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/50932", "createdAt": "2026-08-03T23:46:19Z", "updatedAt": "2026-08-13T07:47:53Z", "timestamp": "2026-08-13T07:47:53Z", "metrics": { "reactions": 2, "comments": 19 }, "labels": [ "bug", "ready", "nvidia" ], "author": "khushali9", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3b07f8fb7e032fe85f8a", "signalId": "github:vllm-project/vllm:pull_request:52121", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52121", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fall back to native rotary when flash-attn is installed but fails to load", "text": "## Purpose `ApplyRotaryEmb.__init__` guards the optional flash-attn rotary import with `suppress(ModuleNotFoundError)`. That covers a *missing* module, but not an *installed but unloadable* one: when the `flash_attn` C extension cannot be loaded, Python raises `ImportError` — the parent class — which escapes the narrower guard and aborts engine startup during model loading. `ApplyRotaryEmb` is constructed by `RotaryEmbedding.__init__` (`rotary_embedding/base.py`), so this guard runs when loading essentially any model that uses RoPE — not just the VL models that call it directly. A broken optional dependency therefore takes down startup for everything. The common trigger is mundane: rebuild or upgrade PyTorch without rebuilding flash-attn. `flash_attn.ops.triton.rotary` resolves fine, then loading `flash_attn_2_cuda` fails on a moved/removed `c10` symbol: ``` ImportError: .../flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE ``` This PR distinguishes the two cases rather than widening the guard blindly: - `ModuleNotFoundError` — flash-attn simply is not installed. Stay silent; this is the normal, expected path and its behaviour is unchanged. - any other `ImportError` — flash-attn is installed but broken. Fall back to native **and** emit `logger.warning_once`, so the user learns their flash-attn is unusable instead of silently running the slow path. A plain `suppress(ImportError)` would fix the crash but hide a genuinely broken install, which I believe is why the narrow exception type was requested in #42679. ## Why this is not a duplicate - **#42679** (merged) introduced the current `suppress(ModuleNotFoundError)`. It fixed FA4 environments where `flash_attn.ops.triton.rotary` is *absent*. It does not cover an import that fails during extension loading. - **#36900** (open since March) targets the same lines, but the failure it describes — `ModuleNotFoundError: No module named 'flash_attn.ops'` with `flash-attn-4` installed — is already handled by #42679's `suppress(ModuleNotFoundError)`. Its diff is also based on the older `find_spec` code that #42679 replaced, so it no longer applies. I believe #36900 can be closed as obsolete; the `ImportError` case is the only part still unaddressed. - **#43684** (merged) added a separate native fallback in this same method for a ROCm grid-limit issue, so falling back to native here is established precedent rather than new policy. I searched open/closed/merged PRs for `rotary_embedding common.py`, `flash_attn.ops.triton.rotary`, `ApplyRotaryEmb`, `suppress(ModuleNotFoundError)` and `undefined symbol` before opening this. ## Test plan and results Hardware: RTX PRO 5000 Blackwell (sm_120), CUDA 13.2, hand-built PyTorch 2.13.0, vLLM built from source. **1. Branch behaviour** — all three paths verified by patching `import_module` in the module under test (needs `set_current_vllm_config`, or the `default_vllm_config` fixture in `tests/conftest.py`): | case | expected | result | |---|---|---| | flash-attn working | fast path selected | `apply_rotary` | | `ImportError` (ABI mismatch) | warn once + fall back | fell back to `None`, one warning | | `ModuleNotFoundError` (absent) | **silent** + fall back | fell back to `None`, no warning | | repeated construction | no log spam | deduplicated by `warning_once` | The third row is the behaviour #42679 deliberately preserved; it is unchanged here. **2. Original failure** — reproduced on this machine with a genuinely ABI-mismatched flash-attn build (2.8.3.post1 compiled against an earlier torch). On unpatched code `vllm serve` aborts during model loading with the `undefined symbol` traceback above and never becomes ready. With the guard widened, the engine starts and serves normally through the native rotary path — verified with a real workload (Qwen3.6-27B-FP8 + DFlash speculative decoding, `--max-num-seqs 3`, `--max-model-len 112000`): 122.7 tok/s and 7.91 ms TPOT single stream, 230.5 tok/s aggregate at concurrency 3. **3. End-to-end with this exact commit** — after rebuilding flash-attn against the current torch, `vllm serve` starts cleanly, `/health` returns 200, no warning is emitted (fast path taken, as expected), and generation is correct with speculative decoding active (accept rate 18.2% at `num_speculative_tokens=16`). No numerical change: this only selects which rotary implementation is used when flash-attn cannot load, and the fallback is the same native path used on every machine without flash-attn. **4. Lint** ```bash pre-commit run --files vllm/model_executor/layers/rotary_embedding/common.py ``` All hooks pass (ruff check, ruff format, typos, mypy 3.10, SPDX, forbidden imports). ## AI assistance disclosure AI assistance (Claude Code) was used to diagnose the failure, confirm the exception hierarchy and the `nm`-level symbol analysis, run the verification above, and draft this description. I reviewed every changed line, reproduced both the failure and the fix on my own hardware, and can defend the change end to end.", "url": "https://github.com/vllm-project/vllm/pull/52121", "createdAt": "2026-08-13T07:46:09Z", "updatedAt": "2026-08-13T07:46:45Z", "timestamp": "2026-08-13T07:46:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "bheu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:67f69c01103eb6a0fb05", "signalId": "github:vllm-project/vllm:pull_request:49287", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49287", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][UT] Fix OOM and skip graph case", "text": "Fix UT on XPU: 1. test_mamba_cache_cg_padding can only run with graph enabled. Otherwise, UT will fall into an infinite loop. 2. To address the issue of insufficient VRAM caused by consecutive use cases (where there is insufficient free memory upon engine startup), the approach is to reuse the existing ROCm VRAM waiting mechanism and provide an appropriate default `gpu_memory_utilization` for the XPU.", "url": "https://github.com/vllm-project/vllm/pull/49287", "createdAt": "2026-07-21T06:24:20Z", "updatedAt": "2026-08-13T07:46:10Z", "timestamp": "2026-08-13T07:46:10Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "intel-gpu", "multi-modality", "kv-connector", "mistral" ], "author": "mayuyuace", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:749a491bed4c098f970b", "signalId": "github:vllm-project/vllm:pull_request:52046", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52046", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][NVIDIA] Add PCP support to DeepSeek V3.2 attention", "text": "## Summary - add prefill context parallelism (PCP) to the existing DeepSeek V3.2 attention implementation - keep TP and PCP on the same fused attention path instead of maintaining a separate PCP forward - materialize fused MLA and indexer K outputs only when PCP needs to gather them before cache insertion - specialize optional materialization and cache writes directly from nullable Triton pointers, without separate compile-time mode flags - preserve dense-MHA dispatch for short prefill while using the sparse-MQA path for long prefill and decode - combine PCP attention outputs with the existing LSE-aware reduction and finalization helpers ## Scope and dependencies This PR contains only generic DeepSeek V3.2 PCP support. It does not change model routing, registries, speculative-decoding model selection, or GLM-specific GEMM behavior. GLM-5.2 end-to-end validation was performed with #49790 applied, because that PR routes GLM-5.2 and its MTP model to the DeepSeek V3.2 implementation. Once #49790 lands, GLM-5.2 can use the PCP support added here without any GLM-specific code in this PR. ## Non-duplication - #49790 supplies GLM-5.2-to-DeepSeek-V3.2 routing; this PR supplies PCP inside the DeepSeek V3.2 attention implementation. - #50005 fixes DCP correctness in the older NVIDIA fused override, including query gathering and output/LSE merging. This PR targets PCP in the current common DeepSeek V3.2 path and additionally handles PCP K materialization, cache gathering/insertion, and short-prefill dense-MHA dispatch. - #46178 addresses generic sparse-attention DCP plumbing rather than this DeepSeek V3.2 PCP path. ## Validation Current PCP-only head: `86793bce26` - touched-file pre-commit suite: passed, including Ruff, formatting, mypy, SPDX, forbidden-import, and accelerator-API checks - `pytest tests/models/deepseek_v32/test_pcp_dispatch.py -q`: 2 passed - B300 fused-kernel suite on the optional-output implementation: 57 passed - GB200 nullable-cache and nullable-`q_pe_out` validation: 29 fused norm/RoPE/cache cases and 24 Triton fused-Q cases passed - 8xB300 GLM-5.2 NVFP4 PCP8/EP8 smoke with #49790 routing: startup, warmup, and requests completed with zero failures - 8xB300 PCP8/EP8 prefill-only profile with #49790 routing, FP8 KV cache, 32K max batched tokens, and 16K long-prefill threshold: 32/32 requests completed, 1,399,021 input tokens in 9.48 seconds, 147.58K input tokens/s; full-32K forward p50 was 366.9 ms No model-quality evaluation has been run on this revision, so the PR remains draft. ## AI assistance and human review OpenAI Codex assisted with investigation, implementation, validation orchestration, and drafting this description. The human submitter must review every changed line and be able to explain and defend the PCP cache-gather and LSE-combine protocol before marking this ready for review.", "url": "https://github.com/vllm-project/vllm/pull/52046", "createdAt": "2026-08-12T21:00:58Z", "updatedAt": "2026-08-13T07:46:07Z", "timestamp": "2026-08-13T07:46:07Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "new-model", "speculative-decoding", "deepseek", "nvidia" ], "author": "GirasoleY", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8c26e18110b85166ac7e", "signalId": "github:vllm-project/vllm:pull_request:51882", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51882", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Remove NIXL reinstall step", "text": "## Purpose Remove the NIXL wheel reinstall step. It is not longer needed (NIXL fixed the dependency issues upstream) and it is not working correctly since it does not use the version pin. Currently it causes nixl-cu13 to float to 1.3.2 while nixl/nixl-cu12 stays on the kv_connectors.txt pin (nixl == 1.3.1). ## Test Plan Verified by simulating the Dockerfile install path in a clean venv. ## Test Result **Old path** (`nixl==1.3.1` then unpinned `--force-reinstall --no-deps nixl-cu13`): ``` nixl 1.3.1 nixl-cu12 1.3.1 nixl-cu13 1.3.2 ← mismatch (same bug as Dynamo nightly) ``` **With fix** (pin only, no force-reinstall): ``` nixl 1.3.1 nixl-cu12 1.3.1 nixl-cu13 1.3.1 PASS: all nixl* == 1.3.1 ``` Removing that line is enough; the `kv_connectors.txt` pin keeps all three aligned. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51882", "createdAt": "2026-08-11T22:22:49Z", "updatedAt": "2026-08-13T07:45:31Z", "timestamp": "2026-08-13T07:45:31Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "documentation", "ready", "ci/build", "kv-connector" ], "author": "ovidiusm", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:d14a34118002ea39e77b", "signalId": "github:vllm-project/vllm:pull_request:51910", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51910", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization][CT] fix the mxfp4 inference for MiniMax-M3 with CT format.", "text": "## Description Running **MiniMax-M3** MXFP4 checkpoints quantized with the **compressed-tensors (CT)** `mixed-precision` format currently fails at model load with a `KeyError` on the vision tower. In these checkpoints the **vision module is not quantized** (it is listed in the CT `ignore` list and stored as plain `.weight`), but vLLM wrongly treats the vision layers as quantized and then fails to find the packed weights. This PR contains two small, targeted fixes that make MXFP4 CT inference work, without changing behavior for the bf16 / mxfp8 paths. ## Root cause & fix ### 1. `vllm/model_executor/models/minimax_m3/nvidia/model.py` — vision `hf_to_vllm_mapper` The CT config's `ignore` list is now transformed through the model's `hf_to_vllm_mapper` by `CompressedTensorsConfig.apply_vllm_mapper()` (it calls `_map_name` on every `self.ignore` entry so ignore targets line up with vLLM's internal layer names). The catch: **`ignore` entries are module names without a trailing dot** (e.g. `...encoder.layers.0.mlp.fc1`), whereas the existing substitutions had trailing dots (`.mlp.fc1.` → `.fc1.`). `_map_name` is a substring replace, so: - `.mlp.fc1.` does **not** match the dot-less ignore name `...mlp.fc1` → the ignore target is left untransformed → it never matches vLLM's internal prefix `...fc1` → the (unquantized) vision layer is built as a quantized layer → `KeyError: '...fc1.weight'` at load. The fix drops the trailing dots so the mapping matches both module names and weight names: ```python orig_to_new_substr={ - \".mlp.fc1.\": \".fc1.\", - \".mlp.fc2.\": \".fc2.\", + \".mlp.fc1\": \".fc1\", + \".mlp.fc2\": \".fc2\", }, ``` **Why this is safe for weight loading (bf16/mxfp8 unaffected):** for actual weight names (`...mlp.fc1.weight`), both the old and new substitutions produce the identical result `...fc1.weight`. The change only *additionally* fixes the dot-less ignore-module-name case. It is a strict superset of the previous behavior. *(Evidence that vision is genuinely unquantized in the checkpoint: in the MXFP4 model index, the vision tower has 0 `weight_packed` tensors — all packed weights live under `language_model...block_sparse_moe.experts` — and 258 vision entries appear in the CT `ignore` list.)* ### 2. `vllm/compilation/passes/fusion/allreduce_rms_fusion.py` — flashinfer `weight_bias` version compat MiniMax-M3 uses a GemmaRMSNorm that scales by `(1 + weight)`, carried by the fused all-reduce kernel's `weight_bias=1.0`. The `weight_bias` argument to `flashinfer.comm.trtllm_allreduce_fusion` was added in **flashinfer 0.6.12**; older flashinfer (e.g. 0.6.11) does not accept it, so unconditionally passing it raises `TypeError`. The fix detects support once at import via `inspect.signature` (guarded by `try/except (TypeError, ValueError)` for signature-less C callables) and: - passes `weight_bias=...` when supported, or - folds the bias into `rms_gamma` (`normed * (gamma + bias)`, mathematically equivalent) with a `warning_once` when not. This path is only exercised under TP + flashinfer all-reduce/RMSNorm fusion and is **quantization-agnostic**. It can be removed once the minimum flashinfer version is ≥ 0.6.12. **Prior art:** the same \"probe the installed flashinfer signature and only pass supported kwargs\" approach was used in SGLang [PR #26197](https://github.com/sgl-project/sglang/pull/26197) to keep the Blackwell all-reduce-fusion path enabled across flashinfer API variants.", "url": "https://github.com/vllm-project/vllm/pull/51910", "createdAt": "2026-08-12T02:20:51Z", "updatedAt": "2026-08-13T07:39:15Z", "timestamp": "2026-08-13T07:39:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization" ], "author": "lkk12014402", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:56171c26037ef3a82296", "signalId": "github:vllm-project/vllm:pull_request:42785", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42785", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MM][CG] Enable encoder CUDA Graph for MiniCPM-V", "text": "## Purpose Add encoder CUDA Graph support for MiniCPM-V 2.5, 2.6, 4.0 as part of tracker #38175. This implementation follows the existing workflow introduced in #38061. The captured graph covers both the ViT encoder (VPM) and the resampler. MiniCPM-V 2.0 is not included, as it predates the slice-based vision architecture required by this implementation. MiniCPM‑V 4.5 is not included, as its dynamic frame fusion introduces input-dependent encoder shapes that are not compatible with the current static-shape CUDA Graph capture mechanism. It continues to use the eager path and does not benefit from encoder CUDA Graph replay, which may reduce performance. ## Key Updates This PR extends the multimodal encoder CUDA Graph cache to key captured graphs by` (token_budget, secondary_capture_axis_key)` instead of `token_budget` alone. `token_budget` continues to determine greedy packing and output buffer sizing for resampler tokens. However, for vision encoders such as those in OpenBMB's MiniCPM-V models, the encoder capture shape also depends on discrete patch-grid / pixel-layout tiers. As a result, different input shapes can share the same token budget while requiring distinct CUDA Graph captures. Without this secondary capture axis, inputs with the same token budget are forced into a single capture shape, which can introduce unnecessary padding and reduce the efficiency benefits of CUDA Graphs. By including `secondary_capture_axis_key `in the cache key, the encoder can maintain separate captures for shape-dependent tiers while preserving the existing token-budget-based scheduling and memory allocation logic. ## Test Plan **Unit test** ```bash pytest tests/v1/cudagraph/test_encoder_cudagraph.py -v ``` **Functional test** GPU: RTX 5090 Model: MiniCPM-V-4_0 No CUDA Graph ```bash vllm serve /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --served-model-name MiniCPM-V-4 \\ --gpu-memory-utilization 0.75 \\ --max-model-len 4096 \\ --max-num-batched-tokens 4096 \\ --limit-mm-per-prompt '{\"video\": 1, \"image\": 1}' ``` With CUDA Graph ```bash vllm serve /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --served-model-name MiniCPM-V-4 \\ --gpu-memory-utilization 0.75 \\ --max-model-len 4096 \\ --max-num-batched-tokens 4096 \\ --limit-mm-per-prompt '{\"video\": 1, \"image\": 1}' \\ --compilation-config '{ \"cudagraph_mm_encoder\": true, \"encoder_cudagraph_token_budgets\": [1024], \"encoder_cudagraph_max_vision_items_per_batch\": 32 }' ``` **benchmark** GPU: RTX 5090 Model: MiniCPM-V-2_6 / MiniCPM-V-4_0 ```bash vllm bench mm-processor \\ --model /root/autodl-tmp/huggingface/hub/MiniCPM-V-4/OpenBMB/MiniCPM-V-4 \\ --trust-remote-code \\ --tokenizer-mode slow \\ --dataset-name random-mm \\ --num-prompts 500 \\ --num-warmups 100 \\ --max-model-len 4096 \\ --seed 42 \\ --gpu-memory-utilization 0.8 \\ --random-mm-base-items-per-request 8 \\ --random-mm-num-mm-items-range-ratio 0.0 \\ --random-mm-bucket-config '{\"(448, 448, 1)\": 1.0}' \\ --compilation-config '{ \"cudagraph_mm_encoder\": true, \"encoder_cudagraph_token_budgets\": [256, 512], \"encoder_cudagraph_max_vision_items_per_batch\": 8 }' ``` ## Test Result ✅ **Unit test** ```bash 36 passed, 16 warnings in 7.11s ``` ✅**Functional test** Image ```bash # No CUDA Graph -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying an image of a ship and some text in Chinese. -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying what appears to be a news broadcast about an aircraft carrier. -------------------------------------------------- # With CUDA Graph -------------------------------------------------- A woman is standing in a room with a television screen in the background displaying an image of a ship and some text in Chinese. -------------------------------------------------- A woman in a green t-shirt and denim shorts standing in a room with a television screen in the background displaying what appears to be a news broadcast about an aircraft carrier. -------------------------------------------------- ``` Video ```bash # No CUDA Graph -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with a mix of dark and light clouds, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, creating a bright line against the dimly lit sky. The reflection of the train's lights shimmers on the water's surface below, adding to the tranquil atmosphere. As the train progresses, the background reveals faint outlines of industrial structures, possibly factories or power plants, with their silhouettes barely visible against the horizon. The overall ambiance is calm and quiet, with the train's movement being the primary action in the scene. -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with varying shades of gray, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, casting a reflection on the water below. As the train progresses, the lights on the train create a shimmering effect on the water's surface. In the background, faint outlines of industrial structures can be seen, adding an urban element to the otherwise natural landscape. The overall ambiance of the video is calm and tranquil, with the train's movement providing a subtle dynamic element to the scene. -------------------------------------------------- # With CUDA Graph -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with a mix of dark and light clouds, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, creating a bright line against the dimly lit sky. The reflection of the train's lights shimmers on the water's surface below, adding to the tranquil atmosphere. As the train progresses, the background reveals faint outlines of industrial structures, possibly factories or power plants, with their silhouettes barely visible against the horizon. The overall ambiance is calm and quiet, with the train's movement being the primary action in the scene. -------------------------------------------------- The video captures a serene evening scene featuring a train crossing a bridge over a body of water. The sky is overcast with varying shades of gray, suggesting it might be either dawn or dusk. The train, illuminated by its own lights, moves steadily across the bridge, casting a reflection on the water below. As the train progresses, the lights on the train create a shimmering effect on the water's surface. In the background, faint outlines of industrial structures can be seen, adding an urban element to the otherwise natural landscape. The overall ambiance of the video is calm and tranquil, with the train's movement providing a subtle dynamic element to the scene. -------------------------------------------------- ``` ✅ **Benchmark:** > Single RTX 5090 · random-mm · 500 prompts / 100 warmups Benchmark results for MiniCPM-V 2.6 and 4.0 show no measurable regression. Latency remains within normal run-to-run variance, with small improvements observed in some runs. | Version | Backend | Mean | P99 | |------|------|------|------| | MiniCPM-V-2_6 | FLASH_ATTN2 | +0.60% (44555.14→44286.37ms) | +0.10% (67712.77ms→67644.05ms) | | MiniCPM-V-4 | FLASH_ATTN2 | +2.01% (44723.41→43822.66ms) | +2.69% (84580.66ms→82305.93ms) | **MiniCPM-V-2_6** No CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.03 0.39 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.16 29.88 1.43 35.51 merge_mm_kwargs_ms 0.44 0.41 0.08 0.63 apply_prompt_updates_ms 6.15 6.39 0.74 7.54 preprocessor_total_ms 37.13 36.90 1.77 42.96 encoder_forward_ms 64.14 63.93 14.47 111.22 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 544 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 44723.41 Median 43812.82 Std 13619.35 P99.0 84580.66 ``` With CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.02 0.38 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.12 29.93 1.31 34.74 merge_mm_kwargs_ms 0.44 0.41 0.06 0.55 apply_prompt_updates_ms 6.16 6.40 0.73 7.38 preprocessor_total_ms 37.09 36.92 1.55 43.03 encoder_forward_ms 62.67 61.63 14.23 113.54 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 543 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 43822.66 Median 42867.49 Std 13261.40 P99.0 82305.93 ``` **MiniCPM-V-4_0** No CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.03 0.39 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.16 29.88 1.43 35.51 merge_mm_kwargs_ms 0.44 0.41 0.08 0.63 apply_prompt_updates_ms 6.15 6.39 0.74 7.54 preprocessor_total_ms 37.13 36.90 1.77 42.96 encoder_forward_ms 64.14 63.93 14.47 111.22 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 544 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 44723.41 Median 43812.82 Std 13619.35 P99.0 84580.66 ``` With CUDA Graph: ```bash ================================================================================ Multimodal Processor Benchmark Results ================================================================================ MM Processor Metrics: Stage Mean Median Std P99.0 get_mm_hashes_ms 0.35 0.35 0.02 0.38 get_cache_missing_items_ms 0.02 0.02 0.00 0.03 apply_hf_processor_ms 30.12 29.93 1.31 34.74 merge_mm_kwargs_ms 0.44 0.41 0.06 0.55 apply_prompt_updates_ms 6.16 6.40 0.73 7.38 preprocessor_total_ms 37.09 36.92 1.55 43.03 encoder_forward_ms 62.67 61.63 14.23 113.54 num_encoder_calls 1.09 1.00 0.28 2.00 Summary: 543 total encoder calls across 500 requests. End-to-End Latency (ms): Metric Value (ms) Mean 43822.66 Median 42867.49 Std 13261.40 P99.0 82305.93 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42785", "createdAt": "2026-05-15T21:19:04Z", "updatedAt": "2026-08-13T07:37:22Z", "timestamp": "2026-08-13T07:37:22Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "documentation", "v1", "multi-modality", "llama", "qwen", "deepseek", "nvidia", "kimi" ], "author": "YunzhuLu", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:a33665628283f123d17f", "signalId": "github:vllm-project/vllm:issue:38175", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:38175", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Support ViT Full CUDA Graph (Tracker)", "text": "### Motivation. Multimodal large language models (e.g., Qwen3-VL, Qwen3.5, GLM-V, Kimi K2.5) rely on a Vision Transformer (ViT) encoder to process visual inputs before feeding them into the language model backbone. In production serving scenarios, the ViT forward pass involves launching a large number of small CUDA kernels — including patch embedding, layer normalization, multi-head self-attention, and MLP projections — each of which incurs non-trivial kernel launch overhead on the host side. Currently, vLLM supports CUDA graph capture for the decoder (LLM) portion of the model, which has proven effective at reducing kernel launch costs and improving throughput. However, the ViT encoder is still executed eagerly, meaning every forward pass re-launches all kernels from scratch. Extending full CUDA graph support to the ViT encoder would allow the entire encoder forward pass to be captured and replayed as a single graph, eliminating per-kernel launch overhead and enabling more consistent, low-latency inference for multimodal models. ### Proposed Change. **Model Integration:** - [x] https://github.com/vllm-project/vllm/pull/35963 @b-mu - [x] https://github.com/vllm-project/vllm/pull/41736 @johncalesp - [x] https://github.com/vllm-project/vllm/pull/40830 @johncalesp - [x] https://github.com/vllm-project/vllm/pull/38061 @shen-shanshan - [x] https://github.com/vllm-project/vllm/pull/42151 @shen-shanshan - [x] https://github.com/vllm-project/vllm/pull/44720 @harsha20032020 - [x] https://github.com/vllm-project/vllm/pull/42224 @JisoLya - [x] https://github.com/vllm-project/vllm/pull/40576 @grYe99 - [x] https://github.com/vllm-project/vllm/pull/40660 @allgather - [x] https://github.com/vllm-project/vllm/pull/41759 @oguzhankir - [x] https://github.com/vllm-project/vllm/pull/41992 @oguzhankir - [x] https://github.com/vllm-project/vllm/pull/50929 @lk-chen - [ ] https://github.com/vllm-project/vllm/pull/42785 @YunzhuLu - [x] https://github.com/vllm-project/vllm/pull/43591 @JisoLya - [x] https://github.com/vllm-project/vllm/pull/46837 @anthonsu - [x] https://github.com/vllm-project/vllm/pull/43586 @shen-shanshan - [ ] https://github.com/vllm-project/vllm/pull/46005 @littlecircle0730 - [ ] https://github.com/vllm-project/vllm/pull/44394 @harsha20032020 - [x] https://github.com/vllm-project/vllm/pull/45254 @qyYue1389 - [ ] https://github.com/vllm-project/vllm/pull/51484 @gangula-karthik > [!NOTE] > **Integration Workflow:** > > 1. Implement ViT CUDA graph interface for the model referring to Qwen3-VL. > 2. Do tests: ut/e2e/benchmark/... > 3. Update supported model list in the [doc](https://docs.vllm.ai/en/latest/design/cuda_graphs_multimodal/#model-integration-via-supportsencodercudagraph). > 4. Add this model to CI test. **Bugfix / Improvement:** - [x] https://github.com/vllm-project/vllm/pull/38040 - [x] https://github.com/vllm-project/vllm/pull/38116 - [x] https://github.com/vllm-project/vllm/pull/40445 - [x] https://github.com/vllm-project/vllm/pull/40580 - [x] https://github.com/vllm-project/vllm/pull/41234 - [x] https://github.com/vllm-project/vllm/pull/41714 - [x] https://github.com/vllm-project/vllm/pull/42288 - [x] https://github.com/vllm-project/vllm/pull/42796 - [x] https://github.com/vllm-project/vllm/pull/43082 - [x] https://github.com/vllm-project/vllm/pull/43321 - [ ] https://github.com/vllm-project/vllm/pull/43403 - [x] https://github.com/vllm-project/vllm/pull/44484 - [x] https://github.com/vllm-project/vllm/pull/45654 - [ ] https://github.com/vllm-project/vllm/pull/45886 - [x] https://github.com/vllm-project/vllm/pull/46034 - [x] https://github.com/vllm-project/vllm/pull/49934 - [x] https://github.com/vllm-project/vllm/pull/51461 **Testing Coverage:** - [x] https://github.com/vllm-project/vllm/pull/40780 **Documentation:** - [x] https://github.com/vllm-project/vllm/pull/37914 - [x] https://github.com/vllm-project/vllm/pull/40355 - [x] https://github.com/vllm-project/vllm/pull/44388 ### Feedback Period. _No response_ ### CC List. @ywang96 @Isotr0py @wangshangsam ### Any Other Things. _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38175", "createdAt": "2026-03-26T02:22:01Z", "updatedAt": "2026-08-13T07:36:09Z", "timestamp": "2026-08-13T07:36:09Z", "metrics": { "reactions": 1, "comments": 27 }, "labels": [ "help wanted", "RFC", "multi-modality", "kimi" ], "author": "shen-shanshan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:23e2e8c11477cf55d1dd", "signalId": "github:vllm-project/vllm:pull_request:52119", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52119", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Kill regex compilation subprocess on timeout", "text": "Replace the thread-based compile_regex_with_timeout() with a fork-based subprocess that is SIGKILL'd on deadline expiry. The previous ThreadPoolExecutor approach could not cancel already-running callables, allowing adversarial regex patterns to accumulate CPU/memory-consuming compiler workers after each timeout error. - Use multiprocessing fork context for fast subprocess creation - SIGKILL + join the child before returning ValueError on timeout - Add global semaphore (VLLM_REGEX_COMPILATION_MAX_CONCURRENT, default 1) to bound concurrent compilation processes - Adapt xgrammar/outlines/lm-format-enforcer call sites to use picklable top-level workers with serialize/deserialize round-trips - Add regression tests asserting zero lingering processes after timeout", "url": "https://github.com/vllm-project/vllm/pull/52119", "createdAt": "2026-08-13T07:34:38Z", "updatedAt": "2026-08-13T07:35:14Z", "timestamp": "2026-08-13T07:35:14Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "structured-output" ], "author": "jperezdealgaba", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bd6b8b4d36b5e3620504", "signalId": "github:vllm-project/vllm:issue:52069", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52069", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MTP speculative decoding cannot start under pipeline parallelism — SupportsPP demanded of the draft model", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Also reproduced on `dev678` and `dev686`. Code references below are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug Speculative decoding with an MTP head cannot start at all under pipeline parallelism. It fails during `create_engine_config`, before any weight is loaded: ``` vllm serve --pipeline-parallel-size 4 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ``` ``` NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the `SupportsPP` interface. ``` Traceback path: ``` vllm/config/speculative.py _verify_args -> self.draft_model_config.verify_with_parallel_config(self.draft_parallel_config) vllm/config/model.py verify_with_parallel_config ``` `speculative.py` L1411-1414 on main: ```python if self.draft_model_config: self.draft_model_config.verify_with_parallel_config( self.draft_parallel_config ) ``` ### Why the check does not apply to a draft model `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so verification then demands `SupportsPP` of the draft. But the drafter is only ever built on one rank — `v1/worker/gpu/model_runner.py`: ```python if self.speculative_config is not None: if self.is_last_pp_rank: self.speculator = init_speculator(self.vllm_config, self.device) ``` A draft model is therefore never split across pipeline stages, so whether it *could* be split is not a question that needs answering about it. ### Scope This is not specific to one model. `Glm4MoeMTP` and `DeepSeekMTP` are declared as `(nn.Module, <...>MixtureOfExperts)` — neither inherits `SupportsPP`, and neither do the other ~20 entries of `MTPModelTypes` (`deepseek_mtp`, `glm4_moe_mtp`, `ernie_mtp`, `qwen3_next_mtp`, `minimax_m3_mtp`, `longcat_flash_mtp`, …). Every MTP head is blocked under `pipeline_parallel_size > 1` by this check. ### Suggested fix Verify the draft as the single-stage model it actually is, and restore the config afterwards so rank and world-size bookkeeping downstream is untouched: ```python if self.draft_model_config: _dpc = self.draft_parallel_config if _dpc.pipeline_parallel_size > 1: _pp = _dpc.pipeline_parallel_size object.__setattr__(_dpc, \"pipeline_parallel_size\", 1) try: self.draft_model_config.verify_with_parallel_config(_dpc) finally: object.__setattr__(_dpc, \"pipeline_parallel_size\", _pp) else: self.draft_model_config.verify_with_parallel_config(_dpc) ``` ### Note on what comes next With this relaxed, MTP under PP gets further but does not yet work: the last rank then dies inside `compile_or_warm_up_model` → `v1/worker/gpu/warmup.py::warmup_kernels` → `_run_decode_step` → `get_pp_group().irecv_tensor_dict(...)`, and after that the output is wrong unless async scheduling is enabled. The scheduling half is filed separately; I am happy to open a third issue for the warmup receive if that is useful rather than noise. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52069", "createdAt": "2026-08-12T23:52:36Z", "updatedAt": "2026-08-13T07:22:14Z", "timestamp": "2026-08-13T07:22:14Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d3cfb8f7733b9e9a0655", "signalId": "github:vllm-project/vllm:pull_request:52116", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52116", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fix DualChunkRotaryEmbedding hard-coded cuda device crash", "text": "## Purpose `DualChunkRotaryEmbedding.__init__` hard-codes `self.device = torch.device(f\"cuda:{device_idx}\")` alongside the portable `torch.accelerator.current_device_index()`. On any non-CUDA backend the subsequent `.to(device=self.device)` inside `_compute_cos_sin_cache` triggers CUDA lazy-init and crashes model loading with \"Torch not compiled with CUDA enabled\" for every Dual Chunk Attention model (e.g. Qwen2.5-*-1M). Reproduced on Ascend NPU when serving Qwen2.5-7B-Instruct-1M: ``` (EngineCore) Starting to load model .../Qwen2.5-7B-Instruct-1M... (EngineCore) EngineCore failed to start. Traceback (most recent call last): File \".../vllm/v1/engine/core.py\", line 1200, in run_engine_core engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) File \".../vllm/v1/engine/core.py\", line 123, in __init__ self.model_executor = executor_class(vllm_config) File \".../vllm/v1/executor/uniproc_executor.py\", line 68, in _init_executor self.driver_worker.load_model() File \".../vllm/model_executor/model_loader/__init__.py\", line 140, in get_model return loader.load_model(...) File \".../vllm/model_executor/model_loader/utils.py\", line 63, in initialize_model model = model_class(vllm_config=vllm_config, prefix=prefix) File \".../vllm/model_executor/models/qwen2.py\", line 458, in __init__ self.model = Qwen2Model(...) File \".../vllm/model_executor/models/qwen2.py\", line 375, in __init__ self.start_layer, self.end_layer, self.layers = make_layers(...) File \".../vllm/model_executor/models/utils.py\", line 711, in make_layers + get_offloader().wrap_modules(...) File \".../vllm/model_executor/offloader/base.py\", line 104, in wrap_modules return list(modules_generator) File \".../vllm/model_executor/models/qwen2.py\", line 263, in __init__ self.self_attn = Qwen2Attention(...) File \".../vllm/model_executor/models/qwen2.py\", line 178, in __init__ self.rotary_emb = get_rope(...) File \".../vllm/model_executor/layers/rotary_embedding/__init__.py\", line 92, in get_rope rotary_emb = DualChunkRotaryEmbedding(...) File \".../vllm/model_executor/layers/rotary_embedding/dual_chunk_rope.py\", line 42, in __init__ self._compute_cos_sin_cache() File \".../vllm/model_executor/layers/rotary_embedding/dual_chunk_rope.py\", line 105, in _compute_cos_sin_cache q_cache = torch.cat((q_cos, q_sin), dim=-1).to(...) File \".../torch/cuda/__init__.py\", line 417, in _lazy_init raise AssertionError(\"Torch not compiled with CUDA enabled\") AssertionError: Torch not compiled with CUDA enabled ``` Fix: use `current_platform.device_type` instead of the \"cuda\" literal - the same portable pattern already used across vllm (config, compilation passes, other layers). The device index still comes from `torch.accelerator.current_device_index()`. No cache values change; only the device the cos/sin caches are placed on. ## Test Plan - New unit test `tests/model_executor/layers/test_dual_chunk_rope.py`: ```bash python -m pytest tests/model_executor/layers/test_dual_chunk_rope.py -v ``` - E2E: `vllm serve` a Dual Chunk Attention model (e.g. Qwen2.5-7B-Instruct-1M) on a non-CUDA backend (Ascend NPU). ## Test Result **Before** (Ascend NPU): `vllm serve Qwen2.5-7B-Instruct-1M` crashes during model loading (traceback above), `AssertionError: Torch not compiled with CUDA enabled`. **After**: - UT: ``` tests/model_executor/layers/test_dual_chunk_rope.py::test_device_follows_platform_type PASSED tests/model_executor/layers/test_dual_chunk_rope.py::test_device_matches_real_platform PASSED ``` - `vllm serve Qwen2.5-7B-Instruct-1M` on Ascend NPU starts and serves normally. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52116", "createdAt": "2026-08-13T07:19:20Z", "updatedAt": "2026-08-13T07:19:56Z", "timestamp": "2026-08-13T07:19:56Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia" ], "author": "danziheng1024", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a5e8643ec78914b18e60", "signalId": "github:vllm-project/vllm:pull_request:50874", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50874", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][R3] Size monolithic routing replay buffer for DP", "text": "## Purpose Fix routing-replay capture for the FlashInfer monolithic MoE kernel under naive data parallelism, including padded sequence-parallel shards when expert parallelism is enabled. Two related assumptions fail in a TP2/DP2 deployment: 1. **Replay buffer capacity.** `max_num_tokens` is a per-rank scheduler limit, while the naive dispatch path all-gathers rank-local batches before invoking the monolithic kernel. This produced an 8192-row replay buffer for a 16384-row kernel input and failed during warmup. The buffer is now sized for the larger of the DP and EP dispatch groups, since under EP the gathered batch is the flattened EP group rather than the DP group. 2. **Gathered shard layout.** Under DP+EP, `topk_ids` can contain gathered sequence-parallel shards whose `dp_metadata.local_sizes` include CUDA-graph/SP padding. Those rows do not match the unpadded per-DP token counts used by the existing capture paths, so the batch-dimension assertion fired. When the gathered tensor matches `sum(local_sizes)`, we locate the current DP rank's contiguous shard group using that exact all-gatherv layout and copy only its real token count, trimming trailing padding. Existing naive-DP, modular-local, and modular-SP layouts are unchanged, and no routing or model-output semantics change. ## Duplicate check I searched open upstream PRs for `routing replay buffer`, `routed experts capture`, `routing replay all-gather`, and `local_sizes routing`; none fixes this allocation or the padded-shard capture: - #48698 scales a separate FlashInfer B12x MoE workspace and does not touch the routing replay output added by #44214. - #50940 also edits `routed_experts_capturer.py`, but only the shape configuration; it leaves the `capture()` batch-layout branches alone. - #50759 skips MoE padding inside the router implementations, upstream of the gathered-batch layout handled here. ## Validation - `git diff --check`: passed - vLLM pre-commit on the changed files, run in the repo's Python 3.12 venv: Ruff check/format, `mypy-3.10`, typos, SPDX, root-lazy-import, forbidden-import, torch-CUDA-API, config-validation, and sign-off hooks all passed. (Under a host Python 3.9 two of these hooks crash on 3.10+ syntax; that is an environment artifact, not a finding.) - Runtime validation of the exact TP2/DP2+EP default-backend configuration was run on the corresponding branch of my fork on GB200: mixed and decode CUDA-graph capture completed, both API servers started, and a `/v1/completions` request was served with routed-experts capture enabled. The original routing batch-dimension assertion did not fire. - Model evaluation: not applicable; routing decisions and model outputs are unchanged by this fix. AI assistance was used. As the human submitter I have reviewed every changed line and the recorded runtime validation.", "url": "https://github.com/vllm-project/vllm/pull/50874", "createdAt": "2026-08-03T13:20:04Z", "updatedAt": "2026-08-13T07:18:09Z", "timestamp": "2026-08-13T07:18:09Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "bug", "ready" ], "author": "TomerBN-Nvidia", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:510a311490713bd6cac7", "signalId": "github:vllm-project/vllm:pull_request:48212", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48212", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Add CPU-tuned autotune configs for FLA (GDN) Triton kernels", "text": "> **Depends on #49583.** That PR bumps the triton-cpu pin to a commit that includes triton-cpu #275 (the `CPUOptions.hash()` runtime-only cache-key fix). Without it, the configs here recompile once per `num_cpu_threads` candidate, so the warmup speedup is only realized after #49583 lands. ## Purpose The FLA op kernels used GPU-oriented autotune configs that vary num_warps/num_stages. On the Triton CPU backend those parameters are ignored, so the search wasted many minutes recompiling identical x86 code during first-run warmup. For each FLA kernel, when current_platform.is_cpu(): - Replace the block-size search with @triton.heuristics that pick tile sizes deterministically via min(triton.next_power_of_2(dim), 64/128), keeping tiles L1-resident. - Autotune only num_cpu_threads over [16, 32, 64, 96] (the real CPU parallelism knob controlling the OpenMP launch). Guarded on current_platform.is_cpu() so GPU behavior is unchanged. Change-Id: I3edba2bd1fd5940d0f4d7ad6d9b752752c3b3d2e ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48212", "createdAt": "2026-07-10T05:36:58Z", "updatedAt": "2026-08-13T07:17:33Z", "timestamp": "2026-08-13T07:17:33Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "cpu" ], "author": "ganeshr10", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a2fe0ff1a674201ecf3f", "signalId": "github:vllm-project/vllm:pull_request:36951", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:36951", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Add persistent cache mounts and fix test download paths", "text": "This PR avoids cold network downloads during AMD CI test collection and execution. The immediate motivation was a Buildkite multimodal shard spending time downloading before the selected tests even ran: https://buildkite.com/vllm/ci/builds/76781/canvas?jid=019f3b96-c3dd-4f37-98b2-0599cfe73733&tab=output. The biggest issue was `test_phi4mm.py` resolving the 12.9 GB Phi-4 multimodal snapshot at module import time, even when pytest later deselected that file. This PR moves that setup into lazy fixtures, shares the Phi-4 model and LoRA paths across the relevant tests, and removes other top-level tokenizer/config loads that made collection expensive or network-dependent. Concretely, this adds persistent AMD CI cache mounts, routes direct test downloads through `VLLM_TEST_CACHE`, respects `HF_HOME` for cache-sensitive tests, and removes explicit cache paths that bypass the shared Hugging Face cache. It also covers files that do not naturally live in the HF cache, such as public vLLM assets, small direct-download fixtures, and the `tiktoken-rs` runtime vocab cache. AMD CI sets `VLLM_TEST_CACHE_ONLY=1`, so these helpers fail clearly on a cache miss instead of silently reaching out to the network during a test job. Direct URL helpers only treat non-empty files as cache hits and use atomic writes/copies so partially populated cache entries are not reused.", "url": "https://github.com/vllm-project/vllm/pull/36951", "createdAt": "2026-03-13T04:56:43Z", "updatedAt": "2026-08-13T07:16:28Z", "timestamp": "2026-08-13T07:16:28Z", "metrics": { "reactions": 4, "comments": 9 }, "labels": [ "rocm", "ci/build", "v1", "multi-modality", "tool-calling", "qwen", "gpt-oss", "kv-connector" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f3c92f18439cd83ca378", "signalId": "github:vllm-project/vllm:pull_request:52110", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52110", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[sharded state loader] support pp in sharded state loader", "text": "## Purpose For now, sharded state loader does not support pipeline parallel. Add pipeline parallel support for sharded state loader and keep backward compatibility, i.e., tensor parallel only. ## Test Plan Add some ut to cover newly added functionalities. 1. TP only, backward compatibility 2. PP only, save sharded state + load sharded state + generate 3. TP+PP, save sharded state + load sharded state + generate ## Test Result All pass ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52110", "createdAt": "2026-08-13T06:42:26Z", "updatedAt": "2026-08-13T07:16:26Z", "timestamp": "2026-08-13T07:16:26Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation" ], "author": "andyxning", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7c1ebfc4fd1aea5c4c2d", "signalId": "github:vllm-project/vllm:pull_request:49406", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49406", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Add PARD-2 parallel draft model support", "text": "## Purpose This PR adds support for PARD-2 ([AMD-AGI/PARD](https://github.com/AMD-AGI/PARD)) target-aligned parallel draft models as a speculative-decoding method in vLLM. ## Test Plan **Serve** (PARD-2, target-dependent): ``` vllm serve --max-model-len 4096 --no-enable-prefix-caching --port 8008 --host 0.0.0.0 \\ --speculative-config '{\"model\": \"\", \"num_speculative_tokens\": 5}' ``` **Client** (concurrency sweep): ``` vllm bench serve --port 8008 --model --temperature 0 --backend openai-chat --endpoint /v1/chat/completions --dataset-name spec_bench --dataset-path question.jsonl --spec-bench-category math_reasoning --spec-bench-output-len 256 --chat-template-kwargs '{\"enable_thinking\": false}' --num-prompts 80 --max-concurrency <1..64> ``` - Hardware: **AMD EPYC 9755 128-Core Processor** (single socket, 128 threads), CPU backend, bfloat16. ## Test Result - Target: `Qwen/Qwen3-14B` · PARD-2 draft: `amd/PARD2-Qwen3-14B` · PARD-1 draft: `amd/PARD-Qwen3-0.6B` - Dataset: **GSM8K** (spec_bench `math_reasoning` category), no-thinking - Draft length `K = 5` , greedy (temp 0), 80 prompts. ## Performance Highlights \"image\" - Acceptance length : **PARD-2 = 4.30** (rate 65.9%), PARD-1 = 4.00 (rate 60.0%) ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49406", "createdAt": "2026-07-22T04:36:08Z", "updatedAt": "2026-08-13T07:13:31Z", "timestamp": "2026-08-13T07:13:31Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "new-model", "speculative-decoding", "v1", "llama", "qwen" ], "author": "ganeshr10", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:eefc212ba981a89ade65", "signalId": "github:vllm-project/vllm:pull_request:49953", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49953", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][AITER] Add GDN long-prefill split-QKV fast path", "text": "## Motivation ROCm/aiter#4299 adds a channel-last HIP path for `causal_conv1d_fwd_split_qkv`, which can consume Qwen3.5's contiguous `[T, D]` projection through its logical `[D, T]` transpose without materializing a copy. vLLM already fuses its packed post-conv split, normalization, and gating, so dispatching the AITER op unconditionally regresses short prompts. This change uses the AITER path only for long BF16 Qwen3.5 GDN prefills where it is faster, and retains the existing Triton/FLA path everywhere else. A search of open vLLM PRs/issues did not find another change implementing this Qwen3.5 GDN channel-last split-QKV fast path. ## Summary - Probe for AITER's optional GDN prefill split-QKV op and gracefully fall back with older AITER builds. - Route non-interleaved Qwen3.5 BF16 prefills with at least 8192 tokens and width-4 convolution through the AITER channel-last kernel. - Add a fused Triton postprocessor that normalizes AITER's split Q/K outputs in place and computes GDN gating in the same launch; V is already in its final contiguous layout. - Precompute block-64 causal-conv metadata on the CPU to avoid a device-to-host synchronization at dispatch time while preserving block-8 metadata for the existing path. - Cover variable-length inputs, initial states, the vLLM null cache slot, long production-shaped inputs, and split-output postprocessing. ## AITER dependency - Required feature: ROCm/aiter#4299, now merged (`a96ee7b6e3c2688b099d6e32ec662bce8886efa7`). - End-to-end validation pin: AITER main `d19f33251400bbc21a3a49cc8db421f918716b93`. - AITER at this revision requires FlyDSL >= 0.2.4; the benchmark runtime pins FlyDSL 0.2.4. - Older AITER builds remain supported through the importability probe and use the existing Triton/FLA fallback. ## Test plan - [x] `pytest -q tests/kernels/test_fused_gdn_post_conv.py` — 68 passed - [x] `pytest -q tests/kernels/mamba/test_aiter_gdn_prefill.py` — 2 passed - [x] `pytest -q tests/v1/attention/test_gdn_metadata_builder.py` — 9 passed - [x] AITER `test_causal_conv1d_prefill_split_qkv.py -k hip_channel_last` — 11 passed - [x] `torch.compile(..., fullgraph=True)` smoke test for split-QKV postprocessing - [x] Ruff format/check, forbidden-import, torch-cuda-call, typos, and diff checks - [x] Qwen3.5-35B-A3B TP4 end-to-end serving A/B on MI308X/gfx942 - [x] Full 1319-question GSM8K, 5-shot, concurrency 256 Unit tests used `vllm/vllm-openai-rocm:v0.24.0`. End-to-end tests used that image with FlyDSL 0.2.4, vLLM baseline `afc94523c` vs PR `758e13f15`, and the same AITER `d19f332514` on both sides. ## Serve command Hardware: 4x AMD Instinct MI308X (gfx942), ROCm driver 6.16.13. Model: local BF16 `Qwen3.5-35B-A3B`. ```bash HIP_VISIBLE_DEVICES=0,1,2,3 VLLM_ROCM_USE_AITER=1 vllm serve /models/Qwen3.5-35B-A3B --served-model-name Qwen3.5-35B-A3B --host 0.0.0.0 --port 8000 --tensor-parallel-size 4 --trust-remote-code --dtype bfloat16 --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 256 --max-num-batched-tokens 32768 ``` ## End-to-end performance Each random workload was warmed up and then run three times with `temperature=0`, `ignore_eos=true`, fixed seed 20260727, and the same four GPUs. Values below are medians across the three runs. `ISL/OSL` are tokens per request. | Workload | Baseline total tok/s | PR total tok/s | Baseline median TTFT | PR median TTFT | Baseline p99 TTFT | PR p99 TTFT | |---|---:|---:|---:|---:|---:|---:| | ISL 4096 / OSL 1 / C1 / 8 prompts (fallback control) | 25,197.1 | 25,259.8 (+0.25%) | 161.37 ms | 161.64 ms (-0.17%) | 167.80 ms | 170.33 ms | | ISL 8192 / OSL 1 / C1 / 8 prompts (dispatch threshold) | 29,045.6 | 29,186.3 (+0.48%) | 283.37 ms | 280.48 ms (+1.03%) | 293.99 ms | 291.89 ms | | ISL 16384 / OSL 1 / C1 / 4 prompts (long prefill) | 25,250.4 | 25,486.3 (+0.93%) | 648.12 ms | 643.43 ms (+0.73%) | 662.01 ms | 653.29 ms | | ISL 8192 / OSL 128 / C256 / 256 prompts (loaded serving) | 25,113.6 | 26,989.7 (+7.47%) | 44.034 s | 40.399 s (+8.25%) | 78.858 s | 72.458 s | For the loaded workload, request throughput improved from 3.018 to 3.244 req/s (+7.47%), and mean TPOT improved from 323.73 to 306.22 ms (-5.41%). All runs completed with zero failed requests. The PR server log confirms `module_causal_conv1d_fwd_split_qkv` was built and invoked by the 9020-token probe. The previously reported `1.163x` is **operator latency speedup**, not TTFT, TPOT, or model throughput: for the Qwen3.5-35B-A3B TP4 per-rank shape (`D=2048`, BF16, width 4, T=8192), existing Triton conv + fused postprocess measured 0.1375 ms and AITER channel-last split-QKV + fused in-place postprocess measured 0.1182 ms. ## GSM8K accuracy Command: ```bash python tests/evals/gsm8k/gsm8k_eval.py --port 8000 --num-questions 1319 --num-shots 5 --max-tokens 256 --temperature 0 --seed 42 --max-concurrency 256 ``` | Revision | Correct | Accuracy | Invalid | Questions/s | |---|---:|---:|---:|---:| | Baseline `afc94523c` | 1177 / 1319 | 89.234% | 0% | 15.739 | | PR `758e13f15` | 1173 / 1319 | 88.931% | 0% | 15.658 | The observed delta is -0.303 percentage points (4 questions), with no invalid responses. A separate deterministic 9020-token long-prefill probe produced the correct answer (`42`) on both revisions; wording diverged after a common prefix because the fused path is not bitwise identical. ## AI assistance disclosure AI assistance was used for implementation support, benchmark automation, and result summarization. The human submitter remains responsible for reviewing the diff and validating the reported results before merge.", "url": "https://github.com/vllm-project/vllm/pull/49953", "createdAt": "2026-07-27T08:12:24Z", "updatedAt": "2026-08-13T07:12:49Z", "timestamp": "2026-08-13T07:12:49Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "v1" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:34e5020a5fd612dc89fe", "signalId": "github:vllm-project/vllm:pull_request:52115", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52115", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Preserve Step3-VL zero-patch CUDA graph outputs", "text": "## Summary - honor the encoder postprocessor's `clone` contract when a Step3-VL image has zero local patches - keep the no-clone path as an alias while preserving cloned outputs across later CUDA graph replays - add a focused regression test for both ownership modes Without this, the single-part fast path returns the CUDA graph output buffer directly even when `clone=True`. A later replay can therefore overwrite a previously cached image embedding and make it represent another image. ## Validation - Focused leased RTX 2080 Ti (SM75, FP16) model-level CUDA graph validation: **BASE RED / PATCH GREEN** - Two distinct 512×512 images, `num_patches=[0, 0]`, real CUDA graph capture and production postprocessor - Base: the first retained output changed after replaying the second image - Patch: both replay-time outputs matched eager execution and the first retained output stayed unchanged - `git diff --check` - `python3 -m py_compile vllm/model_executor/models/step3_vl.py tests/models/multimodal/processing/test_step3_vl_image_embeds.py` A local focused pytest was not run because this isolated worktree has no project virtual environment; the GPU model-level harness exercised the target production path directly. ## Duplicate check I searched current open vLLM issues and PRs for Step3-VL, encoder CUDA graph, zero-patch, aliasing, and clone/output ownership. I found no open change addressing this ownership bug. The current `main` implementation still has the faulty single-part fast path. ## Model evaluation This is an ownership/lifetime correction rather than a numerical model change. The leased GPU validation compared both images against eager output at replay time and confirmed exact preservation after the second replay. ## AI assistance AI assistance was used for source investigation, implementation, and test-harness preparation. I reviewed every changed line and independently verified the diff, duplicate status, and GPU RED/GREEN evidence.", "url": "https://github.com/vllm-project/vllm/pull/52115", "createdAt": "2026-08-13T07:10:24Z", "updatedAt": "2026-08-13T07:12:02Z", "timestamp": "2026-08-13T07:12:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "multi-modality", "nvidia" ], "author": "qdivan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d056a8b10d4ee7155dfa", "signalId": "github:vllm-project/vllm:pull_request:47335", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47335", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Enable fused indexer-Q RoPE+quant kernel for DeepSeek/GLM sparse attention", "text": "## Summary Enables the existing fused indexer-Q kernel (`fused_indexer_q_rope_quant`) on ROCm for the DeepSeek Sparse Attention (DSA) indexer used by GLM-5.2 / DeepSeek-V3.2-style models. On CUDA this kernel already collapses the indexer's query-side RoPE + FP8 quantization + weight-scale fold into a single launch; on ROCm the same work was falling back to a chain of small Triton/elementwise kernels per sparse layer per decode token. This is a low-risk, 2-line enablement — the kernel is already on vllm and is architecture-aware. It **removes redundant kernels and HBM round-trips** on the ROCm decode path (4 Q-side kernels → 1), yielding a small but consistent decode improvement with **no CUDA-side change** and no measured regression. ## Kernel-level evidence (torch.profiler, GLM-5.2-FP8, MI325X / gfx942, TP8, rank 0) Before/after self-CUDA over the same steady-state decode capture. The indexer decode kernels are the ones with ~21.5k calls (per decoded token × sparse layers). **Fused away** (present in baseline, gone after): | Kernel | Role | Self CUDA | Calls | |---|---|---|---| | `per_token_group_quant_8bit_kernel` | q → FP8 (ue8m0) quant | 99.8 ms | 21546 | | `triton_poi_fused_3` | RoPE/cat elementwise | 95.8 ms | 21526 | | `triton_poi_fused_2` | RoPE/cat elementwise | 89.0 ms | 20521 | | `triton_poi_fused_mul_slice_unsqueeze_view_4` | weights scale-fold | 85.3 ms | 20500 | **Introduced**: | Kernel | Role | Self CUDA | Calls | |---|---|---|---| | `_fused_indexer_q_rope_quant_kernel` | fused q RoPE + FP8 quant + weight-fold (decode) | 103.7 ms | 21546 | | `_fused_indexer_q_rope_quant_kernel_0` | same, prefill variant | 6.3 ms | 42 | | `triton_poi_fused_add_copy_index_select_mul_slice_…` | residual K-side RoPE (now its own kernel) | 90.6 ms | 20500 | The Q-side collapses from **4 dedicated kernels → 1** (the two `triton_poi_fused_2/_3` RoPE kernels did q *and* k together; after the change q's RoPE is absorbed and only a single K-RoPE kernel remains). Net ≈ **170 ms** self-CUDA reclaimed on rank 0 over the capture (≈0.7% of the ~24.9 s total). Untouched: `k_norm`, `indexer_k_quant_and_cache`, `_gluon_deepgemm_fp8_paged_mqa_logits`, `topKPerRowDecode`. ## Accuracy GLM-5.2-FP8, TP=8 on gfx942 |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9469|± |0.0062| | | |strict-match | 5|exact_match|↑ |0.9469|± |0.0062| ## End-to-end performance (GLM-5.2-FP8, TP8, MI325X / gfx942) `vllm bench serve`, random dataset, `--ignore-eos`, `--random-range-ratio 0.0`. Decode runs under CUDA graphs. | ISL | OSL | conc | TPOT_p50 base | TPOT_p50 fused | TPOT | ITL_p50 base | ITL_p50 fused | ITL | tok/s base | tok/s fused | tput | |---|---|---|---|---|---|---|---|---|---|---|---| | 1024 | 1024 | 1 | 14.98 | 14.93 | 1.004x | 14.98 | 14.94 | 1.003x | 66.32 | 66.34 | 1.000x | | 1024 | 1024 | 8 | 18.39 | 18.32 | 1.004x | 18.40 | 18.30 | 1.005x | 422.81 | 423.68 | 1.002x | | 1024 | 1024 | 32 | 24.83 | 24.71 | 1.005x | 24.31 | 24.19 | 1.005x | 1220.94 | 1222.21 | 1.001x | | 1024 | 1024 | 64 | 32.39 | 32.30 | 1.003x | 31.25 | 31.00 | 1.008x | 1835.24 | 1842.09 | 1.004x | | 8192 | 1024 | 1 | 16.01 | 15.90 | 1.006x | 15.98 | 15.88 | 1.006x | 60.18 | 60.55 | 1.006x | | 8192 | 1024 | 8 | 22.26 | 22.06 | 1.009x | 19.86 | 19.79 | 1.004x | 323.27 | 324.17 | 1.003x | | 8192 | 1024 | 32 | 38.86 | 39.73 | 0.978x* | 25.14 | 24.98 | 1.006x | 715.20 | 718.98 | 1.005x | | 8192 | 1024 | 64 | 63.69 | 63.66 | 1.001x | 29.72 | 29.80 | 0.997x | 925.56 | 925.84 | 1.000x | Consistent small improvement across the sweep (~0.3–0.9%), with no credible regression. *The single `8192/32` TPOT dip is run-to-run variance: ITL (1.006x) and throughput (1.005x) both improve at that same point. ## Related work / positioning - **#43907** (`[ROCm][Perf] DSv3.2: fuse indexer Q-RoPE+quant + K-norm/RoPE/quant/cache`) fuses Q **and** K via an **AITER** kernel, but **explicitly excludes GLM** (`model_type != \"glm_moe_dsa\"`) and depends on a companion aiter kernel behind an env flag. This PR covers `glm_moe_dsa` using the **in-tree Triton** kernel with no aiter dependency and no new flags — complementary, not overlapping. - **#44527** (`[ROCm][DSv3.2] Eliminate per-decode FillFunctor launches`) removes the `Fill` kernels in the same hot loop; stacks cleanly on top of this change. ## Risk / compatibility Low. CUDA behavior is bit-for-bit unchanged; the ROCm change is opt-in via the same static guard and only alters the previously-fragmented eager path. The end-to-end gain is intentionally modest (kernel/HBM-traffic reduction under CUDA graphs), but the change is trivial, carries no measured regression, and compounds with related sparse-indexer cleanups (#44527). Requires the fused branch to fire (`is_inplace_rope == False`), which is the default under `torch.compile`/Inductor.", "url": "https://github.com/vllm-project/vllm/pull/47335", "createdAt": "2026-07-01T20:26:21Z", "updatedAt": "2026-08-13T07:11:28Z", "timestamp": "2026-08-13T07:11:28Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "ready", "deepseek", "quantization" ], "author": "xaguilar-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a006d43bf7ef491f29c0", "signalId": "github:vllm-project/vllm:pull_request:51287", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51287", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable async scheduling when VLLM_BATCH_INVARIANT=1", "text": "## Purpose `VLLM_BATCH_INVARIANT=1` pins the *kernels* so a row's result no longer depends on batch size or composition. It does nothing about **which batch the scheduler puts a request in**. Async scheduling is enabled by default (`SchedulerConfig.async_scheduling = None` -> `True`) and composes the next step's batch before the current step has retired, so batch membership depends on wall-clock timing. Timing jitter therefore reaches the numerics again, and the flag does not deliver what it promises. There is no guard, no warning, and nothing in the docs. The server boots, serves, and quietly is not reproducible. `--no-async-scheduling` fixes it, at roughly 3% throughput. This is not a corner case for the flag's intended users: batch invariance exists so research and regulated workloads can reproduce results, and those are exactly the workloads running large batches of long, variable-length requests. This PR treats `VLLM_BATCH_INVARIANT` as an async-scheduling incompatibility, following the existing structure in `VllmConfig.__post_init__`: - **explicitly enabled** (`--async-scheduling`): `ValueError` pointing at `--no-async-scheduling`, matching the ROCm DeepEP DBO and unsupported-executor cases. - **unset** (the default): disabled with `warning_once`, matching the pooling-model, spec-decode and ROCm DBO cases. The closest precedent is ROCm DeepEP high-throughput DBO, disabled \"because that combination can corrupt DP+EP generation accuracy\", which is the same argument. vLLM already auto-disables cascade attention and custom all-reduce under this same flag. Also documents the interaction in `docs/features/batch_invariance.md`, which currently says nothing about scheduler configuration. Related: #27433. ## Test Plan ### 1. Config tests (CPU only) Added to `tests/test_config.py`, next to the existing `test_draft_model_enables_async_scheduling_by_default`: - `test_batch_invariance_disables_async_scheduling`: with the flag set, `async_scheduling` resolves to `False`. - `test_batch_invariance_rejects_explicit_async_scheduling`: with the flag set and `async_scheduling=True` passed explicitly, `VllmConfig` raises. ### 2. End to end Serve, before this patch, once per row: ```bash # A: default, async scheduling on VLLM_BATCH_INVARIANT=1 vllm serve $MODEL --no-enable-prefix-caching # B: the fix VLLM_BATCH_INVARIANT=1 vllm serve $MODEL --no-enable-prefix-caching --no-async-scheduling ``` Against each, run: ```python # pip install openai import concurrent.futures as cf, hashlib, sys from openai import OpenAI MODEL, BATCHES, REPEATS = sys.argv[1], (1, 5, 10, 50), 3 client = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\") WORDS = (\"market revenue guidance segment margin pipeline capacity demand headwind \" \"backlog utilization cadence inventory logistics currency hedging\").split() def text(n, shift=0): return \" \".join(WORDS[(i + shift) % len(WORDS)] for i in range(n)) PROBE = \"Summarize this in detail, listing every theme you find.\\n\\n\" + text(9000) def answer(prompt, limit): m = client.chat.completions.create( model=MODEL, messages=[{\"role\": \"user\", \"content\": prompt}], temperature=0, max_tokens=limit).choices[0].message return (getattr(m, \"reasoning_content\", \"\") or \"\") + (m.content or \"\") def probe_hash(batch): jobs = [(PROBE, 600)] + [(text(3000, i), 400 + i % 8 * 150) for i in range(batch - 1)] with cf.ThreadPoolExecutor(batch) as pool: probe = list(pool.map(lambda j: answer(*j), jobs))[0] return hashlib.sha256(probe.encode()).hexdigest()[:12] hashes = {} for rep in range(REPEATS): for batch in BATCHES: h = probe_hash(batch) hashes.setdefault(h, []).append(f\"rep{rep} batch{batch}\") print(f\"rep{rep} batch{batch:>3} {h}\") print(\"PASS\" if len(hashes) == 1 else f\"FAIL {len(hashes)} outputs for one input\") ``` Two properties of that script are load-bearing. The **probe is long**: short prompts are reproducible even on a badly broken config. The **fillers use staggered `max_tokens`**: they retire at different decode steps, so batch composition churns while the probe is still generating. Drop either one and this passes on a server that is not batch invariant, which is exactly why the existing tests miss it (below). After this patch, A and B are the same run: async scheduling is off by default under the flag, and the extra argument is no longer needed. ## Test Result Measured on `openai/gpt-oss-120b`, single A100-80GB (sm80), `TRITON_ATTN`, temperature 0. | | probe reproducible | |---|---| | A, default (async scheduling on) | **9/10**, one request flips between runs | | B, `--no-async-scheduling` | **10/10** | Throughput cost of B: roughly 3%. The flipping request is stable when run alone (6/6 identical), so this needs co-residency: it is batch composition, not the request itself. It bites inputs sitting on a near-tie. In one case the same prompt produced either a long answer or an essentially empty one, a sub-ULP difference selecting between \"emit content\" and \"emit nothing\". ### Why the existing determinism tests do not catch this `tests/v1/determinism/test_batch_invariance.py` passes on a configuration where the above is failing. Two structural blind spots: 1. **`needle_prompt = \"There once was a \"`**: the request whose output is *checked* is 7 characters, and only the fillers are long. Short prompts are reproducible even when the config is broken. 2. **`max_tokens = 128` for every request**: all requests decode the same number of steps and retire *together*, so decode batch composition is constant for the whole test. The condition that breaks is batch composition **changing mid-decode**, not batch size. ### Caveat The end-to-end numbers were taken on v0.23.0 with #46639's Marlin MoE kernel back-ported and compiled for sm80, since MXFP4 MoE has no batch-invariant kernel on sm80 otherwise. The config change here is architecture independent and covered by the CPU-only tests. A maintainer re-running the determinism suite on a natively supported configuration would be worth doing before merge. The serve configuration, the build recipe for that sm80 kernel, and a fuller version of the test script above are at https://github.com/GiesDSRS/batch-invariant-vllm-a100.", "url": "https://github.com/vllm-project/vllm/pull/51287", "createdAt": "2026-08-06T17:27:06Z", "updatedAt": "2026-08-13T07:11:21Z", "timestamp": "2026-08-13T07:11:21Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "documentation", "needs-rebase" ], "author": "fsaudm", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ca44d869f743e2ed6f14", "signalId": "github:vllm-project/vllm:pull_request:44298", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44298", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Add unit test for write_zeros_to_output MoE kernel", "text": "Adds a unit test for `write_zeros_to_output` (`vllm/model_executor/layers/fused_moe/fused_moe.py`), part of RFC #48480. The function zeroes the output tile when a block's expert is not on the current EP rank (`off_experts == -1`). It is a `@triton.jit` device function, so a launcher kernel wraps it and results are compared against a PyTorch reference. 4 tests / 11 cases cover the token mask, padding sentinel ids, a partial column tail and a strided output view. Test-only change. ## Test Plan ``` pytest tests/kernels/moe/test_write_zeros_to_output.py ``` ## Test Result All 11 tests passed", "url": "https://github.com/vllm-project/vllm/pull/44298", "createdAt": "2026-06-02T07:57:22Z", "updatedAt": "2026-08-13T07:09:18Z", "timestamp": "2026-08-13T07:09:18Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "pmanczak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e109592bf58f2fb0a734", "signalId": "github:vllm-project/vllm:pull_request:52082", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Fix memory pinning in Qwen3_VisionTransformer for non-gpu", "text": "Fix for CI failure introduced by https://github.com/vllm-project/vllm/pull/51841", "url": "https://github.com/vllm-project/vllm/pull/52082", "createdAt": "2026-08-13T02:35:39Z", "updatedAt": "2026-08-13T07:05:43Z", "timestamp": "2026-08-13T07:05:43Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "ready", "needs-rebase", "qwen" ], "author": "njhill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b5d6e3fc4d664c3de071", "signalId": "github:vllm-project/vllm:pull_request:51560", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51560", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fail fast on Inkling's unsupported GPU architectures", "text": " ## Purpose Fixes the startup crash reported in #51405. Serving Inkling on SM120 (RTX PRO 6000 Blackwell, GeForce Blackwell, DGX Spark) died during startup profiling with ``` AssertionError: Paged KV not supported on SM 12.0 in this PR RuntimeError: Worker failed with error 'Paged KV not supported on SM 12.0' ``` raised from the vendored FA4 relative-attention forward, after weights had already been loaded and the KV cache sized. The FA4 kernel has no paged-KV forward on SM12x, and vLLM always attends over the paged KV cache, so there is no configuration that makes the model run there. **Scope note:** an earlier revision also rejected SM8x, since FA4 asserts `page_table is None` there too. @TheVerdman demonstrated Inkling serving end-to-end on 4×A100 through a FlexAttention fallback carrying the relative bias via `score_mod`, so SM8x is deliberately left open for that work and this PR rejects SM12x only. The capability tables are a deny-list of broken architectures rather than an allow-list of working ones, so adding a fallback means adding a path, not deleting a guard. This does not add SM120 support. The kernel is fetched at build time from `vllm-project/tml-fa4` (pinned in `cmake/external_projects/tml_fa4.cmake`) and `vllm-project/flash-attention`, so a paged-KV SM120 forward cannot be implemented from this repo. What this PR does is take the second option #51405 asks for — *\"a documented supported-hardware note ... would save others the investigation\"* — and reject those architectures up front instead of asserting inside a worker after a multi-GiB weight load. **Changes** - `vllm/platforms/cuda.py` — `CudaPlatformBase.verify_model_arch` rejects the three Inkling architectures, driven by a new `_CAPABILITY_RESTRICTED_MODELS` table. Mirrors the existing `RocmPlatform.verify_model_arch` / `_ROCM_UNSUPPORTED_MODELS` pattern. The capability query is wrapped in `try/except RuntimeError` because on non-NVML installs it goes through torch and can initialize CUDA in the resolving process. - `vllm/config/model.py` — calls `verify_model_arch` from `ModelConfig` once the architecture is resolved. Without this the hook never runs in the front end: `_try_load_model_cls` is its only caller, and front-end resolution goes through `inspect_model_cls` instead, so on SM120 the guard fired inside the worker off the usage-stats path in `init_device()` (see @Rawsejet's [test report](https://github.com/vllm-project/vllm/pull/51560#issuecomment-5257205501)). Both TP workers spawned and dumped a traceback before `EngineCore` shut them down — still before `load_model()`, but noisier and later than it needs to be. The `_try_load_model_cls` call stays as a backstop for paths that build a model without a `ModelConfig`. On ROCm this is a no-op today: `_ROCM_UNSUPPORTED_MODELS` and `_ROCM_PARTIALLY_SUPPORTED_MODELS` are both empty. - `vllm/models/inkling/nvidia/attention.py` — `InklingAttention.__init__` keeps a check of its own for direct construction, naming the device and the missing kernel. This constructor is the single chokepoint for both backbone (`nvidia/model.py`) and MTP (`nvidia/mtp.py`) layers; the vision/audio towers are Triton-only and never reach FA4. - `vllm/models/inkling/nvidia/ops/fa4_rel_attention.py` — adds `check_inkling_fa4_support()`, and folds the three arch predicates scattered through the file into named constants (`_PAGED_KV_MAJORS`, `_SHEARED_BIAS_MAJORS`, `_SPLIT_KV_MAJORS`) with one comment pointing at the tml-fa4 pin they all track. This also fixes `inkling_fa4_num_splits`, which keyed off `major == 9` and so returned a split count of up to 128 on architectures whose kernel has no split-KV support at all - `docs/models/supported_models.md` — Inkling had no rows in either table. Adds `InklingForCausalLM` and `InklingForConditionalGeneration`, plus a note recording that SM120/SM121 are unsupported, why, and that ROCm uses a separate Triton implementation and is unaffected. - Tests — coverage for the guard, for the platform hook, a test that `InklingAttention.__init__` actually invokes the guard before touching anything else, and `tests/test_config.py::test_platform_rejects_unsupported_arch_during_config`, which pins the front-end call so the hook cannot silently regress to worker-only again. Adds `skip_if_capability_restricted` to `tests/models/utils.py`, used by `test_initialization.py` and `test_registry.py`: both resolve every registered architecture, and `_try_load_model_cls` calls `verify_model_arch` outside its `try/except`, so without the skip they would error rather than skip on a GPU that cannot run Inkling. **No behavior change on any architecture that could already run Inkling.** On SM8x/SM90/SM100/SM110 the guard is a no-op and `inkling_fa4_num_splits` returns exactly what it returned before (`major == 9` → 1, unchanged; 10/11 → the same computed value). No kernel is added and no numerics are altered on any architecture that could run Inkling previously, so there is no accuracy or serving delta to evaluate. The only behavioral change is on architectures where the model could not run at all **Not duplicating existing work.** `gh pr list --repo vllm-project/vllm --state open --search \"51405 in:body\"` returns nothing, and #51405 is open with no comments. The open Inkling PRs are unrelated: #51517 and #50528 (parser), #49120 (fp32 head dtype), #49315 (warmup infrastructure), #48768 (multi-depth MTP ) ## Test Plan Lint and static checks over every changed file: ```bash pre-commit run --files docs/models/supported_models.md \\ tests/models/inkling/test_fa4_rel_attention.py \\ tests/models/test_initialization.py tests/models/test_registry.py \\ tests/models/utils.py tests/test_config.py vllm/config/model.py \\ vllm/models/inkling/nvidia/attention.py \\ vllm/models/inkling/nvidia/ops/fa4_rel_attention.py vllm/platforms/cuda.py ``` Unit tests covering the new guard, the platform hook, and the arch constants: ```bash .venv/bin/python -m pytest tests/models/inkling/test_fa4_rel_attention.py -v .venv/bin/python -m pytest tests/test_config.py -k platform_rejects_unsupported_arch -v ``` Regression check on the two suites that resolve every registered architecture, since `verify_model_arch` now runs in that path: ```bash .venv/bin/python -m pytest tests/models/test_registry.py -v .venv/bin/python -m pytest \"tests/models/test_initialization.py::test_can_initialize_large_subset\" -v ``` End-to-end on an unsupported device (SM120) — the failure should now appear during startup, naming the device, rather than as a worker assertion after the weight load: ```bash vllm serve thinkingmachines/Inkling-Small-NVFP4 \\ --tensor-parallel-size 2 \\ --tokenizer-mode inkling \\ --trust-remote-code ``` And on a supported device (SM90/SM100), to confirm serving is unaffected: ```bash vllm serve thinkingmachines/Inkling-Small-NVFP4 --trust-remote-code ``` ## Test Result **Lint — all hooks pass, and the run is idempotent (a second pass modifies nothing):** ``` ruff check.......................................................Passed ruff format......................................................Passed typos............................................................Passed markdownlint-cli2................................................Passed Run mypy for Python 3.10.........................................Passed Check SPDX headers...............................................Passed Check root lazy imports..........................................Passed Check for spaces in all filenames................................Passed Update Dockerfile dependency graph...............................Passed Check for forbidden imports......................................Passed Prevent new 'torch.cuda' APIs call...............................Passed Validate configuration has default values and that each field has a docstring................................................Passed Check for boolean ops in with-statements.........................Passed Suggestion.......................................................Passed ``` **Before, on SM120** (as reported in #51405):weights load, the FP4 MoE backend selects, the KV cache is sized, and the worker then dies in the forward pass with `AssertionError: Paged KV not supported on SM 12.0 in this PR`. **After, on SM120** — tested by @Rawsejet on the #51405 reporter's hardware ([full report](https://github.com/vllm-project/vllm/pull/51560#issuecomment-5257205501)): 2× RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, driver 595.58.03, CUDA 13.2, vLLM `0.26.1rc1.dev376+g613411a90`, torch `2.13.0+cu130`, TP=2, runtime hunks applied onto the installed nightly. Both guards behave as intended on the live device: ``` check_inkling_fa4_support() -> ValueError (names the device, 12.0) verify_model_arch(InklingForConditionalGeneration) -> ValueError (12.x) verify_model_arch(InklingForCausalLM) -> ValueError (12.x) verify_model_arch(InklingMTPModel) -> ValueError (12.x) verify_model_arch(LlamaForCausalLM) -> OK (no raise) ``` `vllm serve ... --tensor-parallel-size 2 --tokenizer-mode inkling --trust-remote-code` exits in ~25 s with the clean `ValueError` and **no weights loaded**, against a full ~152 GiB load before the assertion previously. That run also showed the guard firing **in the worker**, not the front end, which the description previously claimed — `_try_load_model_cls` was the only caller of `verify_model_arch`, and front-end resolution does not reach it. The `vllm/config/model.py` change above fixes that; the worker-side check remains as a backstop. Re-test on SM120 welcome. **Everything else — not yet run.** I don't have access to SM90/SM100 hardware or a GPU build, so I have not executed the pytest suites or the supported-device `vllm serve` command above, and I'd rather say so than paste results I can't stand behind. Requesting CI to run the unit and registry suites. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51560", "createdAt": "2026-08-09T13:32:22Z", "updatedAt": "2026-08-13T07:01:19Z", "timestamp": "2026-08-13T07:01:19Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "bug", "documentation", "nvidia" ], "author": "m4xkushnir", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f93ce52f996c9824be5a", "signalId": "github:vllm-project/vllm:issue:49210", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:49210", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Engine core livelock (100% CPU, no crash) with MTP speculative decoding + xgrammar structured outputs - regression from v0.24.0", "text": "## Your current environment ``` vLLM version: 0.25.1 (pip install, venv) xgrammar version: 0.2.1 PyTorch: 2.11.0+cu130 CUDA: 13.0 GPU: 1x NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) Driver: 580.159.03 OS: Ubuntu 25.10, kernel 6.17.0-40-generic Model: nvidia/Qwen3.6-27B-NVFP4 (compressed-tensors, NVFP4 weights, FP8 KV cache) ``` Full serve command: ``` vllm serve /path/to/nvidia--Qwen3.6-27B-NVFP4 \\ --gpu-memory-utilization 0.92 \\ --served-model-name local_qwen \\ --tensor-parallel-size 1 \\ --max-model-len 131072 \\ --max-num-seqs 256 \\ --max-num-batched-tokens 32768 \\ --long-prefill-token-threshold 4096 \\ --enable-prefix-caching \\ --renderer-num-workers 4 \\ --mm-processor-cache-gb 0 \\ --trust-remote-code \\ --quantization compressed-tensors \\ --dtype bfloat16 \\ --kv-cache-dtype fp8 \\ --calculate-kv-scales \\ --limit-mm-per-prompt '{\"image\":20}' \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_xml \\ --default-chat-template-kwargs '{\"enable_thinking\": false}' \\ --scheduling-policy priority \\ --speculative-config '{\"method\":\"qwen3_next_mtp\",\"num_speculative_tokens\":3}' \\ --structured-outputs-config '{\"backend\": \"xgrammar\", \"disable_any_whitespace\": true}' ``` ## 🐛 Describe the bug Under sustained concurrent load with structured outputs (JSON schema / tool calling), the engine core enters a **silent livelock**: the `VLLM::EngineCore` main thread spins at 100% CPU indefinitely, the scheduler never advances, and the API server keeps accepting requests that never return. There is **no crash, no `EngineDeadError`, no watchdog trigger** - the service stays `active (running)` while being completely unresponsive. In our case it stayed in this state for 8.5 hours until we noticed and restarted manually. This does **not** reproduce on v0.24.0 with a byte-identical configuration (same model, same flags, same xgrammar 0.2.1, same load) — details below. ### Timeline (from journald) Engine start 07:48. For ~25 minutes everything works, but xgrammar logs a steady stream of rollback warnings - **652 occurrences, every single one for token id 198** (the `\\n` token in the Qwen tokenizer): ``` [08:11:36] /project/cpp/grammar_matcher.cc:612: Warning: The matcher has terminated after accepting the stop token, but is trying to accept new token with id 198. ``` At 08:12:25, ten seconds before the hang, the spec decoding metrics report a **draft acceptance rate above 100%**, i.e. more accepted than drafted tokens — which should be impossible and suggests accounting is already corrupted at this point: ``` 08:12:25 SpecDecoding metrics: Mean acceptance length: 1.98, Accepted throughput: 5.00 tokens/s, Drafted throughput: 4.80 tokens/s, Accepted: 50 tokens, Drafted: 48 tokens, Per-position acceptance rate: 0.353, 0.333, 0.294, Avg Draft acceptance rate: 104.2% ``` One more normal stats interval, then the last line the engine ever logs: ``` 08:12:35 Engine 000: Avg prompt throughput: 1545.3 tokens/s, Avg generation throughput: 440.0 tokens/s, Running: 3 reqs, Waiting: 1 reqs, Deferred: 1 reqs, GPU KV cache usage: 4.3% 08:12:45 Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 3 reqs, Waiting: 1 reqs, Deferred: 1 reqs, GPU KV cache usage: 4.3% ``` After that: zero log output of any kind (checked 8.5 hours later - nothing, not even the 10s stats logger). ### Process state during the hang ``` $ ps -p -o pid,stat,etime,pcpu,comm PID STAT ELAPSED %CPU COMMAND 3213241 Rl 08:51:56 101 VLLM::EngineCore ``` Main thread in state `R` at ~100% CPU the whole time (8h51m accumulated CPU), all other threads sleeping. `nvidia-smi` reports 100% GPU utilization and full memory allocation held by the process, consistent with a busy-wait / spin rather than real work. The API server process stays healthy: it accepts connections and requests, which then never complete (a trivial 5-token completion runs into any client timeout). ### Control experiment: identical config on v0.24.0 is stable We run a second machine with the **byte-identical** serve command (same model snapshot, same flags including the same `--speculative-config` and `--structured-outputs-config`, two instances on two GPUs) on **vLLM 0.24.0 with the same xgrammar 0.2.1**, serving the same workload: - zero `grammar_matcher.cc:612` warnings in 7 days of journald logs, - no hangs, uptime spanning multiple days of sustained traffic. So with the variables isolated: same model, same config, same xgrammar version, same workload - v0.24.0 never triggers the rollback warning; v0.25.1 floods it and livelocks within ~25 minutes. This looks like a regression introduced with the v0.25 execution-path changes (Model Runner V2 default / reworked spec decoding), not an xgrammar version issue. ### Root cause hypothesis The symptoms match the FSM state corruption described in #27210: xgrammar's `GrammarMatcher.rollback()` does not reset the internal `terminated_` flag when rolling back past a terminating token, so the matcher believes the FSM is terminated while vLLM's wrapper believes it is active. Every subsequent `accept_token` for the draft-proposed `\\n` then fails with the `grammar_matcher.cc:612` warning. #27210 (on the pre-0.25 path) ended in a hard `AssertionError` crash and was closed via #28298, which clears `spec_token_ids` after scheduling instead of asserting. My suspicion is that on the v0.25 MRv2 path the same underlying rollback/terminated inconsistency no longer crashes but instead leaves the scheduler in a state it can never advance from - turning the former crash into a silent livelock (arguably worse operationally, since nothing restarts the engine). The impossible >100% acceptance-rate metric right before the hang supports the theory that draft-token accounting and grammar state disagree at that point. Possibly related: #34650 (MTP breaks stop/think-token detection in structured outputs), #40926 / #41530 (MTP worker hangs under sustained traffic, though those die with `EngineDeadError` via RPC timeout, while this one spins forever with TP=1 and no RPC in the path). ### Reproduction conditions - MTP speculative decoding (`qwen3_next_mtp`, 3 speculative tokens) + xgrammar structured outputs with `disable_any_whitespace: true` - Sustained concurrent traffic (~10–35 concurrent requests), high share of JSON-schema / tool-call requests - Hang occurred ~25 min after engine start under this load; the `token id 198` warnings start almost immediately after startup and flood continuously until the hang, so warning onset is quick even if the full livelock takes a while Happy to provide full logs, run a patched build, or test with `py-spy` attached if that helps narrow it down.", "url": "https://github.com/vllm-project/vllm/issues/49210", "createdAt": "2026-07-20T15:00:02Z", "updatedAt": "2026-08-13T06:58:47Z", "timestamp": "2026-08-13T06:58:47Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [], "author": "KotlinFactory", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:23b63cfdc92e2eded62d", "signalId": "github:vllm-project/vllm:pull_request:46690", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46690", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix UVA offload fallback copies", "text": "## Purpose Fixes the non-UVA fallback path reported in #37883. The issue was closed automatically, but received new reports on August 10 and 11. When UVA is unavailable or disabled, `UVAOffloader` copies offloaded module state to the accelerator for each forward call. The previous wrapper had several problems: - `state_dict()` inside `forward()` is not traceable by Dynamo. - Temporarily replacing `module.forward` lets exceptions and concurrent calls lose or bypass the wrapper. - State captured before post-load processing can become stale when parameters, buffers, or child modules are replaced. - Calling `functional_call` on the wrapped module dispatches module hooks a second time. - CPU-to-device copies used `non_blocking=True` even when CPU memory was not pinned. The wrapper now stays installed during every call. A private proxy invokes the saved original forward while `functional_call` supplies the current parameters and persistent buffers. Eager calls use a per-module reentrant lock because functional reparameterization is temporary. Dynamo capture bypasses the lock and traces the functional call. The copy mode now follows the actual pin-memory setting. ## Duplicate check This does not duplicate the other open offloader changes: - #43453 keeps externally cached quantization scales on the accelerator. It does not change fallback invocation or state resolution. - #48468 wires the offloader into the V2 model runner. It does not change `UVAOffloader`. - #41496 enables pinned memory on WSL2 so that WSL can use UVA. This PR fixes the fallback used when UVA remains unavailable or disabled. - Closed #29054 addressed the `state_dict()` trace failure, but not exception recovery, replaced state, hook behavior, or concurrent eager calls. ## Validation Human review and test: ```text .venv/bin/python -m pytest -q tests/model_executor/test_uva_offloader.py 10 passed ``` Repository checks: ```text pre-commit run --files tests/model_executor/test_uva_offloader.py vllm/model_executor/offloader/uva.py all applicable hooks passed ``` RTX 5090 checks also passed for in-place fullgraph compilation, recursive calls, CUDA Graph capture and replay, non-default stream copies, and 200 compiled calls across eight threads. Parameters remained on CPU between calls. A model evaluation was not run because this patch does not change model math or output selection. The focused suite compares outputs across eager and compiled calls and covers post-load state replacement. ## AI assistance OpenAI Codex assisted with the implementation and test preparation. The human submitter reviewed every changed line and ran the focused test suite before the branch update.", "url": "https://github.com/vllm-project/vllm/pull/46690", "createdAt": "2026-06-25T04:32:40Z", "updatedAt": "2026-08-13T06:56:50Z", "timestamp": "2026-08-13T06:56:50Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug" ], "author": "wgu9", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:cab2cf8fb6d5cd59baef", "signalId": "github:vllm-project/vllm:issue:52113", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52113", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Session-Aware KV Cache Hints for Agentic Workloads", "text": "### Motivation. Agentic applications often know more about future KV cache reuse than the inference engine: - a child agent is waiting for a tool and will probably resume soon; - several agents share a long system/tool prefix; - a branch has completed and its KV cache can be released; - an offloaded branch will be resumed and can be prefetched. Today, after a request finishes, these prefixes are ordinary free cached blocks managed by the same replacement policy. vLLM cannot distinguish a temporarily idle but valuable agent context from a prefix that will never be used again. This RFC proposes optional `agent_hint` metadata so an application can communicate session identity, a short retention TTL, and explicit cache management operations. The goal is to improve prefix-cache hit rate and TTFT for multi-turn and multi-agent workloads. The proposal addresses an information gap rather than replacing vLLM's cache policy. An agent runtime knows when a tool call is outstanding, a sub-agent has finished, a conversation has been suspended, or a checkpoint is about to be resumed. vLLM knows which KV blocks are resident and when capacity is needed. Neither side can make the best decision with only its own information. Hints provide a narrow, best-effort contract between those two layers. ### Workload scenarios and expected value The largest opportunity is not generic chat traffic, but workloads with long prefixes and explicit lifecycle transitions: | Application event | Hint/action | Expected system value | | --- | --- | --- | | A long-running agent waits for a tool or user response | Protect the request's cache for a short TTL, or offload it for a longer pause | Avoid an accidental LRU eviction and repeated prefill when the agent resumes | | A leader creates multiple workers from a common context | Associate parent and child sessions; protect the active branches | Preserve normal content-based sharing of the common prefix while retaining each branch only as long as it is useful | | A child agent completes or a failed branch is abandoned | Evict that session's cached context | Reclaim capacity earlier than an engine-only timeout or replacement policy can | | An application switches, resumes, or rewinds a session | Prefetch the target context before inference | Overlap KV movement with application state transition and reduce resume TTFT | | Context is compacted or archived | Evict obsolete cache, or offload context that may be restored | Align physical KV residency with the application's actual context lifecycle | | A session moves between instances in pooled/disaggregated serving | Retain or prefetch the corresponding remote KV entries | Improve cross-instance reuse and avoid recomputing a long prefix after routing changes | These actions target three concrete outcomes: - **Lower and more stable TTFT:** fewer long-prefix cache misses and an opportunity to hide remote-load latency behind tool execution, scheduling, or session switching. - **Higher effective KV capacity:** dead branches can be released promptly, while temporarily idle but expensive contexts receive bounded protection. The same HBM budget can therefore support a more useful working set, which may translate into more concurrent agent contexts or less capacity needed for the same service target. - **Less recomputation and data movement:** the application can distinguish a likely resume from a terminal state, allowing vLLM to spend prefill compute and connector bandwidth on contexts with known reuse value. The exact gain depends on prefix length, pause duration, cache pressure, and backend bandwidth. This RFC therefore treats these as benchmark hypotheses, not guaranteed improvements. The rollout section proposes measuring hit rate, recomputed prompt tokens, TTFT, throughput, and transfer volume. This is a cache-policy hint, not a new cache identity: - block hashes and prefix-cache matching remain unchanged; - identical content can still be reused across sessions; - `session_id` is not included in the cache key; - vLLM may ignore or override hints under memory pressure; - requests without hints behave exactly as they do today. ### Goals - Retain likely-to-be-reused cache blocks for a bounded time. - Release known-dead session context promptly. - Allow optional offload, prefetch, and evict operations. - Keep session policy separate from `BlockPool`. - Add negligible overhead to requests that do not use the feature. - Support local cache first and optional CPU/remote KV backends later. - Let applications express useful lifecycle intent without exposing vLLM block size, block tables, or physical block IDs. ### Proposed Change. ### 1. Initial request-level implementation The complete design can describe subranges, but the implementation currently being validated intentionally starts with a simpler profile: protection, offload, prefetch, and eviction operate on a whole request context. The agent runtime does not calculate or send block indices. For a normal inference request, vLLM associates `session_id` with the complete cacheable prefix materialized for that request. If `cache_control` is present, that complete association receives the requested TTL. For a pure management request, the operation applies to the complete context currently recorded for `session_id`. ```python class AgentHintParams(OpenAIBaseModel): session_id: str | None = None parent_session_id: str | None = None cache_control: CacheControlParams | None = None context_management: ContextManagementParams | None = None class CacheControlParams(OpenAIBaseModel): type: Literal[\"ephemeral\"] = \"ephemeral\" ttl: float = Field(default=300.0, ge=0, le=3600) # Optional fine-grained extension. None protects the whole request context. block_offset: int | None = Field(default=None, ge=0) class ContextManagementParams(OpenAIBaseModel): manage_request: bool = False edits: list[ContextEdit] = Field(default_factory=list) class ContextEdit(OpenAIBaseModel): type: Literal[\"offload\", \"prefetch\", \"evict\"] # Optional fine-grained extension. Both None target the whole request context. block_start: int | None = Field(default=None, ge=0) block_end: int | None = Field(default=None, ge=0) # exclusive ``` Protect a request context: ```json { \"model\": \"example-model\", \"messages\": [{\"role\": \"user\", \"content\": \"Inspect the repository.\"}], \"agent_hint\": { \"session_id\": \"worker-7\", \"parent_session_id\": \"orchestrator-1\", \"cache_control\": {\"type\": \"ephemeral\", \"ttl\": 300} } } ``` Prefetch a recorded session before it resumes: ```json { \"model\": \"example-model\", \"messages\": [{\"role\": \"user\", \"content\": \"\"}], \"agent_hint\": { \"session_id\": \"worker-7\", \"context_management\": { \"manage_request\": true, \"edits\": [{\"type\": \"prefetch\"}] } } } ``` This initial request-level implementation has several practical advantages for a vLLM integration: - the public API does not expose allocator-specific block indices; - all mapping work occurs at existing request allocation/completion boundaries; - validation can focus on lifecycle semantics and measurable benefit before adding range-selection complexity; - the implementation remains useful for the common cases of suspend, resume, branch completion, and session deletion. Internally vLLM still operates on blocks, but that is an implementation detail: SAM resolves the recorded request/session association to the current physical blocks and validates the expected hashes before changing metadata. In other words, the application is request-aware while vLLM remains block-aware. ### 2. Keep fine-grained fields, but defer their use in the initial implementation `block_offset`, `block_start`, and `block_end` remain in the request model as optional fields. They preserve the complete design and provide a compatible path to fine-grained management after the request-level workflow is validated. The current tests simply omit them: - `cache_control.block_offset=None` protects the whole cacheable request context; - `ContextEdit.block_start=None` and `block_end=None` target the whole recorded request context; - when fine-grained support is enabled, `block_start` and `block_end` must be supplied together and describe a session-relative half-open logical range; - a supplied `block_offset` protects the cacheable blocks from start to that logical offset of the request. These values are logical indices, never physical block IDs. Therefore retaining the fields does not require the simplified client workflow to understand vLLM's allocator or physical block table. When `context_management.manage_request` is true, vLLM skips model execution and applies the requested edits. With the default value `false`, edits run after the normal request completes. In the actual running process, message to token to block index translation is implemented by the upper-layer routing, or the parsing can be placed in the VLLM. ### 3. Add a small SessionAwareManager Add a `SessionAwareManager` (SAM) beside `KVCacheManager`. SAM owns: - session and optional parent/child relationships; - session-to-block and block-to-session indexes; - block-level TTL entries; - validation and execution of context-management edits. `BlockPool` continues to own physical block allocation, the free queue, and the prefix hash map. It does not store session IDs or session trees. ```mermaid flowchart LR API[\"Request + agent_hint\"] --> Scheduler[\"V1 Scheduler\"] Scheduler --> KVCM[\"KVCacheManager\"] Scheduler --> SAM[\"SessionAwareManager\"] KVCM --> Pool[\"Existing BlockPool\"] SAM -->|\"one metadata API\"| KVCM SAM -. \"optional lifecycle operations\" .-> Connector[\"KV connector capability\"] Connector -.-> Backend[\"CPU / remote KV store\"] ``` SAM records a logical block index and expected block hash with each association. ### 4. Add two aggregate fields to KVCacheBlock ```python @dataclass(slots=True) class KVCacheBlock: # Existing fields remain unchanged. _session_ref_cnt: int = 0 _ttl_expire_at: float = 0.0 ``` `_session_ref_cnt` is independent of the existing active-request `ref_cnt`. Detailed session IDs stay in SAM, so the additional per-block memory is constant and prefix-cache lookup does not need a session parameter. ### 5. Add one KVCacheManager metadata API SAM does not modify blocks directly. It uses one guarded method: ```python def update_block_meta( self, *, block_id: int, expected_block_hash: BlockHashWithGroupId | None = None, session_ref_delta: int = 0, ttl_expire_at: float | None = None, ) -> BlockMetaUpdateResult: ... ``` The method validates that: - the physical block still contains the expected cached content; - `_session_ref_cnt` cannot become negative; - SAM never changes active-request `ref_cnt`; - null blocks are not retained. If the block is free, the method also asks the free queue to reconsider its allocation priority. `KVCacheManager` only needs to notify SAM at existing request boundaries: 1. after cached/new blocks are assigned to a request; 2. when a request completes or a block is reassigned. ### 6. Best-effort TTL retention Free blocks are logically classified as: | Class | Condition | Allocation order | | --- | --- | --- | | A | TTL expired and no session reference | First | | B | TTL expired and has session references | Second | | C | TTL not expired | Last resort | This can be implemented by extending the existing free linked list with two boundaries; it does not require a new cache allocator. ```text [A: ordinary free] -> [B: session-associated] -> [C: TTL-protected] ``` TTL is deliberately soft: 1. allocate A blocks first; 2. then allocate B blocks; 3. if only C blocks remain and inference needs memory, break a C reservation. This guarantees that inaccurate hints cannot block scheduling or cause an avoidable out-of-memory failure. #### TTLManager and timer wheel SAM owns a `TTLManager` backed by a fixed-size timer wheel. Entries are keyed by `(block_id, session_id)`: ```python @dataclass class TTLBlockEntry: block_id: int session_id: str expire_at: float ``` `register()` is invoked when a block with a TTL is allocated or the TTL of a block is updated, and notifies the remote backend to keep the block alive of the corresponding session. `remove()` cancels an entry during evict or session cleanup, and notifies the remote backend to stop the block keepalive for the corresponding session. On every scheduler iteration, `tick(now)` advances elapsed wheel slots and calls `SAM.on_ttl_expired(block_id, session_id)` for truly expired entries. `on_ttl_expired` will call remove to delete the entry. The callback removes the session record, decrements the aggregate session reference, recomputes the block's latest remaining deadline, and calls `update_block_meta()`. Thus, expiration of one session does not remove a later TTL owned by another session. With a one-second tick and the proposed one-hour maximum TTL, the wheel has 3,600 slots. Registration, refresh, and removal are O(1); each tick processes only entries in elapsed slots. `time.monotonic()` is used throughout. Allocation also checks the block deadline, so scheduler delay or a skipped tick cannot prevent expired C-zone blocks from being reclaimed. ### 7. Session lifecycle and management operations Session registration is lazy and idempotent. Reusing a session refreshes its associations without double-incrementing `_session_ref_cnt`. `free_session(session_id)` removes that session's records and TTL entries. `free_session_tree(session_id)` optionally releases children before their parent. Neither operation invalidates blocks used by active requests or other sessions. In the initial implementation, management edits resolve to all blocks in the recorded request context for the target session. A later fine-grained extension may use session-relative half-open logical block ranges: - `evict`: cancel selected local and remote retention. - `offload`: persist selected KV data remotely, then release its local retention. - `prefetch`: load selected remote KV data into the local cache. Active request references always win. An edit never invalidates a block while its existing `ref_cnt` is positive. Offload and prefetch are capability based. If no backend supports an operation, vLLM returns `unsupported`; the local prefix-cache behavior remains valid. ### 8. Pooling and remote KV stores Local cache management is the first milestone. Pooled CPU or remote KV stores need the same lifecycle semantics, but the integration should not make vLLM core understand backend-specific PoolKeys, transfer threads, or eviction policies. #### Compatibility path for current connectors For connectors that only expose their existing lookup/save/load paths, an optional `SessionAwarePoolingManager` (SPM) can act as an adapter. It listens to SAM lifecycle events and maintains a temporary mapping: ```text session_id -> block hashes / connector keys ``` After a connector finishes saving KV data, SPM associates the saved keys with the session. It can then: - refresh active or TTL-protected keys at low frequency when the backend's `exists` operation also refreshes LRU state; - stop keep-alive after a session or key becomes orphaned, allowing the backend's existing LRU policy to reclaim it; - submit a bounded, low-priority load through the existing connector for prefetch, while reserving local capacity for normal inference; - confirm that offload/save completed before local retention is released. This adapter provides a way to test end-to-end semantics without requiring an immediate change to every connector or backend. It is deliberately optional and is not involved in local block allocation or prefix matching. ```mermaid flowchart LR Edit[\"offload / prefetch / evict\"] --> SAM[\"SAM: resolve logical blocks\"] SAM --> Local[\"KVCacheManager: local metadata\"] SAM --> SPM[\"SPM: resolve PoolKeys\"] SPM --> Tracker[\"SessionKeyTracker\"] SPM --> Queue[\"Bounded prefetch / eviction queue\"] Queue --> Store[\"CPU or remote KV store\"] ``` ### 9. Expected vLLM change surface The initial request-level implementation keeps the required vLLM core change small and localized: | Area | Change | | --- | --- | | Request protocol/internal request | Add optional request-level `agent_hint` fields | | New SAM module | Session indexes, TTL, edit controller | | `KVCacheBlock` | Add two aggregate metadata fields | | `KVCacheManager` | Add one metadata method and lifecycle notifications | | Free-block queue | Add metadata-based priority/reclassification | | Scheduler | Construct SAM, tick TTL, dispatch management requests | Fine-grained ranges and pooled-cache management are separate, optional layers: | Optional area | Change | | --- | --- | | Logical range support | Translate stable logical ranges only if the community chooses to expose them | | Compatibility SPM | PoolKey tracker, keep-alive, and remote operation queues for current connectors | | Connector capability | Propagate logical session/request identity and expose remote observation/control | | Remote backend | No required change; precise TTL/delete remains optional | The proposal does **not** change: - block hash calculation; - prefix-cache matching; - cache keys or `cache_salt`; - attention kernels; - model runners; - block tables sent to workers; - sampling or generated results; - existing connector behavior when no session-aware capability is enabled. SAM, range support, and pooled-cache integration should have separate experimental flags. With them disabled, the current cache and connector paths remain unchanged. In particular, connector support is not a prerequisite for reviewing or merging the local request-level path. ### Feedback Period. _No response_ ### CC List. @JuneHM @fc-liu @luokui183 @wangxiaochao6 @LookAround0301 ### Any Other Things. ### Co-Authors @HiC4Sh1e @JiahongZhang-Work @Xavier-Zeng @socrahow @liudi60 ### Results Based on the SWE-bench Verified data set and Qwen3-235B model, openJiuwen did a series of comparative tests: covering typical scenarios such as Bug fixes, feature development, and code refactoring, simulating 10 users concurrently using JiuwenSwarm to perform tasks, comparing the effect with and without computing affinity. The results show that the first token delay (TTFT) is reduced by 57.46% , the model request end-to-end (E2E) delay is reduced by 27.61% , the Prefix Cache hit rate is increased by 33% , and the peak Cache usage is reduced by 25.24% . \"Image\" Additionally, we also conducted a simple test without the agent framework. Experimental configuration: 20 sessions, 5 concurrent requests, request length 10-30K. By sending requests directly to vLLM. We observed that the TTFT was optimized by approximately 21%. | Metrics | Agent Hint OFF | Agent Hint ON | Agent Hint ON + Evict | | --- | --- | --- | --- | | TTFT (average) | 4.52 s | 3.58 s | 3.57 s | | TTFT (P50) | 4.57 s | 3.47 s | 3.57 s | | TPOT (average) | 0.030 s | 0.034 s | 0.032 s | | TPOT (P50) | 0.029 s | 0.034 s | 0.032 s | ### Backward compatibility and safety Requests without `agent_hint` behave exactly as before, and hints cannot affect generated tokens. `session_id` is not authorization: management operations must be tenant-scoped, while `cache_salt` continues to provide cache isolation. TTL, sessions, records, edit ranges, and prefetch work must be bounded; session IDs must not be metric labels. ### Performance and observability The target overhead is two scalars per block, O(1) reclassification, and work proportional to associated blocks at request boundaries. There is no session lookup during prefix matching or per-token decode work. Remote tracking and control run only when both a connector and session hints are enabled. Suggested metrics include: - sessions, records, protected blocks, and pressure overrides; - edit/prefetch results by operation and status; - tracked/shared/orphaned keys, keep-alive, and remote eviction counts; - connector transfer bytes, queue delay, completion latency, and failures; - prefix-cache hit rate and TTFT for hinted versus unhinted requests. ### Rollout and testing The change can be split into small PRs: 1. request schema and shadow SAM accounting; 2. local TTL retention and session cleanup; 3. local evict and structured management responses; 4. SPM with PoolKey tracking and keep-alive; 5. optional connector offload/prefetch/explicit-evict capabilities. Benchmarks should compare prefix-cache hit rate, recomputed prompt tokens, TTFT, throughput, scheduler CPU time, and metadata memory overhead against the current LRU baseline. ### Future direction #### Planned direct connector integration The longer-term integration should make session-aware management an optional connector capability instead of relying on SPM to infer remote state from side-effects. A capable connector is the component that already knows how local block hashes map to remote keys, whether data is resident, and when an asynchronous transfer has completed. It is therefore the right boundary for both observing and controlling remote KV state. The exact interface should be developed with connector maintainers, but the capability would cover operations equivalent to: ```python class SupportsSessionAwareKVManagement(Protocol): def register_cache(self, session_id, request_id, block_hashes) -> None: ... def offload(self, session_id, request_id=None) -> OperationHandle: ... def prefetch(self, session_id, request_id=None) -> OperationHandle: ... def evict(self, session_id, request_id=None) -> OperationHandle: ... def release(self, session_id, request_id=None) -> None: ... def query_residency(self, session_id, request_id=None) -> Residency: ... ``` The names and argument types above are illustrative, not a proposed final API. The important properties are: - **direct visibility:** connectors report save/load completion, remote residency, misses, and backend pressure through structured results or events; - **direct control:** offload, prefetch, retain/release, and explicit eviction use connector methods when supported, rather than scheduler-created virtual requests or assumed `exists()` side-effects; - **logical inputs:** core passes a session/request association and content hashes; connector-specific keys remain private to the connector; - **capability negotiation:** unsupported operations return `unsupported` and never change normal request execution; - **backend freedom:** a connector may implement exact TTL/delete, map release to stopped keep-alive, or rely on backend LRU according to its capabilities. ```mermaid flowchart LR Hint[\"request-level hint\"] --> SAM[\"SAM: logical lifecycle\"] SAM --> Local[\"KVCacheManager: local policy\"] SAM --> API[\"optional connector capability\"] API --> Observe[\"residency and completion events\"] API --> Control[\"offload / prefetch / evict / release\"] Observe --> Store[\"pooled CPU or remote KV store\"] Control --> Store ``` AscendStore can be an initial integration by propagating optional `session_id` and request identity in connector metadata, reporting saved block hashes after put completion, and reusing its lookup/save/load paths. As connector capability methods mature, its adapter-specific tracking can move behind the connector boundary. Other KV connectors can opt in independently; none of these remote capabilities are required for the initial local implementation. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52113", "createdAt": "2026-08-13T06:54:55Z", "updatedAt": "2026-08-13T06:54:55Z", "timestamp": "2026-08-13T06:54:55Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [ "RFC" ], "author": "FermatGo", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:571190f9045d49621d2c", "signalId": "github:vllm-project/vllm:pull_request:50729", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50729", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Mamba] Fix overlapping state copy race", "text": "[PR #30877](https://github.com/vllm-project/vllm/pull/30877) introduced generic Mamba state copies, and [PR #40172](https://github.com/vllm-project/vllm/pull/40172) added the fused GPU copy. A speculative-decode convolution-state shift can copy within the same physical block with overlapping source and destination ranges. Parallel memcpy-style loads/stores do not provide memmove ordering, which explains the intermittent [AMD CI failure](https://buildkite.com/vllm/amd-ci/builds/11556/list?sid=019fbc8d-6c5e-4172-83cb-f45a49946645&tab=output). This version is rebased over [PR #49436](https://github.com/vllm-project/vllm/pull/49436): - Temporal state copies retain #49436's 3D grid and 16-way tiled copy. - DS convolution copies use stable row-to-lane ownership while iterating tokens low to high. - SD convolution copies retain `_memcpy_u64_tiled(..., NUM_TILES=1)` for distinct physical blocks and exact self-copies, and use an ordered token copy only for a true same-physical-block left overlap. - Conv copies remain single-CTA (`tile_idx > 0` returns), so no grid-wide barrier is assumed or required. - The generic fallback is unchanged. MRv1 and MRv2 already share this fused copy kernel for the relevant hybrid/speculative/aligned path; routing every MRv1 configuration through the MRv2 lifecycle would break unsupported/custom state layouts such as ReplaySSM and would not cover the CPU fallback. - Existing overlap assertions derive expected values from untouched snapshots. The broad Python-vs-fused golden uses disjoint pages so the generic memcpy helper is not treated as a memmove oracle. ## Validation - `HIP_VISIBLE_DEVICES=0 .venv/bin/python -m pytest -q -s tests/v1/worker/test_mamba_utils.py`: **37 passed** - `uvx pre-commit run --files vllm/v1/worker/mamba_utils.py tests/v1/worker/test_mamba_utils.py`: **passed** - The original failing revision reproduced locally on MI355; the rebased implementation passed the same snapshot checks. - Qwen3.5-0.8B MTP checks across 8×MI355, MRv1/MRv2, SD/DS, and batch 1/8 produced matching output hashes. A counterbalanced device-kernel benchmark on MI355 covered both runners, SD/DS, same-page and distinct-page copies, and batch 1/8. For a 24-layer Qwen3.5-shaped state copy: | Path | Batch | Result vs current main | |---|---:|---:| | SD same-page, MRv1 | 1 / 8 | +14.7 µs / +1.1% | | SD same-page, MRv2 | 1 / 8 | +18.0 µs / +1.2% | | SD distinct-page, both runners | 1 / 8 | -2.5% / +1.7–1.8% | | DS, both runners and mappings | 1 / 8 | 95.5–98.5% lower latency | The SD same-page cost is the overlap-safe ordering; the common distinct-page path retains the tiled memcpy. MRv1 and MRv2 copy-kernel latency is effectively equivalent because they already execute the same fused mechanism. This PR was developed with assistance from OpenAI Codex.", "url": "https://github.com/vllm-project/vllm/pull/50729", "createdAt": "2026-08-02T05:57:32Z", "updatedAt": "2026-08-13T06:54:33Z", "timestamp": "2026-08-13T06:54:33Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "bug", "ready" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a2b827ea7c3400af1365", "signalId": "github:vllm-project/vllm:issue:52107", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52107", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain", "text": "# NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain ## Summary Native MTP speculative decoding for Nemotron-3.5-Lightning-30B-A3B produces **0% draft acceptance** on Intel Arc Pro B70 (Xe2/Battlemage) via vLLM XPU nightly. After extensive debugging (9+ GPU cycles, 3 identified bugs patched, 7 instrumentation attempts), we found the V1 speculator's execution path **completely bypasses** the standard sampling/verification chain (`_process_engine_step`, `post_step`, `sample`, `RejectionSampler.__call__`). Drafts are proposed (spec counters increment) but never verified. ## Environment - **Hardware**: Intel Arc Pro B70 (32 GB GDDR6, Xe2/Battlemage, 256 XMX, `xe` driver) - **Image**: `vllm/vllm-openai-xpu@sha256:1da0a95485455f08588c11080b9718992fd7d434c6a965d74654903a9d999c57` - **vLLM**: `0.26.1rc1.dev668+g3ee2df303` - **vllm-xpu-kernels**: `0.1.12.3` (with grouped-GEMM `at::zeros` fix — see below) - **Model**: Nemotron-3.5-Lightning-30B-A3B, local symmetric GPTQ INT4 G64 conversion (base model from `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`; published at `SergiiioB/Nemotron-3.5-Lightning-30B-A3B-GPTQ-INT4-G64-sym`) - **Config**: `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`, `--enforce-eager`, `--no-enable-prefix-caching` ## Three identified bugs (all patched, still 0% acceptance) ### 1. Double-norming of hidden states `NemotronHModel.forward` applies `norm_f` (final RMSNorm) to hidden states BEFORE passing to the MTP draft. The draft's `NemotronHMTPAttentionDecoderLayer` applies its OWN `hnorm` → double-normalized → garbage draft attention input. **Fix**: `patch_nemotron_prenorm_mtp.py` — saves pre-norm state (`hidden_states + residual`) and exposes via `get_mtp_target_hidden_states()`. ### 2. Concatenation order mismatch The MTP fusion layer concatenates `[inputs_embeds, hidden_states]` but MTP/EAGLE architectures may train with reversed order. **Fix**: `patch_concat_order.py` — swaps to `[hidden_states, inputs_embeds]`. ### 3. Non-gated MoE w13 initialization NemotronH uses a non-gated MoE (no `gate_proj`). The fused w13 mapping in `routed_experts.py` warns `\"Unexpected gate/up projection names: up_proj, .\"` and skips fused mapping. The XPU WNA16 expert backend handles this correctly (its own loading path), but the torch FusedMoE path (used by unquantized drafts) leaves expert weights uninitialized. ## The critical finding: V1 execution path bypass After patching all three bugs and testing across 5 GPU-cycle combinations (INT4/BF16 × prenorm/concat/control — ALL 0% acceptance), we instrumented the V1 pipeline with file-based logging (baked into a custom Docker image, confirmed via grep): | Method | File | Reached? | |---|---|---| | `EngineCore.__init__` | `engine/core.py` | ✅ (file write confirmed) | | `_process_engine_step` | `engine/core.py:1442` | ❌ NEVER executed | | `post_step` | `engine/core.py:615` | ❌ NEVER executed | | `step` | `engine/core.py:583` | ❌ NEVER executed | | `model_runner.sample` | `model_runner.py:1227` | ❌ NEVER executed | | `RejectionSampler.__call__` | `rejection_sampler.py:233` | ❌ NEVER executed | | `rejection_sample` | `rejection_sampler_utils.py:922` | ❌ NEVER executed | | `DraftTokensHandler.get_draft_tokens` | `spec_decode/utils.py` | ❌ NEVER executed | **All patches confirmed present in source (grep verified) and the EngineCore subprocess loads from `/workspace/vllm/` (confirmed via `__init__` file write).** Yet NONE of the inference-time methods execute during MTP spec decode. The spec counters DO increment (`spec_decode_num_drafts_total: 31`, `spec_decode_num_draft_tokens_total: 63`, `spec_decode_num_accepted_tokens_total: 0`), confirming the speculator IS running — but through a completely non-standard execution path. ## Bonus: grouped-GEMM atomic race fix (already applied) During this investigation, we found and fixed a **one-word bug** in `vllm-xpu-kernels` that caused non-deterministic output in XPU graph mode: **File**: `csrc/xpu/grouped_gemm/xe_2/grouped_gemm_xe2_interface.hpp` ```diff - at::Tensor atomic_buffer = at::empty({static_cast(1)}, ...); + at::Tensor atomic_buffer = at::zeros({static_cast(1)}, ...); ``` The persistent block-scheduler's atomic counter was allocated with `at::empty` (uninitialized). SYCL doesn't guarantee `group_id == 0` launches first, so other groups `atomicAdd` into garbage → inflated counters → out-of-bounds tiles → non-deterministic MoE output. In XPU graph mode the dirty buffer is captured and guaranteed to poison every replay. **Impact**: This fix enabled deterministic XPU graph capture for NemotronH, raising decode from 21.8 → 92.65 t/s (4.25×) with byte-identical temperature-0 replay. The fix has zero performance cost. ## Request 1. **Investigate the V1 MTP speculator execution path** — why does the standard `_process_engine_step` → `step_fn` → `post_step` chain not execute during MTP spec decode? What alternative path does the speculator use? 2. **Verify the three identified bugs** — are they real issues on other platforms (CUDA)? 3. **Consider the `at::zeros` grouped-GEMM fix** for upstream merge (it affects all XPU users, not just NemotronH) ## Repro ```bash docker pull vllm/vllm-openai-xpu@sha256:1da0a95485455f08588c11080b9718992fd7d434c6a965d74654903a9d999c57 # Apply patches: patch_nemotron_prenorm_mtp.py, patch_xpu_grouped_topk_native_v2.py # (available at https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook/tree/main/patches) vllm serve /model --dtype bfloat16 --quantization gptq \\ --max-model-len 16384 --gpu-memory-utilization 0.90 --enforce-eager \\ --no-enable-prefix-caching \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}' # Send any completion request → check /metrics: # spec_decode_num_accepted_tokens_total = 0 (should be >0) ```", "url": "https://github.com/vllm-project/vllm/issues/52107", "createdAt": "2026-08-13T06:29:39Z", "updatedAt": "2026-08-13T06:50:38Z", "timestamp": "2026-08-13T06:50:38Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "intel-gpu", "quantization" ], "author": "SergiioB", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:b4125d8e9e6a9eb8430d", "signalId": "github:vllm-project/vllm:pull_request:51797", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51797", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Match Qwen3.5 GDN BF16 semantics", "text": "## Summary - preserve activation-dtype Q/K normalization and beta semantics for Qwen3.5 BF16 activations - cover fused prefill, generic/speculative decode, and packed decode with the same model-scoped gate - propagate the gate through Triton warmup so compiled variants match runtime selection - add sensitivity tests for prefill and both incremental-decode paths Fixes #51779. ## Root cause The Qwen3.5 GDN CUDA kernels widened BF16 Q/K and beta inputs to FP32 before normalization or sigmoid and only rounded when storing the final result. The official Transformers formula returns BF16-rounded intermediate tensors, so the fused kernels implemented a different numerical function. This change introduces explicit BF16 rounding boundaries around the square, reduction result, epsilon addition, inverse square root, final Q/K product, and beta sigmoid. The gate is enabled only by Qwen3.5 and only for BF16 inputs; existing behavior remains the default for other callers and FP32 inputs. ## Standalone reproducer The reproducer is intentionally **not included in this PR's repository diff**. It is available as a separate downloadable script: - [View or download `reproduce_github_issue_51779.py`](https://gist.github.com/mrjoema/18b0005b93e6fdeaad8f52be723b289d) ```bash curl -L \\ https://gist.githubusercontent.com/mrjoema/18b0005b93e6fdeaad8f52be723b289d/raw/042da542f90f32c8746c4083354358426b7a4517/reproduce_github_issue_51779.py \\ -o /tmp/reproduce_github_issue_51779.py # On upstream main: .venv/bin/python /tmp/reproduce_github_issue_51779.py --expect buggy # On this PR branch: .venv/bin/python /tmp/reproduce_github_issue_51779.py --expect fixed ``` The fixed expectation requires exact zero mismatch for Q, K, and beta against the Transformers BF16 formula. ## Validation Passed locally: - repository pre-commit hooks on all nine changed files - Ruff lint and formatting checks - Python bytecode compilation for all changed Python files - `git diff --check` Passed on an NVIDIA A100-SXM4-40GB: ```console $ .venv/bin/python scripts/reproduce_github_issue_51779.py --expect fixed GPU: NVIDIA A100-SXM4-40GB seed=51779, tokens=64, dtype=torch.bfloat16 relative L2 error exact mismatch rate BF16 reference FP32 promoted BF16 ref FP32 promoted q: 0.000000e+00 2.915142e-03 0.00% 27.87% k: 0.000000e+00 2.925246e-03 0.00% 28.09% beta: 0.000000e+00 1.559851e-03 0.00% 100.00% FIXED: all outputs exactly match BF16 Transformers semantics. ``` ## Duplicate-work check No open PR references #51779. PR #49953 is an unrelated ROCm/AITER long-prefill split-QKV optimization and does not address Qwen3.5 CUDA BF16 semantics. ## AI assistance AI assistance was used to develop this change. Before marking the PR ready, the human submitter will review every changed line and run the pending GPU and model evaluations.", "url": "https://github.com/vllm-project/vllm/pull/51797", "createdAt": "2026-08-11T10:01:04Z", "updatedAt": "2026-08-13T06:50:19Z", "timestamp": "2026-08-13T06:50:19Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen", "verified" ], "author": "mrjoema", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5beb25b9e353a958a7e8", "signalId": "github:vllm-project/vllm:pull_request:49086", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49086", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][feature] Add new moe backends supporting int4/int8 weight-only…", "text": "## Purpose 1. Add Int8EmulationTritonExperts and TritonWNA16OTFExperts to EMULATION MoE backend to support int4/int8 quantization for WNA16; 1.1 Int8EmulationTritonExperts: dequant at load time; 1.2 TritonWNA16OTFExperts: dequant on-the-fly, supporint sym/asym/static/dynamic int4 quantization schemes, and also int8 quantization; 1.3 TritonWNA16OTFExperts is added to replace legacy MoeWNA16Method (same functionality with different structure to enable LoRA/EP/DP); 2. Update moe backend selection logic to honor --moe-backend cmd option for WNA16 case for ROCm branch", "url": "https://github.com/vllm-project/vllm/pull/49086", "createdAt": "2026-07-19T09:39:15Z", "updatedAt": "2026-08-13T06:47:05Z", "timestamp": "2026-08-13T06:47:05Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "needs-rebase", "quantization" ], "author": "qli88", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:48fe7b827ee21d4d5ab0", "signalId": "github:vllm-project/vllm:pull_request:51280", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51280", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Solidify entrypoint LLM lifecycle", "text": "- Replace direct `LLM(...)` construction throughout `tests/entrypoints` with the shared `VllmRunner` lifecycle. - Add an `ExitStack`-backed runner factory for tests that need one long-lived runner or several concurrent runners. - Consolidate multimodal, structured-output, offline-mode, collective-RPC, pooling, and weight-transfer cleanup onto the complete runner shutdown path. - Defer ROCm VRAM settling while multiple runners coexist, then release runner references and perform one bounded wait after every engine exits. - Preserve direct-constructor behavior where required, including pooling model length and chunked-prefill defaults. This addresses AMD CI instability caused by entrypoint tests constructing `LLM` directly and relying on partial or ad hoc cleanup. Those paths could leave engine processes or lazily reclaimed ROCm allocations alive long enough to starve a later model startup. Routing construction through `VllmRunner` guarantees engine shutdown, Dynamo reset, distributed cleanup, and bounded VRAM settling even when a test fails. The multi-engine GPU-utilization case requires deferred settling because no individual runner can reach baseline while its peers remain live.", "url": "https://github.com/vllm-project/vllm/pull/51280", "createdAt": "2026-08-06T16:25:21Z", "updatedAt": "2026-08-13T06:45:00Z", "timestamp": "2026-08-13T06:45:00Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "structured-output" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3407573d5216e6b1c78e", "signalId": "github:vllm-project/vllm:pull_request:52112", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52112", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Fix a few int4/int8 quantization errors", "text": "## Purpose Fix a few quantization bugs on ROCm introduced by #44120 to enable int4/int8 quantized models like cyankiwi/MiniMax-M3-AWQ-INT4, QuantTrio/Qwen3-235B-A22B-GPTQ-Int8. ## Test Plan 1. VLLM_USE_BREAKABLE_CUDAGRAPH=0 vllm serve cyankiwi/MiniMax-M3-AWQ-INT4/ --block-size 128 -tp 4 --attention-backend TRITON_ATTN --tool-call-parser minimax_m3 --enable-auto-tool-choice --reasoning-parser minimax_m3 --moe-backend triton --max-model-len 32768 2. vllm serve QuantTrio/Qwen3-235B-A22B-GPTQ-Int8/ -tp 8 --moe-backend triton Both generate correct answers.", "url": "https://github.com/vllm-project/vllm/pull/52112", "createdAt": "2026-08-13T06:44:17Z", "updatedAt": "2026-08-13T06:44:55Z", "timestamp": "2026-08-13T06:44:55Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm", "quantization" ], "author": "qli88", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:915c333520240cd8a9f2", "signalId": "github:vllm-project/vllm:issue:50851", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50851", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][Spec Decode] DSpark speculative decoding broken on nightly", "text": "## Summary DSpark speculative decoding (`method: \"dspark\"`) is non-functional on current main. Multiple code paths assume only `\"dflash\"` when DSpark reuses the same proposer/speculator infrastructure (`DSparkSpeculator` extends `DFlashSpeculator`, both use `DFlashProposer`). The result is that DSpark either crashes at startup or produces wrong results at inference time. Tested on GLM-5.2-FP8 (753B MoE) with `RedHatAI/GLM-5.2-speculator.dspark` on multi-node Wide-EP H200 clusters. ## Bugs ### 1. `use_dflash()` excludes DSpark — model runner never creates `DFlashProposer` **File:** `vllm/config/speculative.py` (line ~1395) ```python def use_dflash(self) -> bool: return self.method == \"dflash\" # BUG: excludes \"dspark\" ``` `use_dspark()` exists (line ~1398) but is never checked in the model runner's drafter initialization elif chain (`gpu_model_runner.py`). DSpark falls through without getting `DFlashProposer` or `use_aux_hidden_state_outputs = True`, so the target model doesn't extract multi-layer aux hidden states. **Fix:** `return self.method in (\"dflash\", \"dspark\")` ### 2. `DFlashProposer.__init__` asserts `method == \"dflash\"` **File:** `vllm/v1/spec_decode/dflash.py` ```python assert vllm_config.speculative_config.method == \"dflash\" ``` Even after fixing #1, this assert rejects DSpark at proposer creation time. **Fix:** `assert method in (\"dflash\", \"dspark\")` ### 3. `llm_base_proposer.py` missing \"dspark\" in method lists **File:** `vllm/v1/spec_decode/llm_base_proposer.py` Three locations where DSpark needs to be included alongside DFlash: - **`model_returns_tuple()`** (~line 1015): exclusion list `(\"mtp\", \"draft_model\", \"dflash\")` missing `\"dspark\"` → `ValueError: too many values to unpack (expected 2)` at line ~591 - **`combine_hidden_states` gate** (~line 526): inclusion list `(\"eagle3\", \"dflash\")` missing `\"dspark\"` → aux hidden states not combined for the drafter - **`parallel_drafting_token_id` ValueError** — raises for unknown method instead of setting a default for DSpark ### 4. FA3 AOT scheduler head count mismatch (separate PR) Already addressed in #50694 — the FA3 AOT scheduler reads head counts from the target model config, but the draft model may have a different GQA ratio. ## Error Traces ### Without any hotfixes — `model_returns_tuple` crash: ``` ValueError: too many values to unpack (expected 2) File \"vllm/v1/spec_decode/llm_base_proposer.py\", line 591 ``` ### After adding \"dspark\" to `model_returns_tuple` but not fixing routing — `combine_hidden_states` dimension mismatch: ``` ValueError: DFlash drafter expects 30720 concatenated aux hidden features but received 6144 File \"vllm/model_executor/models/qwen3_dflash.py\", line 761 File \"vllm/v1/spec_decode/llm_base_proposer.py\", line 536 ``` (30720 = 5 layers × 6144 hidden_size — target model only outputs last hidden state because `use_aux_hidden_state_outputs` was never set) ## Environment - vLLM nightly (`vllm/vllm-openai:nightly`, v0.26.1rc1.dev255) - Model: `zai-org/GLM-5.2-FP8` + `RedHatAI/GLM-5.2-speculator.dspark` - Config: `--speculative-config {\"model\":\"...\",\"num_speculative_tokens\":5,\"method\":\"dspark\"}` - Multi-node Wide-EP with H200 GPUs, DP=16, EP enabled", "url": "https://github.com/vllm-project/vllm/issues/50851", "createdAt": "2026-08-03T10:28:17Z", "updatedAt": "2026-08-13T06:44:40Z", "timestamp": "2026-08-13T06:44:40Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [], "author": "elvircrn", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:574f8a9dbb404c0e1eec", "signalId": "github:vllm-project/vllm:pull_request:52079", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52079", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3] Add GEMM-RS for sequence parallelism", "text": "## Purpose Add GEMM-RS kernel for Blackwell, based on https://github.com/NVIDIA/cutlass/blob/dcf215a/examples/python/CuTeDSL/cute/blackwell/kernel/distributed/distributed_gemm_reduce_scatter_blackwell.py (`multimem.ld_reduce`) - Supports any value of M e.g. M=1023. However, only uses GEMM-RS when M>=128 since the kernel was not optimized for small/medium M. Only supports TP<=16, and requires all rank on the same NVLink domain. - The sharding behavior follows existing RS logic i.e. eac rank holds `ceil(M / world_size)`, with the exception of the last rank - Requires an opt-in flag `VLLM_KIMI_K3_GEMM_RS`, which enables GEMM-RS for O-proj and shared experts+dense MLP down-proj - Symmetric memory workspace (per-GPU): `max_num_batched_tokens x 7168 x 2 bytes` = 448 MiB for MNBT=32k. Confirmed in vLLM logs KV memory 39.99 GiB (before) -> 39.78 GiB (after) -> not much Initialization and runtime logic - Whether to initialize GEMM-RS: done in `maybe_init_gemm_rs()`, which also logs the reason if it fails. When `VLLM_KIMI_K3_GEMM_RS=0`, it doesn't do anything - At each layer's `__init__()`, we call `self.run_gemm_rs = get_gemm_rs().can_run(self.down_proj.weight)`. This is to further validate supported weight shapes and dtype - When the checks fail, we fallback to standard behavior. - In `forward()`, we check again with `should_run()`, which is the heuristics M>=128. The kernel supports any values of M, but right now the baseline is better for M<128 Though technically this can work with any SP in general, this PR only enables GEMM-RS for Kimi-K3. A future extension is to make this into GEMM-AR by adding `multimem.st` (all-gather) after `multimem.ld_reduce` (reduce-scatter). ## Microbenchmark `benchmarks/kernels/benchmark_kimi_k3_gemm_rs.py` in this PR. CUDA graph with rotating buffers. All benchmarks were done with GB300. ### TP4 Note: K=1536 is shared expert down-proj, K=3072 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 1536 | 48.13 | 46.27 | 38.74 | 1.242 | 1.195 | | 512 | 7168 | 1536 | 84.64 | 60.51 | 50.69 | 1.67 | 1.194 | | 2048 | 7168 | 1536 | 119.92 | 110.83 | 83.74 | 1.432 | 1.323 | | 8192 | 7168 | 1536 | 286.58 | 343.22 | 209.25 | 1.37 | 1.64 | | 32768 | 7168 | 1536 | 1044.66 | 1265.86 | 719.7 | 1.452 | 1.759 | | 128 | 7168 | 3072 | 52.37 | 48.27 | 43.38 | 1.207 | 1.113 | | 512 | 7168 | 3072 | 92.35 | 66 | 53.09 | 1.74 | 1.243 | | 2048 | 7168 | 3072 | 145.6 | 133.81 | 89.44 | 1.628 | 1.496 | | 8192 | 7168 | 3072 | 383.65 | 431.82 | 234.38 | 1.637 | 1.842 | | 32768 | 7168 | 3072 | 1402.05 | 1628.43 | 1001.49 | 1.4 | 1.626 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 1536 | 17.65 | 39.89 | 38.74 | | 512 | 7168 | 1536 | 21.52 | 50.94 | 50.69 | | 2048 | 7168 | 1536 | 36.54 | 90.37 | 83.74 | | 8192 | 7168 | 1536 | 104.14 | 191.68 | 209.25 | | 32768 | 7168 | 1536 | 401.31 | 652.77 | 719.7 | | 128 | 7168 | 3072 | 18.96 | 41.38 | 43.38 | | 512 | 7168 | 3072 | 28.21 | 50.48 | 53.09 | | 2048 | 7168 | 3072 | 59.25 | 90.56 | 89.44 | | 8192 | 7168 | 3072 | 195.92 | 196.91 | 234.38 | | 32768 | 7168 | 3072 | 763.71 | 650.18 | 1001.49 | ### TP8 Note: K=768 is shared expert down-proj, K=1536 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 768 | 49.47 | 44.93 | 40.08 | 1.234 | 1.121 | | 512 | 7168 | 768 | 70.29 | 56.58 | 47.46 | 1.481 | 1.192 | | 2048 | 7168 | 768 | 110.34 | 100.75 | 81.1 | 1.36 | 1.242 | | 8192 | 7168 | 768 | 264.43 | 300.98 | 202.27 | 1.307 | 1.488 | | 32768 | 7168 | 768 | 928.83 | 1109.26 | 697.5 | 1.332 | 1.59 | | 128 | 7168 | 1536 | 51.26 | 46.86 | 41.63 | 1.231 | 1.126 | | 512 | 7168 | 1536 | 72.22 | 62.64 | 49.82 | 1.45 | 1.257 | | 2048 | 7168 | 1536 | 121.71 | 109.84 | 82.08 | 1.483 | 1.338 | | 8192 | 7168 | 1536 | 306.91 | 347.92 | 209.04 | 1.468 | 1.664 | | 32768 | 7168 | 1536 | 1108.27 | 1289.57 | 706.02 | 1.57 | 1.827 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 768 | 17.3 | 39.2 | 40.08 | | 512 | 7168 | 768 | 18.61 | 49.44 | 47.46 | | 2048 | 7168 | 768 | 26.64 | 91.39 | 81.1 | | 8192 | 7168 | 768 | 60.18 | 209.14 | 202.27 | | 32768 | 7168 | 768 | 216.29 | 714.27 | 697.5 | | 128 | 7168 | 1536 | 17.22 | 39.66 | 41.63 | | 512 | 7168 | 1536 | 23.17 | 51.39 | 49.82 | | 2048 | 7168 | 1536 | 36.64 | 90.43 | 82.08 | | 8192 | 7168 | 1536 | 104.14 | 210.16 | 209.04 | | 32768 | 7168 | 1536 | 400.02 | 713.71 | 706.02 | ## E2E prefill-only benchmark All benchmarks were done with 8xGB300, TP8+EP+SP (DeepGEMM MegaMoE), `--max-num-batched-tokens 32768`, 8k input - 1 output requests. Baseline is 7aa248f Concurrency | Baseline TTFT (median) | GEMM-RS TTFT (median) | Baseline TPGS | GEMM-RS TPGS -- | -- | -- | -- | -- C1 | 312.81 ms | 298.44 ms (-4.59%) | 3,172.7 tok/GPU/s | 3,432.2 tok/GPU/s (+8.18%) C32 | 7,240.71 ms | 6,799.51 ms (-6.09%) | 4,500.5 tok/GPU/s | 4,794.0 tok/GPU/s (+6.52%) ## Test Plan Unit test (also added to distributed CI) ``` tests/kernels/test_kimi_k3_gemm_rs.py ``` E2E testing, TP8+EP+SP (DeepGEMM MegaMoE) - GSM8K: 96.82% - OCRBench: 88.40% ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/52079", "createdAt": "2026-08-13T01:15:53Z", "updatedAt": "2026-08-13T06:44:14Z", "timestamp": "2026-08-13T06:44:14Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "performance", "ci/build", "kimi", "k3" ], "author": "gau-nernst", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:fc8d9989c5554fdf1f6c", "signalId": "github:vllm-project/vllm:pull_request:42963", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42963", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunnerV2] Support prompt embeds", "text": "## Purpose Support prompt embeds for ModelRunnerV2. ## Test Plan ```bash VLLM_USE_V2_MODEL_RUNNER=1 pytest -sv tests/basic_correctness/test_basic_correctness.py::test_models -k \"True-uni or True-mp\" ``` Before ```bash E pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig E Value error, VLLM_USE_V2_MODEL_RUNNER does not yet support: prompt embeds [type=value_error, input_value=ArgsKwargs((), {'model_co... 'shutdown_timeout': 0}), input_type=ArgsKwargs] E For further information visit https://errors.pydantic.dev/2.13/v/value_error vllm/engine/arg_utils.py:2171: ValidationError ============================================================ warnings summary ============================================================ :488 :488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute :488 :488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute ../.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings /root/vllm-workspace/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================================================== short test summary info ========================================================= FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig ======================================== 8 failed, 8 deselected, 16 warnings in 237.83s (0:03:57) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` After ```bash ======================================== 8 passed, 8 deselected, 16 warnings in 443.73s (0:07:23) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ``` vllm serve --enable-prompt-embeds ``` ``` \"\"\"Smoke test for prompt_embeds over the OpenAI-compatible HTTP server. Usage: # Terminal 1 (server): see vllm serve command in the chat. # Terminal 2: .venv/bin/python test_embeds_serve.py \"\"\" import io import openai import pybase64 as base64 import torch from transformers import AutoModelForCausalLM, AutoTokenizer MODEL_PATH = \"/root/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B\" SERVED_NAME = \"Qwen/Qwen3-0.6B\" # must match --served-model-name BASE_URL = \"http://localhost:8000/v1\" PROMPT = \"The capital of France is\" def to_b64_embed(tensor: torch.Tensor) -> str: buf = io.BytesIO() # torch.save is what the server expects (it calls torch.load on the bytes). torch.save(tensor, buf) return base64.b64encode(buf.getvalue()).decode(\"utf-8\") def main() -> None: tok = AutoTokenizer.from_pretrained(MODEL_PATH) hf = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16) with torch.no_grad(): ids = tok(PROMPT, return_tensors=\"pt\").input_ids embeds = ( hf.get_input_embeddings()(ids) .squeeze(0) .to(torch.bfloat16) .cpu() .contiguous() ) del hf encoded = to_b64_embed(embeds) client = openai.OpenAI(base_url=BASE_URL, api_key=\"EMPTY\") # Case 1: prompt_embeds only. out = client.completions.create( model=SERVED_NAME, prompt=None, # leave empty so the server falls through to prompt_embeds max_tokens=16, temperature=0.0, extra_body={\"prompt_embeds\": encoded}, ) print(f\"[prompt_embeds] {out.choices[0].text!r}\") # Case 2: same prompt via text path, for sanity comparison. out_text = client.completions.create( model=SERVED_NAME, prompt=PROMPT, max_tokens=16, temperature=0.0, ) print(f\"[text ] {out_text.choices[0].text!r}\") if out.choices[0].text == out_text.choices[0].text: print(\"MATCH: prompt_embeds output equals text output\") else: print(\"DIVERGE: outputs differ (expected only if tokenizer/embedding \" \"scaling differs from raw lookup)\") if __name__ == \"__main__\": main() ``` ## Test Result ``` python test_embeds_serve.py [transformers] `torch_dtype` is deprecated! Use `dtype` instead! Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 6652.97it/s] [prompt_embeds] ' Paris. The capital of France is also the capital of the Republic of France.' [text ] ' Paris. The capital of France is also the capital of the Republic of France.' MATCH: prompt_embeds output equals text output ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42963", "createdAt": "2026-05-18T10:04:55Z", "updatedAt": "2026-08-13T06:43:42Z", "timestamp": "2026-08-13T06:43:42Z", "metrics": { "reactions": 1, "comments": 13 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "gcanlin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5f61e45e1249f26f4abc", "signalId": "github:vllm-project/vllm:issue:50682", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50682", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[ROCm][AMD] Kimi-K3 Gap and Roadmap Tracking", "text": "Kimi-K3 is a very successful launch. This issue is to track vLLM upstream work for feature enablement and performance optimization on ROCm.
Day 0 features/baselines - AITER fused-moe: a16w4 (GENERAL) and a8w4 (INTERLEAVE ) were integrated for performance. Flydsl/opus a8w4 is perf is 3.5% higher than flydsl a16w4. - AITER MLA attention: non-causal mla-gluon kernel was used for 12 head. - AITER triton conv2d - AMD vendor specific KDA - DSpark speculative decoding: Target model used AITER MLA (gluon) and draft model used TRITON_MLA. Perf improvement on lower concurrency (c1-c16). - Day 0 docker: vllm/vllm-openai-rocm:kimi-k3 - Day 0 recipes: https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x&features=reasoning - vLLM upstream PRs for day 0 K3 code: #50000 [#50089](https://github.com/vllm-project/vllm/pull/50089) [#50090](https://github.com/vllm-project/vllm/pull/50090)
**Feature Gaps**: - [ ] fp8 kv-cache-dtype support: Needed to avoid AgentX performance degradation. - asm-mla path: - https://github.com/vllm-project/vllm/pull/51040 - https://github.com/vllm-project/vllm/pull/51011 - gluon-mla path: https://github.com/vllm-project/vllm/pull/50619 - - [ ] gfx950/MI350 distributed inference: Required for multi-node K3 enablement and scaling. - https://github.com/vllm-project/vllm/pull/51052 - [ ] DSpark: prefix-caching needs work - https://github.com/ROCm/aiter/pull/4474 - [ ] offloading performance/compatibility: Needed for offloading support and agentic workload. - cpu offloading + rocm 7.14/7.15 - [ ] Dependency: 1. [ ] ROCm Base Docker: v0.1.19+ AITER upgrade - https://github.com/vllm-project/vllm/pull/49361 2. [ ] Triton 3.6 to 3.7+ upgrade - https://github.com/vllm-project/vllm/pull/50607 - Resolving issue of triton 3.7.x with gluon mla kernel on DistributedLinearLayout -- in progress * triton: https://github.com/ROCm/triton/pull/960 * vLLM to pick the new commit https://github.com/vllm-project/vllm/pull/51464 3. [ ] Triton-kernel package/naming management. - https://github.com/vllm-project/vllm/pull/50328 - [ ] model runner V2 was disabled @vllmellm (Ye Hur, Pin Siang) - https://github.com/vllm-project/vllm/pull/51653 - [ ] gfx942 support and distributed inference - https://github.com/vllm-project/vllm/pull/50817 - **Performance Optimization**: - DSpark Speculative decoding: . * Goal: Agentic workflow: spec perf should be better than non-spec - fp8 kv-cache-dtype support: Dspark+fp8 kv cache+asm-mla path - minimal fix: https://github.com/vllm-project/vllm/pull/51860 - refactored: https://github.com/vllm-project/vllm/pull/51606 - qlen (K, num_speculative_tokens) tuning and support - https://github.com/vllm-project/vllm/pull/51590 - https://github.com/vllm-project/vllm/pull/51171 - - Update AITER in vLLM base docker: - https://github.com/ROCm/aiter/pull/4521 (merged) - https://github.com/ROCm/aiter/pull/4452 (merged) - https://github.com/ROCm/aiter/pull/4555 (merged) - https://github.com/ROCm/aiter/pull/4474 (merged) - https://github.com/ROCm/aiter/pull/4494 (merged) - https://github.com/ROCm/aiter/pull/4646 - https://github.com/ROCm/aiter/pull/4647 - https://github.com/ROCm/aiter/pull/4509 - - vLLM CPU offloading: Improve CPU-offloading performance (ROCm 7.14+). * https://github.com/vllm-project/vllm/pull/43018 * upgrade base docker to rocm 7.14 or above * - Agentic workload/benchmarking optimization. * profiling * max-num-seqs tuning - Mixed Attention optimization (Linear KDA 3 + MLA 1) - KDA : - Fusion: - #50654 - Prefill: - https://github.com/vllm-project/vllm/pull/51862 - https://github.com/vllm-project/vllm/pull/51540 - - MLA: - AITER Gluon MLA optimization (need new aiter release): * https://github.com/ROCm/aiter/pull/4509 (?) * https://github.com/ROCm/aiter/pull/4555 - Dispatch small head to AITER asm MLA (Done) * #50371 and #50578 and #51088 overlapped. Need consolidate to #50578 and add an opt-in env - - MoE: Flydsl a8w4 optimizations (multi-node, multi-stream) - [[FlyDSL] [MoE]: reuse stage-1(gate up) scratch buffer across layers and graph captures by xiaohuguo…](https://github.com/ROCm/aiter/pull/4647) - - Attention Res (triton and gluon), - Fusion: https://github.com/vllm-project/vllm/pull/50593 - Agentic workload (LMCache/Mooncake) - #51004 - gemm tuning - MORI-io + K3: (1) K3 KDA statement management. (2) UMBP (Unified Memory & Bandwidth Pool) to reduce cost without sacrificing performance. **Misc: Bug fixes, cleanup/refactor, documentation, recipe, packaging** - https://github.com/vllm-project/vllm/pull/50582 - #50649 - https://github.com/vllm-project/vllm/pull/51682 - - [ ] - LMCache packaging to vLLM docker https://github.com/vllm-project/vllm/pull/51208 - [ ] - Mooncake packaging to vLLM docker **PR List** https://github.com/vllm-project/vllm/pulls?q=is%3Aopen+is%3Apr+label%3Arocm+label%3Ak3 Note that: (1) There are some overlap and duplicated efforts, for example, #50371 and #50578. (2) PRs that do not meet vLLM Policy will not be reviewed, and may be closed by maintainers. ** Live Issues** https://github.com/vllm-project/vllm/issues?q=is%3Aissue%20state%3Aopen%20label%3Arocm%20label%3Ak3 --- Things are moving fast. Please comment if you have a PR or PRs for review. For each PR, please ensure you run the full gsm8k eval if for text-only. For multi-modal fixes, please run OCRBench/MMMU Pro. For performance optimization PR, in addition to ensure accuracy (eval test result is mandatory), please show end-to-end `vllm serve` command and environment variables you used, and the before/after comparison.", "url": "https://github.com/vllm-project/vllm/issues/50682", "createdAt": "2026-08-01T12:39:29Z", "updatedAt": "2026-08-13T06:43:07Z", "timestamp": "2026-08-13T06:43:07Z", "metrics": { "reactions": 0, "comments": 16 }, "labels": [ "rocm", "kimi", "k3" ], "author": "hongxiayang", "state": "open", "assignees": [ "hongxiayang" ], "change": "updated" } }, { "id": "event:0e99225f0e824b4ce2ec", "signalId": "github:vllm-project/vllm:pull_request:50597", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50597", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm]Remove special-case SiTU support model-specific gating", "text": "## Purpose Mxfp4MoEMethod previously contained a model-specific predicate (_use_k3_situ_aiter) that special-cased the Kimi-K3 SiTU activation, gating three separate code paths — backend selection in __init__, size round-up bypass in maybe_roundup_sizes, and an entirely separate weight-shuffle method _setup_kernel_k3_situ in process_weights_after_loading. This tied SiTU behavior to a specific model identity rather than to the activation type itself. This PR removes that special-casing and routes SiTU through the standard machinery: MoEActivation.SITU is added to AiterExperts._supports_activation, so the oracle's is_supported_config accepts SiTU models without any model-specific intervention in Mxfp4MoEMethod mxfp4_round_up_hidden_size_and_intermediate_size gains an activation parameter; the ROCm 256-alignment is skipped when activation == MoEActivation.SITU, since the SiTU FlyDSL kernel pads internally and the round-up would OOM native intermediate sizes convert_weight_to_mxfp4_moe_kernel_format gains an activation parameter; the AITER_MXFP4_BF16 branch dispatches to the shuffle_weight_a16w4/e8m0_shuffle shuffle for SITU and the existing _shuf_w/_shuf_s shuffle for all other models _setup_kernel passes activation=self.moe.activation through to both functions above, and its shape assertions are gated on activation != MoEActivation.SITU since SITU's native intermediate sizes are not 256-aligned process_weights_after_loading calls _setup_kernel unconditionally for all MXFP4 paths _use_k3_situ_aiter, self.is_k3_situ_aiter, and _setup_kernel_k3_situ are deleted. Any future model using SiTU activation with MXFP4 weights on ROCm gfx950 will route correctly without requiring new special-case code. This work was done with AI assistance and reviewed by a human. ## Test Plan Validate Kimi-K3: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 vllm serve moonshotai/Kimi-K3 --host localhost --port 8001 --quantization mxfp4 --tensor-parallel-size 8 --max-model-len 32768 --max-num-seqs 59 --trust-remote-code ## Test Result lm_eval --model local-completions --model_args model=moonshotai/Kimi-K3,base_url=http://localhost:8001/v1/completions,tokenizer_backend=huggingface,trust_remote_code=True --tasks gsm8k --num_fewshot 5 Original: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9666|± |0.0049| | | |strict-match | 5|exact_match|↑ |0.9666|± |0.0049| Updated: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9651|± |0.0051| | | |strict-match | 5|exact_match|↑ |0.9651|± |0.0051|", "url": "https://github.com/vllm-project/vllm/pull/50597", "createdAt": "2026-07-31T16:57:26Z", "updatedAt": "2026-08-13T06:43:05Z", "timestamp": "2026-08-13T06:43:05Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "rocm", "ready", "needs-rebase", "quantization" ], "author": "stacyroberts", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:932d9d6a2d1a11c2199e", "signalId": "github:vllm-project/vllm:pull_request:50817", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50817", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Enable AITER MXFP4 MoE on gfx942 and optimize tile configurations for MI325X Target Kimi K3 running on MI325X", "text": "## Summary Makes Kimi-K3 run on AMD MI325X (gfx942 / CDNA3). The model loaded but aborted in MLA decode; once past that it fell back to a dequantise-to-BF16 MoE emulation path. Both are fixed, and a third change addresses AITER MoE tile shapes being sized for CDNA4 on all architectures. Measured on 8x MI325X with the real 2.75T-parameter MXFP4 checkpoint: | | before | after | | --- | --- | --- | | Kimi-K3 decode (TP=8, CUDA graphs, warm) | 4.27 tok/s | **99.67 tok/s** (23x) | | Prefill throughput (36k prompt) | 3,522 tok/s | **6,537 tok/s** (1.9x) | | MLA decode at 12 heads/rank | `AssertionError` | works, bit-exact | ## Motivation Three independent blockers, in the order they surfaced. **1. MLA decode aborted on any model with <16 heads per rank.** ``` AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 ``` AITER's small-head MLA decode kernel asserts CDNA4, but the backend selected it on head count alone. Kimi-K3 has 96 attention heads, so TP=8 gives 12 per rank — and TP=8 is the *only* viable parallelism: 1.5 TB of weights against 2 TB of HBM rules out TP=6 (260 GB/GPU, and 7168 % 6 != 0) and TP=4 (390 GB/GPU). The existing pad-to-16 fallback only handled head counts dividing 16; `repeat_interleave(16 // num_heads)` is a silent no-op for 12, leaving q at 12 heads while the output was allocated for 16. **2. MXFP4 MoE fell through to BF16 emulation on gfx942.** AITER's CK MXFP4 kernel needs CDNA4 native FP4 MFMA, so gfx942 landed on `OCP_MXQuantizationEmulationTritonExperts`, which dequantises the *entire* weight tensor every forward: ```python w1_dequant = self._dequantize_weights(w1, ...) # all 896 experts w2_dequant = self._dequantize_weights(w2, ...) # 16 are routed to ``` That is 56x the required work. For K3: 896 experts x 33.0M params / 8 GPUs x 2 bytes ~= 7.4 GB per layer, x 93 layers ~= 688 GB materialised per token, written then read back ~= 1.4 TB of HBM traffic, / ~6 TB/s ~= 229 ms/token. Predicted ~4.4 tok/s; measured 4.27. That was the entire performance story. AITER *does* ship a Triton `moe_gemm_a16w4` kernel which dequantises in-register during the GEMM. It runs correctly on gfx942 — the arch gate simply never admitted it. The ROCm base image already builds AITER with `AITER_ROCM_ARCH=\"gfx942;gfx950\"`, so these kernels were compiled and present all along. **3. AITER MoE tile selection is architecture-blind.** `get_kernel_config` hardcodes `block_k=256`, `block_n=512`, `num_stages=1` for every AMD part. Those suit CDNA4's 160 KiB LDS; CDNA3 has 64 KiB. `arch_info._LDS_CAP_BYTES` already records the per-arch capacity and is consulted by `pa_decode_sparse`, `gemm_config_utils` and `sparse_attention_dsv4` — the MoE GEMM is the one kernel that skips it. ## Changes ### MLA small-head decode (`vllm/v1/attention/backends/mla/rocm_aiter_mla.py`) - Add `AiterMLAHelper.gluon_decode_available()` (gfx950) and require it in `use_gluon_decode()` and in the multi-token verify branch, so non-CDNA4 falls through to the padded path instead of asserting. - `get_mla_padded_q` / `get_mla_unpadded_o` keep whole-repeat expansion where the head count divides 16, and zero-pad the head dimension otherwise. Zero-padding is exact, not approximate: the decode kernel treats query heads independently, so filler heads cannot perturb retained ones. The new test pins that property — running 16 real heads, then the same tensor with heads >= N zeroed, gives **bit-identical** outputs for heads 0..N-1. ### MXFP4 MoE on gfx942 - `mxfp4_utils.py`: new `use_aiter_mxfp4_triton_moe()` — a single predicate for \"AITER MXFP4 runs the Triton kernel\", replacing scattered `on_gfx1250()` checks that independently decided weight-preparation format. It also requires `has_triton_kernels()`, so a build without that package falls through to previous behaviour instead of asserting at weight load. - `aiter_mxfp4_w4a8_moe.py`: admit gfx942 in `AiterW4A16ExpertsMonolithic._supports_current_device`. - `oracle/mxfp4.py`: split the `AITER_MXFP4_BF16` weight-prep branch — the Triton path swizzles and returns `PrecisionConfig`s, the CK path keeps the de-interleave plus aiter shuffle. - `quantization/mxfp4.py`: route parameter-vs-wrapped-tensor assignment, the `is_shuffled` marking, and the quant-config scale source through the same predicate. The **de-interleave of w13 gate/up pairs is CK-only.** A first attempt applied it to the Triton path and produced uncorrelated output (cosine 0.034 vs the TRITON backend). The Triton kernel consumes the checkpoint's interleaved layout like every other Triton backend. ### SiTU activation support K3's MoE uses SiTU — `beta*tanh(gate/beta)*sigmoid(gate)*up` — which cannot be expressed through the a16w4 kernel's `alpha`/`limit`/`add_residual` knobs. `aiter_triton_kernel_w4a16_moe_forward` now runs gemm1 unfused for SiTU and applies the existing compiled `torch.ops._C.situ_and_mul` between the two GEMMs. Only activations round-trip through HBM (a few hundred KB); weights never do. K3 already stores w13 as `[gate_all, up_all]`, exactly what `situ_and_mul` splits on, so no reshuffling is needed. `_setup_kernel_k3_situ` emits the Triton layout on gfx942 rather than the CK shuffle, and drops the registered Parameters before rebinding since the swizzled tensors are `triton_kernels` wrappers. ### Tooling and CI - `benchmarks/kernels/benchmark_moe_a16w4_tiles.py` — sweeps MoE tile shapes and reports the best against stock. Overrides only tiles at/above `--min-block-m` (default 64), leaving decode alone. - `tools/vllm-rocm/patch_aiter_cdna3_moe_tiles.py` — LDS-aware override for AITER's tile selector on gfx942, with `--revert`. - `tools/vllm-rocm/README_kimi_k3_mi325x.md` — serving recipe and two operational traps (below). - `.buildkite/test-amd.yaml` — add `vllm/models/kimi_k3/amd/ops/third_party/kda/` to the KDA job's source dependencies; edits to the AMD kernel copies did not trigger the AMD job. - `kimi_k3/amd/.../kda/__init__.py` — correct a provenance note claiming byte-identity with a NVIDIA copy that has since diverged. ## Results All on 8x MI325X (gfx942), ROCm 7.2.3 container, AITER v0.1.19, real `moonshotai/Kimi-K3` weights, TP=8, CUDA graphs, `max_num_seqs=512`, `gpu_memory_utilization=0.97`. **Decode** | backend | tok/s | | --- | --- | | EMULATION (before) | 4.27 | | Triton a16w4 + SiTU, cold | 65.66 | | Triton a16w4 + SiTU, warm | **99.67** | Cold carries ~18 Triton JIT compilations inside the timed window. **MoE GEMM tile shapes** (4096 tokens; stock = `block_n=512 block_k=256 warps=8 stages=1`) | GEMM | stock | best gfx942 | | | --- | --- | --- | --- | | gemm1 K=3584 N=768 | 8.208 ms | 1.303 ms (`bn=128 bk=128 s=1 w=4`) | 6.30x | | gemm2 K=384 N=7168 | 8.019 ms | 1.701 ms (`bn=512 bk=128 s=2 w=8`) | 4.72x | `block_k` is the dominant variable — halving it shrinks both LDS tiles at once. Triton names the constraint when it does not fit: `OutOfResources: shared memory, Required: 100352, Hardware limit: 65536`. **End-to-end prefill** | prompt tokens | before | after | | | --- | --- | --- | --- | | 3,697 | 2,268 tok/s | 4,459 | 1.97x | | 14,497 | 3,156 tok/s | 5,910 | 1.87x | | 36,097 | 3,522 tok/s | 6,537 | 1.86x | | 72,098 | 3,427 tok/s | 6,585 | 1.92x | Smaller than the kernel-level 5-6x because prefill also pays for MLA, 69 KDA layers, norms and TP collectives. ## Testing ```bash pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_small_head_pad.py # 11 passed VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \\ pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py -k test_rocm_mxfp4_moe_oracle # 3 passed, 1 skipped (AITER_MXFP4_FP8 still requires gfx950) VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \\ pytest -v -s tests/kernels/moe/test_ocp_mx_moe.py \\ tests/kernels/moe/test_mxfp8_aiter_backend_selection.py # 22 passed, 117 skipped, 0 failed pytest -v -s tests/models/kimi_k3/test_amd_attn_res.py # 4 passed pytest -v -s tests/models/kimi_k3/test_kda.py tests/models/kimi_k3/test_kda_metadata.py # 45 passed, 6 skipped ``` Additional verification beyond the suites: - Isolated a16w4 kernel vs dequantised reference GEMM: max rel err **5.7e-4** - Full MoE vs the trusted TRITON backend, identical weights: cosine **0.999906** - MLA head padding: **bit-identical** outputs for retained heads ## Model evaluation Greedy decode (`temperature=0`) through the OpenAI-compatible server on real weights returns identical text to the EMULATION baseline, including `17 * 23 = 17 * 20 + 17 * 3 = 340 + 51 = 391`. Post-tile-patch spot checks: 4/4 correct, including `4839 * 271 = 1311369`. I did **not** run a formal lm-eval / gsm8k harness. Given the change is kernel-path only and matches the reference backend to cosine 0.999906, I consider that proportionate, but I will run one if a reviewer wants it. ## Not included / known limitations - **The AITER tile fix is a runtime patch**, not a vLLM change. It modifies a third-party package and does not survive an AITER upgrade. The durable fix is upstream in ROCm/aiter, deriving `block_k` from `_LDS_CAP_BYTES`. Its constants are tuned to K3's two GEMM shapes; the `n <= 1024` split is a coarse proxy that lands on both measured optima but is not a general autotuner. - **Speculative decoding on gfx942 with <16 heads is untested.** Gating the Gluon multi-token branch means that case now falls through to the padded path. It is dormant for K3 (`num_nextn_predict_layers=0`), but anyone enabling MTP/EAGLE3 would be on an unvalidated causality path. - **`AITER_SITUV2_A8W4` is ignored by the new Triton path.** The CK branch reads it to select a gate/up-interleaved layout; the Triton path always assumes separated. Default-off, but a silent trap if set. - **No tuned MoE configs for MI325X** at K3's shapes (`E=896,N=384,device_name=AMD_Instinct_MI325X.json` is absent). - `AITER_MXFP4_FP8` (W4A8) remains gfx950-only — it needs the CDNA4 swizzle. ## Operational notes for anyone serving Kimi-K3 on MI325X Two traps that look like model corruption and are not: - **`max_num_seqs` must stay below the Mamba/KDA state block count.** K3 has 69 KDA layers and needs one state block per concurrent decode sequence, so the default 1024 fails CUDA graph capture at 854 available blocks. - **The `kimi_k3` reasoning parser strips the think channel on its closing marker.** If `max_tokens` truncates before that marker, raw chain-of-thought is returned as `content` instead of an answer. Combined with the template's `thinking_effort=max` default this reads as garbage output. Measured tool-call emission, 6 trials per cell: | schema | default (max) | low | high | thinking off | | --- | --- | --- | --- | --- | | 1-param | 6/6 | 6/6 | 6/6 | 0/6 | | 2-param | 3/6 | 6/6 | 6/6 | 3/6 | | 3-param | 6/6 | 6/6 | 6/6 | 0/6 | `thinking_effort: low` is 18/18 and ~3.5x cheaper in tokens than the default. `thinking: false` must not be used with tools.", "url": "https://github.com/vllm-project/vllm/pull/50817", "createdAt": "2026-08-03T05:44:20Z", "updatedAt": "2026-08-13T06:42:53Z", "timestamp": "2026-08-13T06:42:53Z", "metrics": { "reactions": 7, "comments": 11 }, "labels": [ "performance", "rocm", "needs-rebase", "ci/build", "quantization", "kimi", "k3" ], "author": "seanfilimon", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2a3e3c576553c184ebf9", "signalId": "github:vllm-project/vllm:pull_request:52058", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52058", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Bound KV block zeroing launch geometry", "text": "## Purpose Fix the `KVBlockZeroer` launch overflow reproduced on [`main` nightly #83443](https://buildkite.com/vllm/ci/builds/83443/canvas?jid=019ff2a1-641e-4916-9149-30a5074c8a9c&tab=output), at commit [`3e372c5ff2`](https://github.com/vllm-project/vllm/commit/3e372c5ff23438eeeafc86c7d8d51026f3dacb6a): ``` OverflowError: signed integer is greater than maximum ``` DeepSeek-V4 combines 181 KV segments with 9,344- and 292-element pages. The old zeroer selected the largest common power-of-two divisor, so the 292-element page forced every segment to use four-element chunks. Zeroing 6,870 blocks therefore flattened to: ``` 6,870 * 181 * 2,336 = 2,904,745,920 programs ``` That overflows the signed launch dimension passed by the NVIDIA wrapper. This change: - maps blocks, segments, and chunks directly onto a 3-D grid, keeping block IDs on the large x-axis; - uses up to 1,024 elements per program and masks each segment's tail, so small pages no longer shrink all chunks; - reduces the failing geometry to `(6,870, 181, 10)`, or 12,434,700 programs (233.6x fewer); - preserves one compiled kernel across different block counts. The H100/H200 `sm90_paged_mqa_logits_metadata` failures from the same CI build were separate. They are already fixed on current `main` by #52035; its [exact 4xH100 KV-offload job](https://buildkite.com/vllm/ci/builds/83596#019ff753-5357-42e3-a6ba-8d51b6b40b2f) passed both DeepSeek-V4 cases (`2 passed`). ## Duplicate-work check Required open PR/issue searches were run for `KVBlockZeroer overflow`, `KV block zeroing grid`, and `signed integer zeroing`. #50485 also proposes a 3-D grid, but this is materially different: - #50485 is currently conflicted and predates packed block strides; - it puts block IDs on the z-axis, which is limited to 65,535 entries on CUDA; - it retains narrow divisor-based chunks; - this change puts blocks on x and adds masked wide chunks, addressing both the observed NVIDIA overflow and the 233.6x excess work. ## Test plan and results ```bash CUDA_VISIBLE_DEVICES=0 .venv/bin/python -m pytest \\ tests/v1/worker/test_kv_block_zeroer.py \\ tests/v1/worker/test_dsv4_packed_zeroer_geometry.py -q # 9 passed CUDA_VISIBLE_DEVICES=0 compute-sanitizer --tool memcheck --error-exitcode 99 \\ .venv/bin/python -m pytest tests/v1/worker/test_kv_block_zeroer.py -q # 8 passed; ERROR SUMMARY: 0 errors .venv/bin/pre-commit run --files \\ vllm/v1/worker/utils.py tests/v1/worker/test_kv_block_zeroer.py # passed ``` An exact-shape B200 validation launched grid `(6870, 181, 10)`, zeroed all storage, and completed in 0.334 seconds. No model eval was run because this only changes how already-selected KV-cache bytes are zeroed, not model outputs or scheduling semantics. ## Disclosure AI assistance (OpenAI Codex) was used to investigate the CI history, develop the change, and draft this description. reviewed by the submitter ## Breaking PR and overlap The failure became reachable after #51749 generalized worker-side KV zeroing to every allocating `AttentionSpec`. That change is needed to prevent stale FP8 sliding-window pages; it exposed a pre-existing launch-geometry scaling bug when DeepSeek-V4 contributes many heterogeneous segments. #52062 addresses the same observed overflow by reverting #51749. This PR is materially different: it preserves generalized zeroing and bounds the kernel launch using a 3-D grid plus masked wide chunks, fixing the overflow without restoring the stale-cache bug that #51749 corrected.", "url": "https://github.com/vllm-project/vllm/pull/52058", "createdAt": "2026-08-12T22:01:44Z", "updatedAt": "2026-08-13T06:42:52Z", "timestamp": "2026-08-13T06:42:52Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "ready" ], "author": "LucasWilkinson", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:6b39aca8efdd6aa037f2", "signalId": "github:vllm-project/vllm:pull_request:43249", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43249", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix MRV2 Gumbel sampling for non-finite logits", "text": "## Summary Fix MRV2 Gumbel sampling when the candidate logits contain `-inf` entries. The current Gumbel path only clamps the uniform draw away from zero. If the draw reaches the upper endpoint after floating-point rounding, the Gumbel noise becomes `inf`. Adding that noise to a masked/non-candidate `-inf` logit produces `NaN`, and a subsequent Triton reduction can return an invalid token index. This change: - clamps fp32/fp64 uniform draws away from both `0` and `1`; - only adds Gumbel noise to finite logits; - adds a CUDA/Triton regression test that deterministically reproduces the `-inf + inf -> NaN` edge case. ## Why this matters We hit this downstream in a Kimi K2.6 MRV2 speculative decoding deployment with DCP8 and Eagle/MTP. The failure surfaced during spec-decode/logprobs warmup as an invalid sampled token id and then a CUDA illegal memory access: ```text Invalid sampled token ids for spec logprobs: count=1 vocab=163840 idx=[508] vals=[9223372034707292159] ``` The DCP8/Kimi setup makes the bug easy to trigger, but the root cause is not DCP-specific. It is the generic Gumbel sampler being asked to sample from residual logits where rejected tokens are represented as `-inf`. A minimal deterministic reproduction of the arithmetic issue is: ```python logits = -inf u = 1.0 gumbel_noise = -log(-log(u)) # inf logits + gumbel_noise # NaN ``` After this patch, the same case stays `-inf` and does not poison the reduction. ## Tests ```text python3 - <<'PY' import importlib.util path = \"tests/v1/sample/test_gumbel.py\" spec = importlib.util.spec_from_file_location(\"test_gumbel_direct\", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(False) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(True) print(\"direct test passed\") PY python3 -m py_compile vllm/v1/worker/gpu/sample/gumbel.py tests/v1/sample/test_gumbel.py git diff --check ``` I could not run the repository pytest entrypoint in this local container because `tests/conftest.py` imports `tblib`, which is not installed in the environment: ```text ModuleNotFoundError: No module named 'tblib' ```", "url": "https://github.com/vllm-project/vllm/pull/43249", "createdAt": "2026-05-20T20:58:39Z", "updatedAt": "2026-08-13T06:42:44Z", "timestamp": "2026-08-13T06:42:44Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "voipmonitor", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4e870e2e1695d3e3acdf", "signalId": "github:vllm-project/vllm:pull_request:51538", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51538", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark", "text": "## Purpose DeepSeek-V4-Flash-0731 could not run reliably through the SM120 sparse MLA backend. This fixes the seven defects that blocked it across all three decode modes -- plain decode, MTP, and DSpark -- verified end-to-end on 8xRTX PRO 6000 Blackwell across in-flight batching and prefill/decode disaggregation. Commits 1-5 unblock DSpark. Commits 6-7 fix a hang that is **not** DSpark-specific: it strands any MTP (`next_n > 1`) server on this backend once the batch drains, and is a pre-existing defect on `main` rather than a regression from this PR. It is filed as #51593, with the full root-cause analysis in [this comment](https://github.com/vllm-project/vllm/issues/51593#issuecomment-5237934676). ## Why this is not a duplicate This consolidates #51042 after coordination with the maintainers (see https://github.com/vllm-project/vllm/pull/51042#issuecomment-5230492529). @ilmarkov is `Co-authored-by` on the SWA-width commit, whose `amd/rocm.py` hunk and `decode_swa_width` field are his work. Two deliberate design differences are described in that comment. Duplicate checks run for `dspark`, `sparse MLA SWA width`, `mxfp4 gemm1_alpha`, and `deepseek v4 expert placement`; no other open PR covers the remaining four commits. For commits 6-7, checks run for `persistent_topk`, `indexer seq_lens spec decode`, and `51593 in:body`. Two open PRs are adjacent but do not overlap: - **#43970** (MLA indexer / MTP) touches the same two files. Its \"drop padded MTP decode slots\" applies to the variable-length flatten branch, which computes `seq_len - decode_len` = `0 - 0` = `0` and is already safe. The uniform and native spec-decode paths that produce the negative length are untouched by it. The two changes are complementary and do not conflict. - **#49139** fixes a different bug in `persistent_topk.cuh` (radix histogram reuse after short rows). Worth flagging the interaction: its precondition is a CTA group processing rows that cross `RADIX_THRESHOLD`, and the out-of-range lengths fixed here cause exactly that spuriously. That PR is still needed for genuinely long rows. ## What is fixed 1. **SWA widths** — non-causal draft batches allocate `decode_swa_indices` wider than `window_size`, but the FlashInfer DSV4 path reshaped with `window_size` and crashed the draft. The dense width is now carried on the metadata. The non-causal width pads to a multiple of 64 (192 for the K=5 shape: 128 sliding-window + 5 draft entries) rather than 128 (256), matching the kernel's 64-entry tile; both dispatch after flashinfer-ai/flashinfer#4380, and 192 measures 13-16% faster at >=8 tokens. 2. **Workspace lanes** — the DSpark target and draft CUDA graphs retain workspace views concurrently, so one buffer per ubatch let a resize for one orphan the other's live tensor. The second lane is allocated only for V2 DSpark. 3. **Graph replay and draft KV** — `sample_idx_mapping` was zero-filled, so capture executed padding rows that scattered into request slot 0; captured backbone outputs could be freed before replay read their storage; draft KV could be written into physical block 0, the null block. Draft sampling also moves to a disjoint Philox counter range, since the rejection sampler keys both its acceptance uniform and its recovery Gumbel noise by token position. 4. **MXFP4 SwiGLU parameters** — `FlashInferExperts` injected the GPT-OSS activation constants (`gemm1_alpha=1.702`, `gemm1_beta=1.0`, `gemm1_clamp_limit=7.0`) for every mxfp4 weight dtype. DeepSeek V4 uses this path under `--moe-backend flashinfer_cutlass`, so its SwiGLU was evaluated with GPT-OSS constants and generation collapsed. GPT-OSS is unaffected: `GptOssMxfp4MoEMethod` supplies the same constants through its quant config, which the added test pins. 5. **SM120 gate** — a FlashInfer build can expose the sparse MLA decode API without carrying the DSV4 specialization a configuration needs. That now fails at model init with the required `(num_q_heads, top_k)` shape instead of an opaque kernel launch failure at the first decode. 6. **Negative indexer context lengths under MTP** — padded decode slots carry `seq_len == 0`, and with `next_n > 1` both spec-decode paths computed a negative per-token context length for the first token of each padded request (`0 - 2 + 0 + 1 = -1`). The sparse-MLA top-k kernels consume `lengths` as `uint32`, so `-1` is read as ~4.29e9. Clamped at 0 in both paths, matching the variable-length path which already yields `0`. With `next_n == 1` the expression collapses to `seq_len`, which is why plain decode never hit this. 7. **Top-k kernels hardened against out-of-range lengths** — `persistent_topk_kernel` cast `lengths` to `uint32` *before* testing `RADIX_THRESHOLD`, so the bogus ~4.29e9 forced the row onto the multi-CTA radix path. Because the `cta_in_group != 0` early exit is decided from the host-side `max_seq_len` while the per-row branch reads device memory, the two disagreed and stranded the group leader on the inter-CTA barrier forever — the kernel never retired, the async output-copy event never fired, and the engine hung waiting for a response that was never sent. The row length is now clamped to `min(stride, max_seq_len)` before any decision, which also removes an out-of-bounds read into the next row. `cooperative_topk` compares signed (so it cannot hang) but then casts to `uint32`, emitting indices `0..TopK-1` instead of `-1` padding; clamped as well. ## Model evaluation DeepSeek-V4-Flash-0731, `--moe-backend flashinfer_cutlass`, `--attention-backend FLASHINFER_MLA_SPARSE_DSV4`, gsm8k `n=1319`: | configuration | strict-match | flexible-extract | acceptance | FlashInfer | | --- | --- | --- | --- | --- | | target-only, TP4 | 0.9454 | 0.9454 | — | main @ 7f614b86 | | DSpark, TP4 | 0.9477 | 0.9477 | 65.5% | main @ 7f614b86 | | DSpark, TP4+EP4 | 0.9462 | 0.9462 | 67.1% | dc963cc0 | | DSpark + P/D (mooncake) | 0.9492 | 0.9500 | — | dc963cc0 | Standard rejection sampling is distribution-preserving, and DSpark lands 0.0023 from target-only against a difference-stderr of ~0.009 (0.26 sigma), on the same checkpoint and the same tree. MTP on the base checkpoint (`num_nextn_predict_layers = 1`), the configuration that #51593 hangs on, gsm8k `n=1319`: | configuration | strict-match | | --- | --- | | MTP, TP4 (commits 6-7 applied) | 0.9477 | That sits inside the 0.9454-0.9492 band spanned by the target-only and DSpark rows above, on the same checkpoint and tree, so the clamp does not move accuracy. No unfixed baseline is quoted for this row: without commits 6-7 the same server wedges under the drain reproducer within 30-180 s (see Test plan), and I did not obtain a clean unfixed eval run to compare against. Isolating fix 4 — identical configuration with only that commit removed: | configuration | strict-match | flexible-extract | | --- | --- | --- | | target-only, TP4 | 0.0000 | 0.0220 | | DSpark, TP4 | 0.0000 | 0.0174 | The TP4 pair was measured on both FlashInfer revisions with no significant change (DSpark 0.9454 -> 0.9477, target-only 0.9462 -> 0.9454), so the two rows measured on the earlier revision are directly comparable. ## Test plan ``` pytest tests/v1/worker/test_workspace.py \\ tests/v1/spec_decode/test_dflash_prepare_inputs.py \\ tests/v1/worker/gpu/spec_decode/test_utils.py \\ tests/kernels/attention/test_flashmla_sparse.py \\ tests/kernels/test_compressor_kv_cache.py \\ tests/v1/attention/test_flashinfer_sparse_mla_sm120_api.py \\ tests/kernels/moe/test_ocp_mx_moe.py ``` Result: **71 passed, 141 skipped**. For commits 6-7, a drain-shaped reproducer (waves of concurrent completions that retire unevenly, wedging the unfixed build in 30-180 s) against `deepseek-v4-flash` with `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`, TP4: | build | runs | result | | --- | --- | --- | | unfixed | — | hangs within 30-180 s; all 4 GPUs pinned at 100% util / 0% memory util | | commit 6 only | 13 | all clean | | commit 7 only (commit 6 reverted, so negative lengths still reach the kernel) | 3 | all clean | The commit-7-only run is the one that shows the kernel guard is load-bearing rather than dead code: with commit 6 in place nothing negative ever reaches the kernel, so it would pass either way. Root cause was confirmed on a live hang with cuda-gdb: exactly one of 16 CTAs resident, spinning in `wait_ge`, with `arrival_counter == 1` against `target_val == 2` — a group leader waiting on a peer that had already taken the `max_seq_len` early exit. ## AI assistance This work was produced with AI assistance. Every changed line has been reviewed by the submitter, and the tests and evaluations above were executed on the hardware described, with results reported as measured.", "url": "https://github.com/vllm-project/vllm/pull/51538", "createdAt": "2026-08-09T08:05:15Z", "updatedAt": "2026-08-13T06:41:54Z", "timestamp": "2026-08-13T06:41:54Z", "metrics": { "reactions": 2, "comments": 14 }, "labels": [ "bug", "speculative-decoding", "ready", "nvidia", "mrv2" ], "author": "lucifer1004", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:22d43c94fc5f14644974", "signalId": "github:vllm-project/vllm:pull_request:52092", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52092", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True", "text": "## Summary - Build and install a pre-built `triton-cpu` wheel in the CPU build/test images instead of `pip install`-ing it from source inside CI, unblocking the Triton topk-topp kernel to run as a normal (non-soft-fail) test. - Move the topk-topp Triton kernel test out of the soft-fail `CPU-ModelRunnerV2 Tests` suite into `CPU-Kernel Tests`, and the linear-attention chunked-prefill correctness test into `CPU-Language Generation and Pooling Model Tests`, then remove the now-empty `CPU-ModelRunnerV2 Tests` suite. - Guard the hardcoded `pin_memory=True` calls in `vllm/v1/worker/gpu/` (shared by the CPU V2 runner) and the multimodal model code it drives behind the existing `PIN_MEMORY` flag, since CUDA pinned memory isn't available on CPU. Also stub `torch.cuda.stream` in the CPU shim and log which model runner is active. ## Test plan - [x] `python3 -c \"import yaml; yaml.safe_load(open('.buildkite/hardware_tests/cpu.yaml'))\"` — confirms the reorganized pipeline YAML is still valid. - [ ] CPU CI (`CPU-Kernel Tests`, `CPU-Language Generation and Pooling Model Tests`, and CPU image build) — not run locally in this session; relying on CI to validate the Dockerfile and runtime changes on real Intel CPU hardware.", "url": "https://github.com/vllm-project/vllm/pull/52092", "createdAt": "2026-08-13T03:55:43Z", "updatedAt": "2026-08-13T06:40:57Z", "timestamp": "2026-08-13T06:40:57Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "qwen", "cpu", "mrv2" ], "author": "bigPYJ1151", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a1f7ccade6a30a26590b", "signalId": "github:vllm-project/vllm:pull_request:46009", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MoE] Preserve unquantized weight storage on ROCm", "text": "## Summary - Preserve unquantized MoE parameter storage during ROCm post-load padding by copying the padded view into existing storage instead of rebinding `.data`. - Return directly after AITER unquantized MoE `shuffle_weights()` so the AITER runtime layout is not forced through an extra `.contiguous()` conversion. - Add a ROCm MoE weight replay test covering both AITER and TRITON backends. ## Motivation In RL-style serving and training loops, external weight updates can trigger `process_weights_after_loading()` more than once after model initialization. Reassigning `param.data` in that path can replace the underlying parameter storage. That is unsafe once CUDA graphs or runtime kernels have captured parameter storage addresses, and can leave later rollouts reading stale or invalid storage. We observed this as a training-time rollout corruption issue after weight updates: generated responses became invalid, and reward/score collapsed to 0. Preserving the existing parameter storage with `.copy_()` keeps captured addresses stable while still applying the padded layout. For AITER unquantized MoE, `shuffle_weights()` already returns the runtime layout consumed by AITER. In local Qwen3.5-35B GRPO measurements, avoiding the final contiguous conversion for AITER was slightly faster after normalizing rollout generation time by response length. ## Replay test The new `tests/rocm/test_moe_weight_replay.py` test constructs a Qwen3.5 MoE LLM with sleep mode and CUDA graphs, generates a deterministic greedy response, runs level-2 sleep/wake, then replays checkpoint-format weights through a framework-agnostic load-then-post-load helper: 1. stream checkpoint-format weights into `model.load_weights(...)` in buckets; 2. call `process_weights_after_loading(...)` once after all buckets are loaded; 3. verify the ROCm MoE padding branch was actually exercised; 4. verify MoE `w13_weight` / `w2_weight` parameter storage addresses do not change; 5. verify greedy token IDs are deterministic after replay. This covers both AITER and TRITON MoE backends as first-class correctness cases. ## Local validation Static/lightweight checks in the PR checkout: - `python3 -m py_compile tests/rocm/test_moe_weight_replay.py vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method.py vllm/model_executor/layers/fused_moe/oracle/unquantized.py` - `python3 -m ruff check tests/rocm/test_moe_weight_replay.py vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method.py vllm/model_executor/layers/fused_moe/oracle/unquantized.py` - `python3 -m pytest tests/kernels/moe/test_unquantized_backend_selection.py tests/kernels/moe/test_moe_weight_loading_padded.py` - 38 passed, 2 skipped Heavy replay validation with Qwen3.5-35B-A3B and tensor parallel size 2: - pre-fix AITER: failed as expected; the test entered the ROCm padding branch and detected MoE parameter `data_ptr` changes. - pre-fix TRITON: failed as expected; the test entered the ROCm padding branch and detected MoE parameter `data_ptr` changes. - fixed AITER: passed. - fixed TRITON: passed. ## Benchmark note Local 10-step Qwen3.5-35B GRPO runs kept score/reward normal with zero aborted responses. Normalized rollout timing for AITER was slightly better without the final contiguous conversion: about 22.95 ms/token vs 23.11 ms/token.", "url": "https://github.com/vllm-project/vllm/pull/46009", "createdAt": "2026-06-18T07:15:27Z", "updatedAt": "2026-08-13T06:40:29Z", "timestamp": "2026-08-13T06:40:29Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "rocm", "ready", "ci/build", "quantization" ], "author": "aaab8b", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9a8391867421292ab5c7", "signalId": "github:vllm-project/vllm:pull_request:51208", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51208", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][AMD][Installation] add LMCache kv-connector installation and runtime packages to docker image", "text": "## Purpose Ship the LMCache KV connector in the ROCm image so it works out of the box. Note: vLLM's CUDA `docker/Dockerfile` installs LMCache when `INSTALL_KV_CONNECTORS=true`, and the release pipeline sets that for every published `vllm/vllm-openai` tag. docker/Dockerfile.rocm never referenced that arg, so vllm/vllm-openai-rocm ships without LMCache and users must build and install it themselves. This adds it so the ROCm release pipeline can enable it the same way. ### What this does - Adds a `build_lmcache` stage that compiles LMCache for HIP from source (`BUILD_WITH_HIP=1`, `CXX=hipcc`, `TORCH_DONT_CHECK_COMPILER_ABI=1`). A source build is required: LMCache publishes **no ROCm sdist on PyPI** (`>=0.4` is wheels-only), so `--no-binary` cannot force a HIP build. - Pinned to the v0.5.3 commit (140819c9d57a); LMCACHE_ROCM_ARCH defaults to gfx942;gfx950, covering MI300X/MI325X and MI350X/MI355X. - Installs the wheel with `--no-deps` plus four runtime packages. - `sortedcontainers` — hard requirement; without it `import lmcache.integration.vllm.lmcache_mp_connector` fails outright. - `opentelemetry-exporter-prometheus` — `lmcache server` calls `init_observability()` unconditionally, which imports `PrometheusMetricReader` (`mp_observability/otel_init.py:96`). Omitting it builds fine but the server dies at startup. - `cupy-rocm-7-0` — GPU cache-registration path. - `aiofile` — local-disk storage tier. - Gates everything behind build args, defaulting to off. ### Build-arg selection `INSTALL_KV_CONNECTORS` is an umbrella mirroring the CUDA Dockerfile's arg; each connector also has its own arg defaulting to it, so future connectors (like Mooncake) compose without renaming anything. | Build args | Result | | --- | --- | | *(none)* | nothing installed (default) | | `INSTALL_KV_CONNECTORS=true` | all connectors — CUDA release-pipeline parity | | `INSTALL_LMCACHE=true` | LMCache only | | `INSTALL_KV_CONNECTORS=true INSTALL_LMCACHE=false` | all except LMCache | Booleans rather than a list because the per-connector build stages are selected via `FROM _${ARG}`, which needs the value in the stage name. With a list the hipcc compile would run on every build and be discarded. ## Test Plan ```bash # (1) build docker buildx build -f docker/Dockerfile.rocm --target vllm-openai \\ --build-arg INSTALL_KV_CONNECTORS=true \\ -t vllm-rocm-lmcache:oob --load . # (2) start the LMCache server from the image, installing nothing docker run --rm --network host --ipc host \\ --device /dev/kfd --device /dev/dri --group-add video \\ -e LMCACHE_L1_SIZE_GB=906 --entrypoint bash vllm-rocm-lmcache:oob -c ' lmcache server --host 127.0.0.1 --port 5555 \\ --http-host 127.0.0.1 --http-port 8080 \\ --l1-size-gb \"$LMCACHE_L1_SIZE_GB\" --l1-init-size-gb 20 \\ --l1-read-ttl-seconds 7200 --chunk-size 1536 --max-workers 8 \\ --eviction-trigger-watermark 0.85 --eviction-ratio 0.10 \\ --eviction-policy LRU' ``` Also verified: build-arg selection matrix, and that a `false` build never runs the hipcc compile. ## Test Result Built and verified on **8x MI355X (gfx950), ROCm 7.2.3, torch 2.11**. **(1) Build** — succeeds. The in-Dockerfile smoke test passes, printing `lmcache 0.5.3`. Built wheel properties: | Check | Result | | --- | --- | | `DT_NEEDED` | `libamdhip64.so.7`, libtorch/c10 — **no CUDA libs** | | GPU archs in `c_ops` | `gfx942`, `gfx950` | | Python ABI | `cp312` | **(2) LMCache server starts out of the box**, nothing installed at runtime: ``` zmq 5555 -> accepting http 8080 -> HTTP 200 ``` Startup log shows shm capacity checked against the L1 request, L1 + L2 eviction controllers started, SessionManager cleanup thread, DeviceHostFuncDispatcher, mp-worker-reaper, and an 8-slot affinity pool matching `--max-workers 8`. **(3) No dependency downgrades** — the reason for `--no-deps`: | Package | Before | After | | --- | --- | --- | | numpy | 2.3.5 | 2.3.5 | | transformers | 5.14.1 | 5.14.1 | | opentelemetry-api / sdk | 1.44.0 | 1.44.0 | | opentelemetry-semantic-conventions | 0.65b0 | 0.65b0 | Added: `lmcache 0.5.3`, `sortedcontainers 2.4.0`, `aiofile 3.12.3`, `cupy-rocm-7-0 14.1.1`, `opentelemetry-exporter-prometheus 0.65b0`. **(4) Connectors register**: `['LMCacheConnectorV1', 'LMCacheMPConnector']`. **(5) Build-arg matrix** — all four rows behave as documented; with `INSTALL_LMCACHE=false` the `build_lmcache` stage is pruned entirely (verified by building `--target export_lmcache`: empty output, no clone, no compile). (6) Image size: 42.00 → 42.92 GB, +921 MB (+2.19%) — ~366 MB of that is CuPy, required by the GPU cache-registration path; the compiled extension is <8 MB. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51208", "createdAt": "2026-08-05T23:46:52Z", "updatedAt": "2026-08-13T06:39:31Z", "timestamp": "2026-08-13T06:39:31Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "rocm", "ready", "ci/build", "kv-connector" ], "author": "hongxiayang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d84ac702075f9bd397f9", "signalId": "github:vllm-project/vllm:issue:51921", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51921", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] v0.27.0 engine permanently stalls after ~1 min idle on 4-node TP=4 (GB10/sm_121, aarch64): shm_broadcast writer starves, requests never reach scheduler", "text": "## Bug On 4-node TP=4 (GB10/sm_121a, aarch64), a v0.27.0 engine serves its first requests normally, then **permanently stalls after roughly one minute of engine idleness**. Requests after the gap never enter the scheduler (`num_requests_running` stays 0), the API keeps answering `/v1/models` and `/tokenize`, and the head's EngineCore logs repeat: ``` [shm_broadcast.py:802] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work ``` every 60s indefinitely. All four rank containers stay alive; all TP workers' last log lines are healthy post-warmup (graph capture complete). Only a full fleet restart recovers. Reproduced 3/3 boots. Keeping the API warm (`/v1/models` polling every 45s) does NOT prevent it; the stall follows engine-level idleness, not connection idleness. Sequence, timestamped from one boot: warmup collectives complete across all ranks (so the broadcast queue works) → first 1-2 completions succeed (~1.3s each) → ~1-2 min idle → all subsequent requests hang forever. ## Environment Official `vllm/vllm-openai:v0.27.0-aarch64` + two patches required to get this far on sm_121: DeepGEMM repinned to `2fd67329` (#51758) and the `masked_mha_available` fix (#51920). GLM-5.2 (GlmMoeDsaForCausalLM), TP=4 via `--nnodes 4 --node-rank N`, mp executor, `FLASHINFER_MLA_SPARSE_SM120` + FLASH_ATTN MLA prefill, `fp8_ds_mla` KV, MTP k=2 spec decode, dual-rail RoCEv2, `VLLM_USE_BREAKABLE_CUDAGRAPH=0` set. torch 2.13.0+cu130 bundled NCCL. Related: #51758 validated v0.27.0 on 2x Spark (TP=2 and PP=2) with a 400-request soak and zero hangs — consistent with our result that the first requests DO work; the failure needs the multi-node idle gap, and possibly the 4-node width. This looks adjacent to the pre-0.27 TP-hang family that release fixed for 2-node. Full per-rank logs, exact launch argv, and timings available; the cluster reproduces this in ~20 minutes per attempt and I can run instrumented builds or targeted env matrices on request.", "url": "https://github.com/vllm-project/vllm/issues/51921", "createdAt": "2026-08-12T04:03:50Z", "updatedAt": "2026-08-13T06:39:25Z", "timestamp": "2026-08-13T06:39:25Z", "metrics": { "reactions": 0, "comments": 9 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:afda6d3ab6c3a9a94d41", "signalId": "github:vllm-project/vllm:issue:51884", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51884", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: FP8 block-scaled weights fail on sm120 (RTX 5090) — DeepGEMM \"Unknown SF transformation\" during process_weights_after_loading", "text": "### Your current environment
Environment ``` vLLM: 0.26.1rc1.dev608+g99a10304d (docker.io/vllm/vllm-openai:muse-glimmer) GPU: 2x NVIDIA RTX 5090 (sm120 / device capability 12.0), PCIe Platform: Linux x86_64, CUDA 12.x Model: RedHatAI/Muse-Glimmer-30B-FP8-block (compressed-tensors, block-scaled FP8 weights) ```
### 🐛 Describe the bug FP8 block-scaled weights (`compressed-tensors` w8a8) fail to load on sm120 with a DeepGEMM assertion. It happens in `process_weights_after_loading` — during weight loading, before any inference — so the engine never starts: ``` RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation ``` Full call chain (both TP workers fail identically): ``` compressed_tensors_w8a8_fp8.py:169 process_weights_after_loading -> kernels/linear/scaled_mm/deep_gemm.py:96 process_weights_after_loading -> quantization/utils/fp8_utils.py:1138 deepgemm_post_process_fp8_weight_block -> quantization/utils/fp8_utils.py:1077 deepgemm_post_process_weight_scale_block -> utils/deep_gemm.py:494 transform_sf_into_required_layout -> _transform_sf_into_required_layout_impl -> deepgemm csrc/apis/layout.hpp:60 \"Unknown SF transformation\" ``` The reason this is reached rather than skipped is that vLLM treats sm120 as a supported DeepGEMM target. `vllm/utils/deep_gemm.py` gates on capability family 100 **or** 120 in two places (lines 42-43 and 79-80), and `is_deep_gemm_supported()` documents \"only Hopper and Blackwell GPUs are supported\" — so consumer Blackwell selects the DeepGEMM path, and then DeepGEMM's own layout transform rejects the scale-factor layout it is handed. Either the SF transform needs an sm120 case, or the capability-family-120 gate should exclude this path for block-scaled FP8 weights. ### Reproduction ```bash docker run --gpus all --ipc=host -e CUDA_VISIBLE_DEVICES=0,1 \\ -v ~/.cache/huggingface:/root/.cache/huggingface \\ vllm/vllm-openai:muse-glimmer \\ --model RedHatAI/Muse-Glimmer-30B-FP8-block \\ --tensor-parallel-size 2 --gpu-memory-utilization 0.88 --max-model-len 131072 ``` Fails ~75 s in, at weight loading, reproducibly. ### Workaround `VLLM_USE_DEEP_GEMM=0` (plus `VLLM_MOE_USE_DEEP_GEMM=0`) makes the same command work. With that set the model loads and serves normally on 2x RTX 5090 — 177.6 tok/s decode with DFlash speculative decoding, 663,458-token KV cache — so only the DeepGEMM SF-layout path appears to be affected, not FP8 on sm120 in general. I have not tested whether NVFP4 or non-block-scaled FP8 checkpoints hit the same transform on this hardware; the NVFP4 variant of the same model loads fine with DeepGEMM left at its default.", "url": "https://github.com/vllm-project/vllm/issues/51884", "createdAt": "2026-08-11T22:34:07Z", "updatedAt": "2026-08-13T06:35:17Z", "timestamp": "2026-08-13T06:35:17Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "quantization" ], "author": "abtraore", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6f03e74b2fe1d2f4f50a", "signalId": "github:vllm-project/vllm:pull_request:51833", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51833", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add DeepGrove Maple (MapleForCausalLM)", "text": "## Purpose Adds support for [`deepgrove/maple-preview`](https://huggingface.co/deepgrove/maple-preview) (`MapleForCausalLM`), a 20B-A1B MoE reasoning model (MIT). Architecture specifics, none of which an existing model file combines: - `layer_types` interleaves sliding-window and global attention 3:1. The checkpoint runs FlashAttention with `window_size=(sliding_window, 0)`, i.e. an inclusive boundary spanning 513 positions, while vLLM maps `sliding_window=N` to `(N - 1, 0)`; the model therefore passes `config.sliding_window + 1`, as `modernbert.py` already does for the same reason. The window is applied per layer via `per_layer_sliding_window`, and `MapleModel.__init__` clears `cache_config.sliding_window` so the global layers do not inherit it. - `nope_on_global_attention`: the `full_attention` layers get no positional encoding, so `rotary_emb` is built only for the sliding layers. RoPE is half-width (`partial_rotary_factor: 0.5` → `rotary_dim=64` of `head_dim=128`). - Per-head QK RMSNorm over the full head dim, before RoPE. - Experts clamp gate/up to ±7, i.e. `SiluAndMulWithClamp` → `FusedMoEFactory(activation=\"silu\", swiglu_limit=...)`, not `swigluoai` (that adds `up + 1` and `alpha=1.702`). - `router_dtype: \"fp32\"` → gate is `ReplicatedLinear(params_dtype=torch.float32)` with `router_logits_dtype=torch.float32`. ### Why `transformers_utils/config.py` is touched The checkpoint declares `auto_map` but **never serializes `model_type`** (it only exists on the class in `configuration_maple.py`). `_CONFIG_REGISTRY` is keyed by `model_type`, so the lookup is skipped entirely and `AutoConfig` demands `trust_remote_code=True`. Registering the class with Transformers does not help either — `has_local_code` is `\"model_type\" in config_dict and ... in CONFIG_MAPPING`, and the first clause short-circuits on the missing key. So the model type is recovered from `architectures`, and the registered class is built directly by reusing the branch that already exists for `_SPECULATIVE_DECODING_CONFIGS`. This keeps `--trust-remote-code` unnecessary and lets vLLM own defaults the checkpoint omits (notably `swiglu_limit`). ### No new parsers The chat template is Qwen3-shaped, so existing parsers work as-is: `--reasoning-parser qwen3 --tool-call-parser hermes --enable-auto-tool-choice`. ## Test Plan ```bash pytest tests/models/test_maple.py tests/transformers_utils/test_missing_model_type.py pytest \"tests/models/test_initialization.py::test_can_initialize_large_subset[MapleForCausalLM]\" pytest tests/models/test_registry.py # 2x H20, bf16 vllm serve deepgrove/maple-preview --max-model-len 8192 [--tensor-parallel-size 2] [--enable-expert-parallel] python tests/evals/gsm8k/gsm8k_eval.py --port 8000 --num-questions 200 --num-shots 5 ``` GSM8K was also run in the model's intended chat/thinking mode (`temperature=0`, answer taken after ``). Numeric parity against the checkpoint's own modeling code, with only its FA3 kernel swapped for an SDPA mask that reproduces `window_size=(w, 0)` exactly, on a 900-token natural-text prompt (`prompt_logprobs=20` vs. `log_softmax` of the reference logits, aligned so that vLLM position `i` is compared against reference position `i - 1`). ## Test Result Lint: all hooks pass; mypy passes on 3.10–3.13. `actionlint` skipped (downloads a Go toolchain, and the diff has no workflow files); `--all-files` unavailable because `pre-commit` needs `git ls-files --deduplicate` (git ≥ 2.31, this box has 2.29.2), hence the diff range. | Test | Result | | --- | --- | | `tests/models/test_maple.py` | 8 passed | | `tests/transformers_utils/test_missing_model_type.py` | 1 passed (verified red when the `architectures` recovery is reverted) | | `test_can_initialize_large_subset[MapleForCausalLM]` | passed | | `tests/models/test_registry.py` | 378 passed, 15 skipped (2 pre-existing failures need `timm`, unrelated) | Weight loading: 9 shards / 37.69 GiB, no missing or unexpected keys. Parity, top-1 agreement in the windowed region (positions > 513) of the 900-token prompt: | | vs reference| | --- | --- | | vLLM (this PR) | **93.78%** (362/386) | | reference vs itself, window shrunk by one token | 93.54% | 93.54% is the ceiling this comparison can reach, so the `+ 1` form is within noise of the reference while the unchanged form is measurably off. Mean |Δ logprob| is 0.091 and next-token top-1 accuracy matches the reference (0.5061 vs 0.5028), confirming there is no systematic divergence. Two findings worth flagging from that harness: - The published modeling code is **silently broken under Transformers 5**: `from_pretrained` leaves the non-persistent `inv_freq` buffer uninitialised (observed values `0.0, 0.0, 2.67e18`), so RoPE degenerates to `cos == 1` everywhere and next-token accuracy collapses from 0.50 to 0.04. vLLM builds its own rotary cache and is unaffected; the numbers above were obtained after restoring `inv_freq` in the reference. - Its RoPE init also fails outright on Transformers 5, since `ROPE_INIT_FUNCTIONS` no longer has a `\"default\"` entry. Another reason to prefer the in-tree `MapleConfig` over `--trust-remote-code`. | GSM8K | chat/thinking, 200 q | 5-shot completion, 200 q | output tok/s | | --- | --- | --- | --- | | TP=1 | **91.0%** | 35.0% | 5020 | | TP=2 | **91.0%** (100 q) | — | — | | TP=2 + EP (128/256 experts per rank) | **90.0%** | 35.5% | 6365 | The low completion-style score is expected rather than a regression: the template opens `` for the assistant, so plain few-shot continuation is not this model's mode. Generations are coherent under both eager and `torch.compile` + full CUDA graphs. Parsers verified end-to-end: `reasoning` and `content` split correctly, and `tool_calls` returns `{\"name\": \"get_weather\", \"arguments\": \"{\\\"city\\\": \\\"Paris\\\"}\"}`. ### Follow-up validation (`a006ca53d6`) This follow-up adds a CPU regression test for the interleaved-cache failure mode seen in earlier sliding-window model fixes: a `full_attention` layer with no per-layer window must not fall back to a model-level cache window. ```bash python -m pytest -v tests/models/test_maple.py \\ tests/transformers_utils/test_missing_model_type.py \\ 'tests/models/test_registry.py::test_registry_imports[MapleForCausalLM]' CUDA_VISIBLE_DEVICES='' VLLM_TARGET_DEVICE=cpu \\ python -m pytest -v tests/transformers_utils/test_missing_model_type.py python -m pytest -v \\ 'tests/models/test_initialization.py::test_can_initialize_large_subset[MapleForCausalLM]' ``` Results: - Maple unit/config/registry selection: **13 passed**. The new test drives `EngineArgs.create_engine_config()` from a config without `model_type` and verifies the recovered 3:1 `layer_types` plus `cache_config.sliding_window is None`. - CPU-only config tests: **2 passed**. - Dummy-weight initialization: passed. - Real `deepgrove/maple-preview` checkpoint on H20: bf16 TP=1 eager serving, prefix caching, default `torch.compile` with full/piecewise CUDA graphs, and repeated identical-prefix requests all generated successfully. The compiled run captured every configured CUDA graph size and produced identical greedy tokens for the repeated request. - Real checkpoint TP=2 with `--enable-expert-parallel`: passed. Each rank held 128 of 256 experts and generated eight tokens successfully. - Spot checks against the original safetensors found exact equality for the embedding, Q/K/V and output projections, QK norms, decoder norms, and LM head. A standalone MoE check against the checkpoint's fp32 router and clamped SwiGLU reference had max/mean absolute errors of 0.015625/0.000626. ## Known limitations - Speculative decoding: `ngram` works (verified). EAGLE3/DFlash/DSpark do not, because `MapleForCausalLM` does not implement `SupportsEagle3` and no draft checkpoint exists for this target; adding the `EagleModelMixin` hooks is left for a follow-up. - `num_shared_experts != 0`, `moe_router_enable_expert_bias` and non-`silu` `hidden_act` raise rather than load something subtly wrong. - `MixtureOfExperts` (EPLB) is not implemented; only `enable_eplb` / `num_redundant_experts` are plumbed through. - Only tested on H20 (Hopper). - Parity is measured at the logits level over one prompt, not per layer. ## Accountability Duplicate-work checks found no existing PR or issue for this model or architecture (`gh pr list`/`gh issue list` over `Maple`, `deepgrove`, `MapleForCausalLM`; the only hits were fuzzy matches on unrelated PRs #42657, #44475, #44633). AI assistance was used to write and test this change. The human submitter has reviewed every changed line, understands the change end-to-end, and ran the tests and evals reported above. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51833", "createdAt": "2026-08-11T14:50:39Z", "updatedAt": "2026-08-13T06:34:37Z", "timestamp": "2026-08-13T06:34:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "new-model" ], "author": "NancyFyong", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:824b1e9bd759ef681a58", "signalId": "github:vllm-project/vllm:pull_request:49365", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49365", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Detect ROCm wheel variant from environment for precompiled wheels.", "text": "Fixes **AMD: Python-only Installation** failing because ROCm precompiled wheels on `wheels.vllm.ai` use a different path layout than CUDA. - **`setup.py`**: Detect installed ROCm at runtime, match against published variants on `wheels.vllm.ai/rocm/{commit}/`, fall back to AMD PyPI. - **`python_only_compile.sh`**: Same variant resolution for the preflight metadata check.", "url": "https://github.com/vllm-project/vllm/pull/49365", "createdAt": "2026-07-21T19:40:03Z", "updatedAt": "2026-08-13T06:33:27Z", "timestamp": "2026-08-13T06:33:27Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "aarushjain29", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:462fd6358556cd638081", "signalId": "github:vllm-project/vllm:issue:50895", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50895", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Populate completion_tokens_details in streaming and non-streaming usage responses", "text": "### 🚀 The feature, motivation and pitch The OpenAI chat/completions spec defines `completion_tokens_details` as part of `CompletionUsage`, which is shared between streaming and non-streaming responses. vLLM does not populate this field in either mode. vLLM 0.23+ fixed this for `prompt_tokens_details` (#44961): both streaming and non-streaming responses now include `cached_tokens` when `--enable-prefix-caching` and `--enable-prompt-tokens-details` are set. `completion_tokens_details` has no equivalent support. The concrete gap: ``` Non-streaming response (vLLM 0.25.x): { \"usage\": { \"prompt_tokens\": 72, \"completion_tokens\": 30, \"total_tokens\": 102, \"prompt_tokens_details\": {\"cached_tokens\": 64}, // works (0.23+) \"completion_tokens_details\": null // missing } } Streaming final chunk (stream_options.include_usage: true): { \"usage\": { \"prompt_tokens\": 77, \"completion_tokens\": 30, \"total_tokens\": 107 // completion_tokens_details: absent } } OpenAI spec expects: { \"usage\": { \"prompt_tokens\": 72, \"completion_tokens\": 30, \"total_tokens\": 102, \"prompt_tokens_details\": {\"cached_tokens\": 64}, \"completion_tokens_details\": { \"reasoning_tokens\": 0, \"accepted_prediction_tokens\": 0, \"rejected_prediction_tokens\": 0 } } } ``` **Why this matters for downstream consumers:** Metering and billing systems that sit behind an OpenAI-compatible gateway parse the `usage` object from the response to produce per-request records. Without `completion_tokens_details`, these systems cannot distinguish reasoning tokens from regular completion tokens. The workaround is to treat all completion tokens as one price tier, which prevents operators from offering pricing that reflects the different cost structure of reasoning-capable models. The Red Hat AI / OpenShift AI ecosystem uses vLLM as the model server behind Kuadrant/Connectivity Link gateways, and the billing layer needs these fields to price accurately. ### Alternatives Without this feature, downstream consumers must either: 1. Treat all completion tokens uniformly (loses pricing granularity for reasoning-capable models). 2. Attempt to recount reasoning tokens by re-parsing the response body outside vLLM (duplicates work the model server already did, fragile, and not possible for all model architectures). ### Additional context There are four open PRs implementing `reasoning_tokens` support, all currently in merge-conflict state: #35037, #44398, #45471, and #45802. Of these, #45802 (@chaunceyjiang) has the deepest integration through the streaming parser engine, and #45471 (@nv-nedelman-1) has the most thorough test coverage. Community contributor @dsgrafiniert has also implemented `count_reasoning_tokens()` for the 7 reasoning parsers that were missing it (posted on #35037). The foundation work is already merged: reasoning parser hooks (#33513) and the streaming parser engine (#45701). This issue tracks the broader ask: the full `completion_tokens_details` object, with streaming parity included from the start, following the same pattern as the `prompt_tokens_details` fix in #44961. We are happy to contribute a PR if that would help move this forward. Fields by relevance: | Field | Source | Status | |-------|--------|--------| | `reasoning_tokens` | Reasoning parsers | Four open PRs (#35037, #44398, #45471, #45802) + parser fork | | `accepted_prediction_tokens` | Speculative decoding | Not yet tracked | | `rejected_prediction_tokens` | Speculative decoding | Not yet tracked | `reasoning_tokens` is the highest priority. The speculative-decoding fields can follow separately. cc @cemigo114", "url": "https://github.com/vllm-project/vllm/issues/50895", "createdAt": "2026-08-03T16:41:50Z", "updatedAt": "2026-08-13T06:31:43Z", "timestamp": "2026-08-13T06:31:43Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [], "author": "rdwj", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a29ba09cf8f52213aea4", "signalId": "github:vllm-project/vllm:pull_request:42662", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42662", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[LoRA][Gemma4] Support vision tower LoRA", "text": "This PR adds the remaining LoRA plumbing needed for Gemma4 multimodal LoRA support. After #43798, Gemma4-MM vision linear layers are already converted through the Transformers backend path, so this PR no longer reimplements the Gemma4 vision tower. Instead, it focuses on the runtime LoRA mapping and token-counting pieces needed by Gemma4 image/video/audio inputs. Main changes: - Add a multimodal LoRA token-count interface so models can report separate tower and connector token counts. - Update Gemma4-MM to report modality-specific LoRA token counts for image, video, and audio inputs. - Size multimodal LoRA wrappers using the largest tower/connector token budget across modalities. FIX https://github.com/vllm-project/vllm/issues/40693 ## Test Plan Additional end-to-end tests for real Gemma4 vision LoRA adapters should also be added in a follow-up. ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42662", "createdAt": "2026-05-14T16:37:41Z", "updatedAt": "2026-08-13T06:29:41Z", "timestamp": "2026-08-13T06:29:41Z", "metrics": { "reactions": 4, "comments": 17 }, "labels": [ "documentation", "ready", "ci/build", "v1", "mrv2" ], "author": "linitra24", "state": "open", "assignees": [ "jeejeelee" ], "change": "updated" } }, { "id": "event:62b4e883911d9a3ceecb", "signalId": "github:vllm-project/vllm:issue:52023", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52023", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: draft_model speculative decoding crashes at init under TP>1 when draft hidden_size > target (TRT-LLM fused allreduce+RMSNorm workspace sized from target only)", "text": "### Summary With `draft_model` speculative decoding under tensor parallelism (`TP>1`), the engine crashes at init when the **draft model's `hidden_size` is larger than the target model's**. The `fuse_allreduce_rms` compilation pass builds the FlashInfer TRT-LLM fused allreduce+RMSNorm workspace with a token budget derived from the **target** model's `hidden_size` only. When the draft model then runs its own forward under TP, its larger `hidden_size` overflows that workspace and `check_trtllm_allreduce_fusion_workspace_metadata` raises, aborting startup. `draft_hidden < target_hidden` and `TP=1` both work; the crash is specific to **`draft_hidden > target_hidden` AND `TP>1`**. ### Environment - vLLM `v0.27.1`, V1 engine, `dtype=torch.bfloat16` - 2×H100 PCIe (80GB), `tensor_parallel_size=2` - `compilation_config.pass_config.fuse_allreduce_rms: True` (default), FlashInfer allreduce backend - Full raw server logs for all arms (`collect_env` included): https://github.com/seongyun1104/depthchart/tree/39b1557/draft_model_tp_bug ### Reproduction | Arm | target (TP) | draft | draft_hidden vs target_hidden | result | |-----|-------------|-------|-------------------------------|--------| | A | Qwen3-30B-A3B (**TP=1**) | Qwen3-4B | **2560 > 2048** | OK (acceptance 83.8%, normal) | | B | Qwen3-30B-A3B (**TP=2**) | Qwen3-4B | **2560 > 2048** | **CRASH** at init | | C | Qwen3-30B-A3B (TP=2) | Qwen3-0.6B | 1024 < 2048 | OK (acceptance 73.1%, coherent) | Arm A vs B isolates the trigger to **`TP>1`**: same `draft_hidden > target_hidden` pair, only the parallelism differs. Arm C shows the same TP=2 target is fine when `draft_hidden < target_hidden`. Arm B command (greedy): ``` vllm serve Qwen/Qwen3-30B-A3B \\ --tensor-parallel-size 2 \\ --max-model-len 4096 --max-num-seqs 8 --gpu-memory-utilization 0.9 \\ --override-generation-config '{\"do_sample\": false}' \\ --speculative-config '{\"method\": \"draft_model\", \"model\": \"Qwen/Qwen3-4B\", \"num_speculative_tokens\": 1}' ``` (Qwen3-4B `hidden_size=2560` as draft; Qwen3-30B-A3B `hidden_size=2048` as target.) ### Traceback (arm B, verbatim) ``` File \".../vllm/compilation/passes/fusion/allreduce_rms_fusion.py\", line 265, in call_trtllm_fused_allreduce_norm ... check_trtllm_allreduce_fusion_workspace_metadata( File \".../flashinfer/comm/trtllm_ar.py\", line 1026, in check_trtllm_allreduce_fusion_workspace_metadata raise ValueError(error_msg) ValueError: Workspace validation failed: - token_num (8192) * hidden_dim (2560) exceeds workspace max_token_num (8192) * hidden_dim (2048). This may cause Illegal Memory Access. ``` `hidden_dim (2560)` is the draft (Qwen3-4B); the workspace budget `hidden_dim (2048)` is the target (Qwen3-30B-A3B). ### Where it comes from The fused-allreduce workspace token budget is `max_token_num = max_size // (hidden_dim * element_size)`, and `hidden_dim` is taken from the target model config in the pass. The draft model's forward under TP issues the same fused allreduce with its own (larger) `hidden_size`, exceeding the target-sized budget. Note (not a proposed fix): patching the pass `__init__` to use `max(target_hidden, draft_hidden)` did **not** resolve it — the run still reports `hidden_dim (2048)` in the validation error, so the workspace metadata that is actually checked is created upstream of that pass field. A correct fix has to reach the workspace **creation** site so its stored `hidden_dim` accounts for the draft model. ### Relationship to #50877 / #50932 Same theme (FlashInfer allreduce workspace/buffer under a draft model that adds allreduce traffic), but a **distinct backend**: - #50877 / PR #50932 — the **MNNVL** path (`trtllm_mnnvl_ar`, Lamport buffer rotation); #50932 touches `flashinfer_all_reduce.py` / `fused_allreduce_gemma_rms_norm.py` and adds an `is_buffer_size_sufficient` admission guard. - This report — the **TRT-LLM fused allreduce+RMSNorm** compilation-pass path (`allreduce_rms_fusion.py` → `flashinfer/comm/trtllm_ar.py`), which #50932 does not touch. Filing separately so the TRT-LLM path is tracked; happy to fold into #50877 if maintainers prefer a single tracking issue.", "url": "https://github.com/vllm-project/vllm/issues/52023", "createdAt": "2026-08-12T17:01:32Z", "updatedAt": "2026-08-13T06:29:04Z", "timestamp": "2026-08-13T06:29:04Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:97a19b9cdb6063b71026", "signalId": "github:vllm-project/vllm:pull_request:51777", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51777", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docker] Update to nixl-1.3.2", "text": "## Purpose The KV-connectors layer installs `nixl` from requirements/kv_connectors.txt, then force-reinstalls the CUDA-matched backend wheel so the correct nixl_ep_cpp.so is present. That second install passes `--no-deps` and no version, which bypasses the `nixl` meta package's `nixl-cu*==` constraint and resolves whatever nixl-cu${CUDA_MAJOR} is newest on PyPI. The image therefore silently ships a different NIXL than it pins: with `nixl == 1.3.1` in kv_connectors.txt, the CUDA 13 image ends up with `nixl 1.3.1` / `nixl-cu12 1.3.1` / `nixl-cu13 1.3.2`. Because the `nixl` meta package dispatches on torch.version.cuda and re-exports nixl_cu13's bindings, 1.3.2 is what actually loads while the pin reads 1.3.1. 1.3.2 fixes a telemetry-gating bug in 1.3.1 where an agent created with telemetry disabled leaves getXferTelemetry returning NIXL_ERR_NO_TELEMETRY, which breaks callers that request transfer telemetry. [Documented in this Dynamo issue](https://github.com/ai-dynamo/dynamo/issues/12952) This PR attempts to fix the silently drifting nixl backend package version, and capture fix for above issue by moving to latest nixl version. ## Fix - ~~Pin the version in the command to update nixl-cu${CUDA_MAJOR}~~ (changed to below after discussion with stakeholders) - Remove the redundant nixl-cu reinstallation. - Bump the pin to nixl 1.3.2, the version CUDA 13 images have been shipping in practice. - Bump the nixl version used by XPU build as well to fix the test that showed this issue. ## Test Plan - Run the test in https://github.com/ai-dynamo/dynamo/issues/12952 ## Test Result Image built with this PR: ``` ============ Serving Benchmark Result ============ Successful requests: 100 Failed requests: 0 Maximum request concurrency: 8 Benchmark duration (s): 55.70 Total input tokens: 102399 Total generated tokens: 25600 Request throughput (req/s): 1.80 Output token throughput (tok/s): 459.61 Peak output token throughput (tok/s): 594.00 Peak concurrent requests: 16.00 Total token throughput (tok/s): 2298.01 ``` Image built without this PR: ``` ### Benchmark result: Traceback (most recent call last): File \"/usr/local/bin/vllm\", line 10, in sys.exit(main()) ^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py\", line 95, in main args.dispatch_function(args) ...... File \"/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py\", line 2095, in main_async benchmark_result = await benchmark( ^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py\", line 865, in benchmark raise ValueError( ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Never received a valid chunk to calculate TTFT.This response will be marked as failed! ### Error in Decode upon first request E0730 15:51:33.763552 436 nixl_agent.cpp:1291] getXferTelemetry: cannot return values when telemetry is not enabled. (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] NIXL transfer failure: transfer_exception. Marking blocks as invalid | Context: {'failure_type': 'transfer_exception', 'request_id': 'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd', 'engine_id': '4c7d2274-6256-41df-82c2-adad8eb2c5b4', 'remote_engine_id': '85d325a3-20c1-469a-a18c-ca82378bac38', 'remote_request_id': 'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-8b2a6ba3', 'remote_host': '192.168.200.198', 'remote_port': 20098, 'num_local_blocks': 16, 'num_remote_blocks': 16, 'local_block_ids_sample': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]} (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] Traceback (most recent call last): (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] File \"/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py\", line 2108, in _pop_done_transfers (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] res = self.nixl_wrapper.get_xfer_telemetry(handle) (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] File \"/opt/venv/lib/python3.12/site-packages/nixl_cu12/_api.py\", line 670, in get_xfer_telemetry (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] return self.agent.getXferTelemetry(handle._handle) (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=436) ERROR 07-30 15:51:33 [base_worker.py:2123] nixl_cu12._bindings.nixlNoTelemetryError: NIXL_ERR_NO_TELEMETRY (EngineCore pid=436) WARNING 07-30 15:51:33 [base_worker.py:1989] Skipping KV post-processing for failed request ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd (EngineCore pid=436) ERROR 07-30 15:51:33 [scheduler.py:2804] Failing 1 request(s) due to KV load failure (failure_policy=fail, 1024 tokens affected). Request IDs: {'ae6b8d99-5ac2-4af5-a1f4-c582fc3a8a26-83d1d9cd'} ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51777", "createdAt": "2026-08-11T07:17:48Z", "updatedAt": "2026-08-13T06:26:40Z", "timestamp": "2026-08-13T06:26:40Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "intel-gpu", "ci/build", "kv-connector", "nvidia" ], "author": "sandeep-maddipatla", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e1a8ec0eb8695e912d7a", "signalId": "github:vllm-project/vllm:pull_request:43671", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43671", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][MiniMax-M2] Fix EAGLE-3 aux hidden-state layer off-by-one", "text": "## Summary When EAGLE-3 speculative decoding is enabled for `MiniMaxAI/MiniMax-M2.5`, the auxiliary hidden states fed to the draft head are taken from the wrong decoder layer, causing position-0 acceptance to collapse to ~10% (vs ~45% expected) and making MTP a throughput *regression* vs no speculative decoding. The root cause is an off-by-one between the two halves of the existing EAGLE-3 plumbing: - `MiniMaxM2Model.forward` (already in vLLM) captures aux state AFTER each decoder layer: ```python for idx, layer in enumerate(islice(self.layers, ...)): hidden_states, residual = layer(positions, hidden_states, residual) self._maybe_add_hidden_state(aux_hidden_states, idx + 1, hidden_states, residual) ``` So a stored index of `k` captures the activation AFTER layer `k-1` ran. - The default `SupportsEagle3.set_aux_hidden_state_layers` passes the draft-config indices straight through: ```python parent_ref.model._set_aux_hidden_state_layers(layers) ``` Net effect: a config that asks for \"layer k\" gets \"output of layer k-1\". The SGLang reference implementation -- which is what the upstream `thoughtworks/MiniMax-M2.5-Eagle3` draft head was trained against -- compensates for the very same offset in [`set_eagle3_layers_to_capture`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/minimax_m2.py): ```python self.model.layers_to_capture = [val + 1 for val in layer_ids] ``` This PR overrides `set_aux_hidden_state_layers` on `MiniMaxM2ForCausalLM` to apply the same +1 shift, so vLLM and SGLang capture identical auxiliary tensors for the same draft config. ## Test plan Bench: `MiniMaxAI/MiniMax-M2.5` + `thoughtworks/MiniMax-M2.5-Eagle3`, TP=4 on AMD MI350, `quantization=fp8`, `attention-backend=ROCM_AITER_UNIFIED_ATTN`, random in=1000 / out=100, `num_speculative_tokens=3`, `method=eagle3`, `temperature=0`, `ignore_eos=true`, `max_concurrency=16`, num_warmups=8, num_prompts=64. | Metric | Before fix | After fix | |------------------------------|-----------:|----------:| | Pos-0 acceptance | ~10 % | 46.5 % | | Pos-1 acceptance | ~5 % | 27.8 % | | Pos-2 acceptance | ~2 % | 16.3 % | | Mean acceptance length | ~1.1 | 1.90 | | Total tok/s (MTP / no-MTP) | ~0.9× | 1.0×* at conc=16; up to 1.73× at conc=1 | ## Notes - The fix is intentionally local to `MiniMaxM2ForCausalLM` rather than changing the `EagleModelMixin._maybe_add_hidden_state` indexing scheme, because other EAGLE-3-supporting models in the tree may already rely on the existing \"after-layer\" indexing convention. - An alternative would be to align `MiniMaxM2Model.forward` with SGLang's \"before-layer\" capture point instead of overriding the setter; both are equivalent in semantics but the override is the minimum-surface change.", "url": "https://github.com/vllm-project/vllm/pull/43671", "createdAt": "2026-05-26T13:15:58Z", "updatedAt": "2026-08-13T06:26:22Z", "timestamp": "2026-08-13T06:26:22Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "ready", "needs-rebase", "v1", "mrv2" ], "author": "pbkowalski", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:609f2ec8be168d29019a", "signalId": "github:vllm-project/vllm:pull_request:51718", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51718", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[6/N][KV-Cache Layout Refactor] Standardize KV cache layout", "text": "## Purpose The core of the KV-cache layout standardization series (RFC #42082). **Stacked on #51704 → #51612** — the diff shown includes them until they land and this retargets. Standardizes every KV cache allocation on the logical `[L, B, H, N, C]` vocabulary: - `KVCacheLayout` enumerates the physical stride permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`, `BLNHC`, `BHLNC`); `reshape_kv_cache` views each flat allocation as dense `[num_blocks, block_bytes]` tiles; per-layer caches are always logical `[B, H, N, C]` views with the physical layout expressed via strides (MLA binds the squeezed `[B, N, C]`). - Layout resolution has a single writer: attention-backend selection publishes the layout into `CacheConfig` (test override > backend-required > `VLLM_KV_CACHE_LAYOUT` > connector preference > `LBNHC`); every consumer reads it from there. - The AITER backends opt into separate K/V head groups (`[B, 2, N, H*hs]`, `LBHNC` required) through `customize_spec`, keeping the fused QK-norm+RoPE+cache kernel dispatched with the exact block-interior contract aiter v0.1.19 asserts — fusion never regresses at any commit in the series. - The KV block zeroer matches main's per-segment form (block dim is always 0 in standardized views); block copies are a plain `unflatten` + index copy; connectors (NIXL, mooncake, moriio, hf3fs, offloading, example) and the attention benchmark runner consume the standardized views and published layout. - `indexes_kv_by_block_stride` and per-backend cache-shape/stride-order hooks are removed — the standardized views make them constant. Replaces #44458 (same content, restacked). ## Test Plan / Result Validated on B200 (full history in #44458): backend-correctness suites (FlashAttention/FlashInfer/Triton/Flex 127P), full MLA suite and `test_prefix_prefill` at exact main failure-set parity, connector unit battery at parity, zeroer/attn-utils/model-runner suites green, e2e smokes across FLASH_ATTN / TRITON(+int8 per-token-head) / FLASHINFER / FLEX / fp8 / CUTLASS_MLA (DeepSeek-V2-Lite). ROCm/AITER validation in progress on AMD. --- AI-assisted (Claude); under review by submitter ## Validation CPU suites (kv-cache-utils, packing, zeroer, attn-utils, gpu-model-runner, nixl / mooncake / moriio / offloading / simple-offload): green. The only failures on this host are environmental and reproduce identically on the base branch (same 5 test ids): GPU 0 is held by another process, and NIXL is not installed. `tests/v1/attention/test_mla_backends.py`: 1729 passed. 20 more OOM'd against a GPU another process had filled; rerun on a reserved GPU they pass (290 passed, 0 failed). E2E on this branch, all deterministic on re-run: | model | exercises | result | |---|---|---| | `Qwen/Qwen3-0.6B` | dense attention | pass | | `deepseek-ai/DeepSeek-V2-Lite-Chat` | MLA specs, compressed KV | pass under LBNHC, LBHNC and BLHNC | | `ibm-granite/granite-4.0-h-tiny` | hybrid attention + Mamba | pass | Not run here (needs a bigger box): DeepSeek-V4 and MiniMax-M3 themselves, so the DSV4 multi-group overlay, the `fp8_ds_mla` 584/656-byte paths, the compact MXFP4 indexer and M3's sparse/indexer backends have unit coverage but no e2e. Those paths are covered by `tests/v1/core/test_contiguous_kv_packing.py` and `tests/v1/worker/test_dsv4_packed_zeroer_geometry.py` using the real DSV4 page sizes. To reproduce e2e: ```bash vllm serve deepseek-ai/DeepSeek-V4 --kv-cache-dtype fp8_ds_mla vllm serve MiniMaxAI/MiniMax-M3 # and repeat each with VLLM_KV_CACHE_LAYOUT=BLHNC ``` --- AI-assisted (Claude); every changed line reviewed and tests run by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/51718", "createdAt": "2026-08-10T18:30:58Z", "updatedAt": "2026-08-13T06:26:04Z", "timestamp": "2026-08-13T06:26:04Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "documentation", "performance", "rocm", "intel-gpu", "ci/build", "deepseek", "cpu", "kv-connector", "nvidia", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2d0438bf1512047c8081", "signalId": "github:vllm-project/vllm:pull_request:50779", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50779", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Extensible (growable) KV cache", "text": "> **Stacked on https://github.com/vllm-project/vllm/pull/51718. Only the commits above that PR's head belong to this one. > This is a **draft** — it cannot merge until https://github.com/vllm-project/vllm/pull/51718 lands, after which the base will be retargeted to `main`. ## Summary Opt-in growable KV cache (`--enable-extensible-kv-cache`) built on the standardized KV cache layout from https://github.com/vllm-project/vllm/pull/51718. This is the productization of the demonstration https://github.com/vllm-project/vllm/pull/47363 from @zhuohan123. vLLM sizes the KV cache from a profiling run, then allocates it up front. Any memory the engine consumes *after* profiling — CUDA graph pools, the spec-decode logits all-gather, workspace growth — has to be guessed at via headroom margins, and a wrong guess is either an OOM or wasted HBM. This change reserves virtual address space for the KV cache capacity but commits physical pages incrementally (CUDA `cuMem*` / ROCm `hipMem*`): 1. Allocate the VA reservation, commit one block per layout segment. 2. Run warmup and CUDA graph capture. 3. Measure *actual* post-warmup free memory and re-size. 4. `extend_kv_cache()` maps the remaining pages under the same base pointer. Because each block keeps a fixed offset within its layout segment and the base pointer never moves, captured graphs and layer views stay valid across the growth — no re-view, no re-capture. --- ### What https://github.com/vllm-project/vllm/pull/51718 buys this The layout standardization removes most of what made the earlier version of this work invasive. Buffer segmentation is now a pure function of `KVCacheLayout.stride_order` (new `num_outer_segments()` helper) instead of per-backend shape/stride/block-dim probing; connector registration views are a plain `narrow(0)` of the logical `[B, H, N, C]` view; and the packed / `block_stride` special cases disappear entirely. The integration is ~30% smaller than the equivalent against `main`. ### Views are authoritative, not `storage.nbytes()` The one contract change worth reviewer attention. Under an extensible cache the untyped storage spans the **reserved** capacity, of which only each view's per-segment block prefix is physically backed. Any code deriving block geometry or registration extents from `untyped_storage().nbytes()` will compute a size that includes unbacked pages. Updated accordingly, with the contract documented on `KVConnectorBase_V1.register_kv_caches`: - `NixlBaseConnectorWorker.register_kv_caches` — collects per-region committed spans and registers those; falls back to the single per-allocation registration when the storage is fully backed. Block-major packing is detected from view strides rather than `nbytes()`. - `get_kv_cache_block_regions` (`v1/worker/utils.py`) — same, which also covers the CPU-offload connector. KV-transfer init is deferred until after `extend_kv_cache`, so connectors only ever register committed memory. On the NIXL path, extend also defragments (release + single-chunk recommit per segment): UCX cannot transfer a VMM region spanning multiple `cuMemCreate` handles. ## Relationship to existing work - **#47363** (@zhuohan123, \"Demo implementation of extensible kv cache memory) — this is the productionization of that demo, not a competing implementation. It adds the V1 + V2 runner integration, the measured post-warmup sizing pass, VMM probe/fallback, sleep-mode support, KV connector support, and a ROCm backend. Happy to fold this into #47363 instead if that's preferred. - **#44458** — strictly a dependency; no overlap in changed behavior. - Searched open PRs for extensible/growable/VMM KV cache work; nothing else overlaps. Contributors: @zhuohan123 (original demo), @zjy0516 (packed-storage bounding, sleep gate). ## Testing Unless noted, run against this branch at its current head. **Unit** — `pytest tests/v1/worker/test_extensible_kv_cache.py tests/utils_/test_extensible_tensor.py tests/v1/engine/test_engine_args.py -v` → 29/29 pass (GB200, aarch64). **e2e, 1×GB200, CUDA graphs on** — outputs byte-identical to a non-extensible baseline for: standard generation, sleep(level=1)/wake, and `--kv-cache-memory-bytes`. **e2e, sizing under pressure** — Qwen3.5-4B TP4, MTP `num_speculative_tokens=100`, chunk 256, `gpu-memory-utilization=0.92` (a config that OOMs without this change): passes in both eager and cudagraph modes. Sizing 35,277→34,721 tokens (eager), 35,243→34,592 (cudagraph). Functional check 64/43 answers correct. **e2e, NIXL P/D disaggregation** — Qwen3-0.6B, HND, block 128, gsm8k via P/D proxy: | config | gsm8k | expected | |---|---|---| | 1P1D eager | 0.422 | ~0.41 | | 1P1D cudagraph | 0.418 | ~0.41 | | P(TP1)→D(TP2) | 0.415 | ~0.41 | | P(TP2)→D(TP2) | 0.415 | ~0.41 | **Model evaluation** — the feature is opt-in and does not change model outputs; the byte-identical-output comparison above and the gsm8k table are the evidence for that. ### Known gaps - The V2 encoder-cache profiling reservation (last commit) is validated on the equivalent branch against `main` (Qwen2-VL-2B: 16,384-token budget reserved, 238,148 blocks committed) but **has not been re-run on this branch** — the cluster battery above has no multimodal configuration. Will re-run before marking ready. - No unit coverage for `commit(defragment=True)` or the connector sleep gate; both are exercised only by the e2e NIXL runs. Adding these before ready. - Multi-node NIXL (NIC RDMA path with the GDR + POSIX-FD allocation flags) is unvalidated; the repro harness is localhost-only. - The ROCm `hipMem*` backend is written but untested on AMD hardware. - Encoder-decoder mixed layouts raise rather than falling back. ## AI assistance AI assistance (Claude Code) was used for this change: for the port onto the #44458 layout model, the connector geometry audit, and drafting. I have reviewed every changed line, and ran the tests and cluster validation above myself.", "url": "https://github.com/vllm-project/vllm/pull/50779", "createdAt": "2026-08-02T22:36:23Z", "updatedAt": "2026-08-13T06:26:03Z", "timestamp": "2026-08-13T06:26:03Z", "metrics": { "reactions": 5, "comments": 17 }, "labels": [ "documentation", "performance", "rocm", "structured-output", "frontend", "intel-gpu", "speculative-decoding", "ready", "needs-rebase", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "kimi", "k3" ], "author": "njhill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:07823bb65f3ecc9951d1", "signalId": "github:vllm-project/vllm:pull_request:52106", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52106", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Core] Restore incremental thinking-budget marker scans", "text": "## Purpose PR #46425 made thinking-budget marker searches incremental by advancing per-search cursors after unsuccessful scans. PR #45984 later removed those cursors while fixing natural `` re-entry and replaced them with a shared `scan_offset` that only advances when a thinking section exits. As a result, while a single long thinking section remains open and the end marker has not been generated, every decode step scans the complete growing output again. The cumulative marker-search work therefore regressed from linear to quadratic. This PR advances `scan_offset` after every marker-search update while retaining `max_marker_len - 1` tokens of overlap. The overlap preserves detection of multi-token markers split across decode steps. Search results are still converted to absolute output indices, and the existing natural-end, forced-end, re-entry, and speculative-decoding state transitions are unchanged. This restores the incremental marker-search behavior introduced by #46425 and regressed by #45984. ## Test Plan - Add a deterministic complexity regression test that instruments `_find_last_sequence_index()` and counts the total number of tokens passed to marker searches. - Assert a linear upper bound based on output length and maximum marker length instead of using a wall-clock threshold. - Run all thinking-budget tests and the complete logits-processor correctness test file. - Run Ruff checks on the modified files. ```text .venv/bin/python -m pytest tests/v1/logits_processors/test_correctness.py -k thinking_budget -q .venv/bin/python -m pytest tests/v1/logits_processors/test_correctness.py -q uvx ruff check vllm/v1/sample/thinking_budget_state.py tests/v1/logits_processors/test_correctness.py uvx ruff format --check vllm/v1/sample/thinking_budget_state.py tests/v1/logits_processors/test_correctness.py ``` ## Test Result ### Complexity regression The table below reports the cumulative number of tokens passed to marker searches while repeatedly appending one token to a long open thinking section. | Revision | 500 steps | 1,000 steps | Growth ratio | |---|---:|---:|---:| | #46425 merge (`7f99e80c3`) | 1,504 | 3,004 | 2.00x | | #45984 merge (`ed908cf0a`) | 126,254 | 502,504 | 3.98x | | Upstream main before this patch | 126,254 | 502,504 | 3.98x | | This patch | 1,504 | 3,004 | 2.00x | The regressed implementation grows approximately fourfold when the sequence length doubles, while the patched implementation grows approximately twofold. ### CPU micro-benchmark The CPU micro-benchmark follows the per-decode-step setup used by #46425: append one token to the full output list on every step and search for an end marker that is not present. Each result is the median of three runs. | Steps | Upstream main | This patch | Speedup | |---:|---:|---:|---:| | 1,000 | 0.021085 s | 0.000700 s | 30.1x | | 2,000 | 0.089897 s | 0.001439 s | 62.5x | | 4,000 | 0.359857 s | 0.002841 s | 126.6x | | 8,000 | 1.462388 s | 0.005671 s | 257.9x | ### Correctness and lint ```text 15 passed, 26 deselected 41 passed All Ruff checks passed 2 files already formatted ``` ## Duplicate-work Check As of August 13, 2026, searches for open PRs containing `thinking_token_budget`, `scan_offset`, and thinking-budget performance did not find another PR addressing this regression. ## AI Assistance AI assistance was used for source-history analysis, implementation, and test preparation. I reviewed the complete diff, verified the algorithmic reasoning, and ran the tests listed above.", "url": "https://github.com/vllm-project/vllm/pull/52106", "createdAt": "2026-08-13T06:24:52Z", "updatedAt": "2026-08-13T06:25:28Z", "timestamp": "2026-08-13T06:25:28Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "Xilluill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8f3b7cb6a4e628e2037f", "signalId": "github:vllm-project/vllm:pull_request:43107", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43107", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][WIP] Check for GPU<->CPU sync during CI", "text": "vLLM now uses asynchronous scheduling by default and in the majority of cases. Performance relies on the absence of any gpu<->cpu synchronizations on the main cuda stream, but such syncs can be opaque and it is easy for them to creep in accidentally. This change adds a `VLLM_GPU_SYNC_CHECK` env var which enables `torch.cuda.set_sync_debug_mode` for the model forward pass and sampler, so that we can easily check for such syncs. I'm trying first to enable it globally in the CI to flush out syncs that need to be fixed or where they are unavoidable and the check needs to be suppressed. A new `gpu_sync_allowed()` context manager can be used to wrap known syncs (i.e. temporary or unavoidable). This PR replaces https://github.com/vllm-project/vllm/pull/40561 - rebased now that man of the exposed \"low hanging\" sync fixes have been merged to main.", "url": "https://github.com/vllm-project/vllm/pull/43107", "createdAt": "2026-05-19T15:45:29Z", "updatedAt": "2026-08-13T06:24:31Z", "timestamp": "2026-08-13T06:24:31Z", "metrics": { "reactions": 3, "comments": 21 }, "labels": [ "rocm", "speculative-decoding", "ready", "needs-rebase", "ci/build", "v1", "multi-modality", "qwen", "kv-connector", "nvidia", "ready-run-all-tests", "quantization", "mistral" ], "author": "njhill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:481c29ef4f2ffb9ae193", "signalId": "github:vllm-project/vllm:pull_request:52059", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52059", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Split MiniMax-M3 prefill index-score K loop", "text": "> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (incorrect expert count caused silent accuracy loss) > > Base image: `vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106` ## Purpose MiniMax-M3 prefill computes index scores by scanning every visible 128-token index-K block for each query block. The current gfx942 kernel uses 64 query tokens per program and processes the full K range in one loop. At long context lengths, this repeatedly loads index-K data and leaves insufficient parallel work for the GPU. ## Suggested Fix For gfx942 prefill with at least 128 query tokens and an 8K sequence length: - Increase the query block size from 64 to 128. - Split the state-free K loop across multiple programs. - Select a power-of-two split count using the query workload, sequence length, and available compute units. - Cap the split count at 16. Each split writes a separate range of score columns, so no atomics or reduction pass are required. Other architectures and smaller shapes retain the existing 64-token, single-split configuration. ## Test Plan ```bash pytest -q tests/kernels/attention/test_minimax_m3.py \\ -k prefill_index_topk_correctness ``` The test covers the existing short-context path and the gfx942 long-context split path. It compares every score written by the kernel, including causal masking, and verifies the final top-k block sets.
Serve command ```bash export HIP_VISIBLE_DEVICES=0,1,2,3 export VLLM_ROCM_USE_AITER=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER_MOE=0 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 vllm serve EmbeddedLLM/MiniMax-M3-FP8-dynamic \\ --served-model-name minimax-m3 \\ --tensor-parallel-size 4 \\ --block-size 128 \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.92 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 32768 \\ --max-num-seqs 32 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --tool-call-parser minimax_m3 \\ --enable-auto-tool-choice \\ --reasoning-parser minimax_m3 \\ --kv-cache-dtype fp8 \\ --language-model-only \\ --port 8000 \\ --attention-backend ROCM_AITER_UNIFIED_ATTN ```
## Test Result Kernel accuracy: 2/2 passed (short-context fallback + long-context split-K). End-to-end serving on 4x MI325X (gfx942), TP4, `EmbeddedLLM/MiniMax-M3-FP8-dynamic`, FP8 KV cache: | ISL/OSL | Concurrency | Metric | Baseline | This PR | Change | |---------|-------------|--------|----------|---------|--------| | 8k/1k | 4 | Output tok/s | 309.18 | 310.03 | +0.3% | | 8k/1k | 8 | Output tok/s | 470.73 | 469.60 | -0.2% | | 128k/1k | 4 | Output tok/s | 93.48 | 95.24 | +1.9% | | 128k/1k | 4 | TTFT median (ms) | 19229 | 18766 | **-2.4%** | | 128k/1k | 8 | Output tok/s | 105.05 | 106.97 | +1.8% | | 128k/1k | 8 | TTFT median (ms) | 34297 | 33430 | **-2.5%** | | 256k/1k | 2 | Output tok/s | 44.19 | 45.69 | **+3.4%** | | 256k/1k | 2 | TTFT median (ms) | 25539 | 24367 | **-4.6%** | | 256k/1k | 4 | Output tok/s | 49.73 | 51.58 | **+3.7%** | | 256k/1k | 4 | TTFT median (ms) | 42422 | 40513 | **-4.5%** | Long-context prefill improvement scales with sequence length. Short-context workloads are unaffected (selector falls back to unsplit path). Long-context prefill shows a consistent TTFT improvement. | Task | Filter | n-shot | Baseline | This PR | |------|--------|--------|----------|---------| | GSM8K (full, 1319) | flexible-extract | 8 | 93.18% ± 0.69 | 92.87% ± 0.71 | | GSM8K (full, 1319) | strict-match | 8 | 93.18% ± 0.69 | 92.80% ± 0.71 |", "url": "https://github.com/vllm-project/vllm/pull/52059", "createdAt": "2026-08-12T22:04:47Z", "updatedAt": "2026-08-13T06:20:04Z", "timestamp": "2026-08-13T06:20:04Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "akii96", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8c9898faa8ff1f34b0f5", "signalId": "github:vllm-project/vllm:pull_request:49688", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49688", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs", "text": "## Purpose CPU GDN attention (Qwen3.5) selects its causal-conv1d implementation via `torch.cpu._is_amx_tile_supported()`: AMX CPUs (Intel GNR) use the C++ kernels (`causal_conv1d_fwd_cpu` / `causal_conv1d_update_cpu`), everything else falls back to `causal_conv1d_fn_cpu` / `causal_conv1d_update_torch`, a pure-PyTorch per-sequence loop. The C++ conv kernels (`csrc/cpu/sgl-kernels/conv.cpp`) do not use AMX tiles but `is_amx` gate blocks it from running on non-AMX platforms that support AVX-512BF16. The float32 SSM state(currently gated by `is_amx` is required for entire CPU GDN path, not just AMX This PR fixes this gating logic. ## Test plan Extended `tests/kernels/mamba/cpu/test_cpu_gdn_ops.py` (all gated on `_is_avx512_bf16_supported()`, so they run on non-AMX CPUs too). Expected Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. ```bash python -m pytest tests/kernels/mamba/cpu/test_cpu_gdn_ops.py -v ``` Qwen-3.5 workload run before and after the patch on both Intel GNR and AMD EPYC Turin ``` VLLM_CPU_OMP_THREADS_BIND=0-63 VLLM_CPU_KVCACHE_SPACE=40 vllm bench throughput \\ --model Qwen/Qwen3.5-9B \\ --trust-remote-code \\ --dataset-name random \\ --input-len 1024 \\ --output-len 1024 \\ --num-prompts 128 \\ --max-num-seqs 32 \\ --dtype bfloat16 ``` Lmeval ``` VLLM_CPU_OMP_THREADS_BIND=0-63 \\ VLLM_CPU_KVCACHE_SPACE=40 \\ lm_eval \\ --model vllm \\ --model_args \"pretrained=Qwen/Qwen3.5-9B,dtype=bfloat16,trust_remote_code=True,max_model_len=4096\" \\ --tasks gsm8k \\ --num_fewshot 5 \\ --batch_size auto ``` ## Test result Python GDN test Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. Workload Qwen3.5-9B, BF16, ISL/OSL 1024/1024, 64-core instance (median of 3 runs) Intel GNR is unchanged (it takes the identical path both ways, so no regression). AMD EPYC Turin gains about 9-10% on this memory-bound, decode-heavy shape, which is the hardest case for a conv speedup to show up. ## Correctness `lm_eval` gsm8k, 5-shot, full set (n=1319), Qwen3.5-9B BF16, before vs after: | Platform | Metric | Baseline | After | |----------|--------|---------:|------:| | Intel GNR (AMX) | exact_match | identical | identical | | AMD Turin (Zen5) | flexible-extract | 0.8696 +/- 0.0093 | 0.8749 +/- 0.0091 | | AMD Turin (Zen5) | strict-match | 0.8734 +/- 0.0092 | 0.8779 +/- 0.0090 | GNR is bit-identical (same code path both ways). On Turin, before/after differ by less than one standard error, so there is no accuracy regression. The C++ conv accumulates in fp32 and is numerically no less accurate than the torch fallback(both round to bf16 at store), which the unit tests also assert against an fp32 oracle. ## Duplicate Check Searched open PRs/issues for GDN / causal_conv1d / Zen / AVX-512BF16 CPU conv work; found none addressing the non-AMX conv fallback. This PR Fixes #49640", "url": "https://github.com/vllm-project/vllm/pull/49688", "createdAt": "2026-07-24T07:35:29Z", "updatedAt": "2026-08-13T06:17:57Z", "timestamp": "2026-08-13T06:17:57Z", "metrics": { "reactions": 3, "comments": 4 }, "labels": [ "bug", "cpu" ], "author": "dineshchitlangia", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a097335dab7d8d4efc76", "signalId": "github:vllm-project/vllm:pull_request:52037", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52037", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Skip unused Jina V5 output layers", "text": "## Purpose Jina Embeddings V5 models are pooling-only, but their vLLM wrappers inherit causal-LM classes. Because these wrappers already declare themselves as pooling models, they bypass the generic pooling adapter that replaces generation-only output layers. The encoder/nano variant therefore retained an unused `ParallelLMHead` with shape `[128256, 768]`. This change applies the existing pooling-model `no_init_weights` pattern to both Jina V5 wrappers, replacing `ParallelLMHead` and `LogitsProcessor` with a `StageMissingLayer`. The decoder/small checkpoint ties its output head to the input embedding (`VocabParallelEmbedding`), so that shared embedding remains intact. On an NVIDIA RTX PRO 6000 Blackwell (TP=1), the nano checkpoint's loaded model state changed as follows: | Metric | Before | After | |---|---:|---:| | Registered parameter bytes | 620,533,248 | 423,532,032 | | Unused `lm_head` bytes | 197,001,216 | 0 | | CUDA memory delta after model load | 1,283,457,024 | 1,086,324,736 | This removes 187.875 MiB of resident parameters (31.75%) and approximately 188 MiB of post-load CUDA memory. This is a resident-memory result; it is not a claim that the transient module-construction peak is eliminated. I searched open vLLM PRs and issues for Jina V5/Embeddings V5 combined with `lm_head`, output-layer, and pooling terms and found no direct or semantic duplicate. #32757 is the already-merged generic pooling mechanism used as the implementation precedent, not another Jina-specific fix. AI assistance (OpenAI Codex) was used to help investigate, implement, and test this change. I reviewed the complete diff and the validation results. ## Test Plan - Load and embed two natural prompts with both the base and patched `jinaai/jina-embeddings-v5-text-nano` checkpoints; compare parameter and post-load CUDA memory plus float32 embedding bytes. - Repeat the embedding comparison with `jinaai/jina-embeddings-v5-text-small` to cover its tied embedding. - Run the existing Jina pooling-model, adapter, and registry tests, then all pre-commit hooks applicable to the changed production file. Commands: ```bash .venv/bin/python -m pytest -q \\ tests/models/language/pooling/test_jina_embeddings_v5.py \\ tests/models/test_adapters.py \\ 'tests/models/test_registry.py::test_registry_imports[JinaEmbeddingsV5Model]' uvx pre-commit run --files \\ vllm/model_executor/models/jina.py git diff --check ``` ## Test Result - Pytest: **12 passed**. - Pre-commit: all applicable hooks passed, including Ruff, formatting, mypy, SPDX, forbidden-import, and configuration checks. - Nano real-model A/B: - Both runs produced embedding SHA-256 `f355f21e03004acfc86af056b3610282290563ab563dd5f9d04e5768bca06688`. - Both output vectors had shape `[768]` and L2 norm `1.0`. - Resident parameter and CUDA memory reductions are reported above. - Small real-model A/B: - Both runs retained 1,192,099,840 parameter bytes and no independent `lm_head`, confirming the tied input embedding was not removed. - Both produced embedding SHA-256 `c740490981d58e61cee407638b98c0ad724674e11341bc94086a7fa19a675942`. - Both output vectors had shape `[1024]` and L2 norm `1.0`.", "url": "https://github.com/vllm-project/vllm/pull/52037", "createdAt": "2026-08-12T19:00:55Z", "updatedAt": "2026-08-13T06:14:38Z", "timestamp": "2026-08-13T06:14:38Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "verified" ], "author": "BabyDrangoner", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:4c134508cead91e27c40", "signalId": "github:vllm-project/vllm:pull_request:51931", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51931", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Use VLLMValidationError in pooling input validation", "text": "## Purpose Part of #48227. Like #51753, this is an independent file-level Step 5 migration. Migrate four caller-caused validation errors in `vllm/entrypoints/pooling/base/io_processor.py` from raw `ValueError` to `VLLMValidationError`: - conflicting offline pooling tasks - untrusted request-level chat templates - mismatched prompt and pooling parameter counts - mismatched prompt and LoRA request counts This preserves the existing validation behavior and error messages. The `parameter` and `value` fields remain unset because the shared helpers serve both online and offline entrypoints whose public argument names differ. The invalid request type, unsupported internal render parameters, and priority-sequence mismatch remain raw `ValueError`s because they are internal guards or are not reachable through current public pooling paths. The existing `LLM.encode()` integration assertion is updated to expect the new semantic exception type. Online serving continues to return HTTP 400, while offline callers now receive `VLLMValidationError` instead of `ValueError`. ## Duplicate-work check I searched open PRs for #48227, `PoolingIOProcessor`, `VLLMValidationError`, and the affected validation messages. I found no open PR migrating these errors. PR #41890 touches the same file for an unrelated import refactor and does not overlap semantically. AI assistance disclosure: Codex assisted with implementation, test generation, and review. I reviewed and understand all changed lines and validated the behavior locally. ## Test Plan ```bash .venv/bin/python -m pytest tests/entrypoints/pooling/test_io_processor.py tests/entrypoints/pooling/scoring/test_utils.py -q .venv/bin/python -m pytest tests/entrypoints/pooling/basic/test_encode.py --collect-only -q env PRE_COMMIT_HOME=/private/tmp/vllm-pre-commit-cache .venv/bin/pre-commit run --from-ref origin/main --to-ref HEAD ``` ## Test Result ```text 8 passed in 0.72s 2 integration tests collected successfully All applicable pre-commit hooks passed. ``` The model-backed `tests/entrypoints/pooling/basic/test_encode.py` suite was not executed locally because it starts a vLLM model. Its updated assertion is covered by CI. Model evaluation is not applicable because this change does not affect inference output.", "url": "https://github.com/vllm-project/vllm/pull/51931", "createdAt": "2026-08-12T06:11:43Z", "updatedAt": "2026-08-13T06:13:25Z", "timestamp": "2026-08-13T06:13:25Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "frontend", "ready", "verified" ], "author": "frank-suwen", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a0963dcd1a7c065bb59e", "signalId": "github:vllm-project/vllm:issue:40756", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:40756", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MTP speculative decoding crash with illegal memory access on long sequences (Qwen3.6-27B-FP8, v0.19.1)", "text": "### Your current environment
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug Description When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on long requests. Environment vLLM version: 0.19.1 Model: Qwen3.6-27B-FP8 TP size: 4, fp8 quantization, prefix caching + chunked prefill enabled Speculative config: method=mtp, model=same as target, num_spec_tokens=5 Symptoms The crash occurs after the request has accumulated ~26 k total tokens and has generated >1200 output tokens. Just before the crash, spec metrics become abnormal: accepted tokens equal drafted tokens, acceptance rate jumps to 100%, and the scheduled draft tokens are all -1. All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)). Excerpt from logs SpecDecoding metrics: Mean acceptance length: 6.00, ..., Avg Draft acceptance rate: 100.0% scheduled_spec_decode_tokens={...: [-1, -1, -1, -1, -1]} ... torch.AcceleratorError: CUDA error: an illegal memory access was encountered File \"gpu_model_runner.py\", line 1706, in _prepare_input_ids prev_common_req_indices_tensor = torch.tensor( To reproduce Serve Qwen3.6-27B-FP8 with MTP speculative decoding using --num-speculative-tokens 5. Send a conversation that grows to 25k+ tokens total. The crash typically happens after 1000+ tokens have been generated. Expected behavior Generation should continue without invalid draft tokens or illegal memory access. Additional notes GPU memory usage was low (~5.4% KV cache), so it’s not an OOM issue. The problem is reproducible; disabling speculative decoding avoids the crash. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/40756", "createdAt": "2026-04-24T02:28:03Z", "updatedAt": "2026-08-13T06:08:26Z", "timestamp": "2026-08-13T06:08:26Z", "metrics": { "reactions": 14, "comments": 36 }, "labels": [ "bug" ], "author": "SongXiaoMao", "state": "open", "assignees": [ "z1ying" ], "change": "updated" } }, { "id": "event:ce7ef8d5cce212ea000a", "signalId": "github:vllm-project/vllm:pull_request:50983", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50983", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve the audio duration-limit error through the PyAV fallback", "text": "## Purpose load_audio wraps every failure on its PyAV fallback path as \"Invalid or unsupported audio file.\" including the rejection raised by the VLLM_MAX_AUDIO_DECODE_DURATION_S guard itself. For containers soundfile cannot probe (like WebM), audio over the duration limit therefore reports a misleading invalid-file error, while soundfile-readable formats (mp3, ogg, flac) report the actionable duration message. This PR types the guard's rejection as AudioDurationExceededError and re-raises it ahead of the generic wrap, so only genuine decode failures are normalized. No behavior change for callers catching ValueError. ## Test Plan ``` pytest tests/multimodal/media/test_audio.py ``` New tests synthesize an opus WebM in memory to exercise the PyAV fallback while a garbage-bytes control verifies genuine decode failures keep the generic error. ## Test Result All 12 tests pass (Linux x86_64, CUDA torch). Without the fix, both new rejection tests fail: the duration error is masked as \"Invalid or unsupported audio file.\". With it, WebM over the limit reports the same actionable message as mp3.", "url": "https://github.com/vllm-project/vllm/pull/50983", "createdAt": "2026-08-04T06:33:26Z", "updatedAt": "2026-08-13T06:04:36Z", "timestamp": "2026-08-13T06:04:36Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "multi-modality" ], "author": "ieaves", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b273bf76959f0d2e96f2", "signalId": "github:vllm-project/vllm:pull_request:51969", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51969", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Security] Enforce server-side num_frames ceiling in VideoMediaIO merge", "text": "## Summary - **Fixes the incomplete CVE-2026-34755 remediation** reported in [GHSA-vxqj-p4gw-9h4c](https://github.com/vllm-project/vllm/security/advisories/GHSA-vxqj-p4gw-9h4c): request-level `media_io_kwargs.video.num_frames=-1` (or an arbitrarily large positive value) could override the engine frame-count ceiling, restoring the unbounded `video/jpeg` decode path that PR #38636 was meant to close. - **Adds server-enforced clamping** in `VideoMediaIO.merge_kwargs`: when `runtime_kwargs` are present, the merged `num_frames` is clamped to the engine ceiling (positive engine value → hard max; engine `-1` → operator unlimited; absent → constructor default of 32). Requests may lower but never raise the ceiling. - **Covers the fps-only override path** where the existing fps/num_frames mutual-exclusion logic could pop the engine ceiling away, leaving the effective value unguarded. ## Test plan - [x] `TestMergeKwargsNumFramesCeiling`: 10 merge-policy unit tests covering clamp of `-1`, oversized, lower-allowed, engine unlimited, absent/empty defaults, fps-only wipe, and no-runtime passthrough. - [x] `test_merge_then_load_base64_enforces_ceiling`: integration test through `merge_media_io_kwargs` → `VideoMediaIO.load_base64(\"video/jpeg\", ...)` confirming frame count is bounded. - [x] All 46 tests in `tests/multimodal/media/test_video.py` pass (including pre-existing GPU-backend and decode tests). - [x] All pre-commit hooks pass (ruff, mypy, typos, etc.). ```bash .venv/bin/python -m pytest tests/multimodal/media/test_video.py -v # 46 passed in 6.87s ``` Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/51969", "createdAt": "2026-08-12T09:12:17Z", "updatedAt": "2026-08-13T06:03:58Z", "timestamp": "2026-08-13T06:03:58Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "multi-modality" ], "author": "jperezdealgaba", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:41d33dbf0523a009d0d5", "signalId": "github:vllm-project/vllm:issue:52088", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52088", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[CI Failure][Anthropic] /v1/messages returns 500 when stop_sequences has more than 4 items", "text": "### Your current environment Observed in vLLM Buildkite CI while running PR #48215. ### Describe the bug The stateless OpenAPI Schemathesis test for `POST /v1/messages` can generate an Anthropic request containing more than four `stop_sequences`. The Anthropic request schema accepts the payload, but converting it to `ChatCompletionRequest` raises a Pydantic validation error because the OpenAI `stop` field permits at most four items. The validation exception is exposed as an HTTP 500 response, so the schema test reports a server error: ```text SUBFAILED entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/messages] schemathesis.core.failures.ServerError: Server error [500] Internal Server Error: 2 validation errors for ChatCompletionRequest stop.str Input should be a valid string [type=string_type, input_type=list] stop.list[str] List should have at most 4 items after validation, not 6 ``` The generated request contained six values: ```json { \"stop_sequences\": [\"sequence-1\", \"sequence-2\", \"\", \"sequence-4\", \"sequence-5\", \"sequence-6\"] } ``` This appears to be a mismatch between the `/v1/messages` OpenAPI request schema and the constraints applied during Anthropic-to-OpenAI request conversion. ### Expected behavior The OpenAPI schema should constrain `stop_sequences` to the supported maximum, or the conversion validation error should be returned as a client error (4xx). Invalid user input should not result in HTTP 500. ### Additional context - Failing test: `entrypoints/openai/test_openai_schema.py::test_openapi_stateless[POST /v1/messages]` - CI build: https://buildkite.com/vllm/ci/builds/83486 - Encountered while testing https://github.com/vllm-project/vllm/pull/48215; that PR changes Ultravox/Whisper model code and appears unrelated to the Anthropic entrypoint. ### Before submitting a new issue... - [x] I searched existing issues and PRs for `test_openapi_stateless`, `/v1/messages`, `stop_sequences`, and the Pydantic maximum-length error. I found related entrypoint/schema issues, but no issue covering this specific failure.", "url": "https://github.com/vllm-project/vllm/issues/52088", "createdAt": "2026-08-13T03:18:56Z", "updatedAt": "2026-08-13T06:00:13Z", "timestamp": "2026-08-13T06:00:13Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "arthurgao2003", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e70ce3e695a861178345", "signalId": "github:vllm-project/vllm:pull_request:48215", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48215", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][LoRA] Add tower/connector LoRA support for Ultravox", "text": "# Purpose Part of #31479: enable `enable_tower_connector_lora` for Ultravox. This is not duplicate work. The open-PR and issue checks for #31479 show this as the only Ultravox tower/connector LoRA implementation; the other open PRs cover different multimodal model families. Ultravox previously could not apply LoRA to its audio tower and connector: - The tower/projector path was not fully built from LoRA-wrappable vLLM modules, so tower/connector adapters could silently become no-ops. - The number of tower and connector rows processed per audio item could not be recovered exactly from the multimodal placeholder count. - Ultravox's variable-length Whisper chunks require padding-aware attention, while tower/connector LoRA requires stable per-chunk row counts. # Implementation - Reuse the native `WhisperEncoder` through a thin Ultravox subclass instead of maintaining a separate encoder implementation. - Extend Whisper encoder self-attention to accept optional multimodal `cu_seqlens` metadata. Each chunk's valid frames and padding tail are separate sequences, so attention cannot cross the valid/padding boundary while all rows still pass through LoRA-wrapped linears. Decoder and cross-attention behavior is unchanged. - Use vLLM-native linear layers for the Ultravox connector. - In tower/connector LoRA mode, pad every audio chunk to the Whisper tower's full context. This makes every chunk contribute exactly `max_source_positions` tower rows and `ceil(max_source_positions / stack_factor)` connector rows. - Add exact `get_num_mm_encoder_tokens` and `get_num_mm_connector_tokens` mappings. The connector count is computed per chunk because `StackAudioFrames` pads each chunk before stacking; a single floor division over all encoder tokens would undercount multi-chunk audio. - Process the encoder batch in one pass under tower/connector LoRA so the token-to-LoRA mapping remains aligned across every chunk. - Load Whisper tower weights with `AutoWeightsLoader`, a `WeightsMapper` for q/k/v fusion and fc1/fc2 nesting, and Whisper's existing synthetic k-projection bias helper. The PR now contains implementation changes only. The test files added in earlier revisions (`tests/models/multimodal/processing/test_ultravox.py` and `tests/lora/test_ultravox.py`) were removed in `1507808e5` at reviewer request. The manual validation below was performed on `1284ff2a4`, the last revision containing those tests; it is recorded here as historical validation and is not part of the current PR's CI coverage. # Validation performed before test removal Manual validation performed on an NVIDIA A10 host: - Processor / placeholder-count boundaries (0.1s, 29.99s, 30s, 30.02s, 60s, 100s audio): actual placeholder and chunk counts compared against the tower/connector mappings and `StackAudioFrames` — **15 passed**. - Tower-only and connector-only nonzero adapters each changed deterministic output logprobs, confirming that neither code path was a no-op — **1 passed**. - Dynamic LoRA vs. an independently merged checkpoint: the same nonzero adapter was folded into the base weights with `W + (alpha / r) * B @ A`. Both runs used identical tower/connector preprocessing and kernel shapes — **1 passed**. Test-host settings, which did not change the code under test: `VLLM_ENABLE_V1_MULTIPROCESSING=0` avoided a multithreaded-pytest `fork()` deadlock, and a 40% GPU-memory budget per engine allowed the two sequential reference engines to coexist during in-process cleanup. The processing test used an equivalent cached local mirror because the test host could not access the gated Meta repository. # Tests run Local, on the two files that remain in this PR: - `.venv/bin/pre-commit run --files vllm/model_executor/models/ultravox.py vllm/model_executor/models/whisper.py` — passed. - `.venv/bin/pre-commit run mypy-3.12 --hook-stage manual --files vllm/model_executor/models/ultravox.py vllm/model_executor/models/whisper.py` — passed. # Model evaluation result The dynamic tower/connector LoRA path and the independently merged checkpoint produced identical 32-token greedy paths; every chosen token's logprob matched within `rtol=0, atol=2e-2`. The tower-only and connector-only adapters each produced nonzero logprob shifts against the same base run. --- ## AI assistance AI assistance (Claude and OpenAI Codex) was used to author and test this change. The submitter has reviewed every changed line and can explain and defend the implementation and validation.", "url": "https://github.com/vllm-project/vllm/pull/48215", "createdAt": "2026-07-10T06:09:30Z", "updatedAt": "2026-08-13T05:58:00Z", "timestamp": "2026-08-13T05:58:00Z", "metrics": { "reactions": 2, "comments": 29 }, "labels": [ "ready", "multi-modality" ], "author": "arthurgao2003", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:5f75e53051cd525ef671", "signalId": "github:vllm-project/vllm:pull_request:49596", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49596", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Enable XPU blockfp8 for DSv3", "text": "## Purpose Enable DeepSeek-V3 on Intel XPU through the oneDNN block-scaled FP8 GEMM (XPUFp8BlockScaledMMKernel). Three issues blocked correct execution: 1. oneDNN requires N to be a multiple of block_n. Some DeepSeek-V3 block-FP8 projections have a per-partition N that isn't 128-aligned. We now pad the weight's N up to the next multiple of block_n in `process_weights_after_loading`, record the unpadded size, and drop the padded output columns after the GEMM in `apply_block_scaled_mm`. 2. Block-scale layout mismatch during weight dequant. The XPU kernel transposes the block scale to `[K/block_k, N/block_n]` layout at load time. `get_and_maybe_dequant_weights` (used by MLA to recover kv_b_proj for weight absorption) assumes the checkpoint `[N/block_n, K/block_k]` layout and asserts on the mismatch. We record `layer.weight_scale_transposed = True` when the kernel repacks the scale and undo the transpose in the dequant helper before scaled_dequantize. 3. Make the shared block base padding-agnostic. `Fp8BlockScaledMMLinearKernel.apply_weights` previously computed the output shape from `weight.shape[0]`, which is the padded N on XPU. It now derives the output shape from the actual GEMM output width (output.shape[-1]) after the matmul. This is a no-op for all non-padding backends (Triton / DeepGEMM / Cutlass / FlashInfer / Aiter-ROCm / CPU), where the returned width already equals `weight.shape[0]`, and it lets the XPU kernel return a sliced result that reshapes correctly. ## Test Plan Run a block-FP8 DeepSeek-V3 checkpoint on Intel XPU Max 1550 and confirm it (a) loads without the oneDNN N-divisibility failure and without the scaled_dequantize shape assertion in the MLA process_weights_after_loading, and (b) produces coherent generations. This did require using two workarounds which were outside the scope of this PR 1. Replace `torch.ops._C_cache_ops.getMemoryInfo(device)` with `torch.xpu.get_mem_info(device)` due to outdated L0 drivers on Max 1550 installation. 2. Topk workaround as detailed in https://github.com/vllm-project/vllm-xpu-kernels/pull/451#discussion_r3626844247 ## Test Result Before: DeepSeek-V3 block-FP8 failed to run on XPU. After: The model loads and generates on XPU (TP×PP×EP, --enforce-eager); the padded kv_a_proj layers run through oneDNN and the MLA kv_b_proj dequant succeeds. A partial run of gsm8k (limit: 40) shows high accuracy: ``` vllm ({'pretrained': 'deepseek-ai/DeepSeek-V3', 'tensor_parallel_size': 4, 'pipeline_parallel_size': 3, 'enable_expert_parallel': True, 'enforce_eager': True, 'max_model_len': 2048, 'max_num_seqs': 32, 'gpu_memory_utilization': 0.95, 'kv_cache_memory_bytes': 1487067239, 'dtype': 'bfloat16'}), gen_kwargs: ({'max_gen_toks': 320}), limit: 40.0, num_fewshot: 5, batch_size: auto |Tasks|Version| Filter |n-shot| Metric | |Value| |Stderr| |-----|------:|----------------|-----:|-----------|---|----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ | 0.95|± |0.0349| | | |strict-match | 5|exact_match|↑ | 0.95|± |0.0349| ```", "url": "https://github.com/vllm-project/vllm/pull/49596", "createdAt": "2026-07-23T14:21:04Z", "updatedAt": "2026-08-13T05:54:02Z", "timestamp": "2026-08-13T05:54:02Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "intel-gpu", "quantization" ], "author": "frost-intel", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:751f384fe3561d79d9cc", "signalId": "github:vllm-project/vllm:issue:52105", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52105", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: sparse_mla_attention forward_mha falls through to unimplemented base class under low max-num-batched-tokens, surfaces as scheduler KeyError", "text": "## Environment - vLLM: v0.27.0, source-built with #51538's two commits (`47f6574`, `db39e67`) applied - Model: GLM-5.2 (compressed-tensors WNA16 Int4-Int8), `GlmMoeDsaForCausalLM` - Hardware: 4x DGX Spark GB10 (sm_121, capability 12), TP=4, `mp` executor - Attention backend: `FLASHINFER_MLA_SPARSE_SM120` (the only one that supports this hardware capability plus a sparse indexer, see below) - Speculative decoding: MTP, `num_speculative_tokens=2` - Relevant flags: `--max-model-len 64000 --kv-cache-memory-bytes 4294967296 --gpu-memory-utilization 0.70 --max-num-seqs 4 --max-num-batched-tokens 2048 --compilation-config '{\"cudagraph_mode\":\"NONE\"}'` Found while narrowing a separate concurrent-load hang (vllm-project/vllm#51921) under a reduced-memory diagnostic configuration. Not the same bug as #51921: confirmed by reverting only `--max-num-batched-tokens` from 2048 to production's 8192 with everything else identical, this crash disappears and the original silent hang (#51921) returns instead. ## Bug description Sending a long prompt (20000 tokens) under `--max-num-batched-tokens 2048` crashes the engine reproducibly (2/2 runs, exact same crash site both times): ``` File \"vllm/v1/core/sched/scheduler.py\", line 1761, in update_from_output req_index = model_runner_output.req_id_to_index[req_id] KeyError: 'chatcmpl-' ``` This is downstream of the real failure. All four TP ranks (the scheduler's own rank 0 worker included) hit an identical `NotImplementedError` at the same timestamp: ``` File \"vllm/v1/attention/backend.py\", line 1052, in forward_mha raise NotImplementedError ``` ## Root cause `vllm/v1/attention/backend.py`'s `MLAAttentionImpl.forward_mha` is a non-abstract base-class method that defaults to `raise NotImplementedError`, meant to be overridden by backends that support MHA-style prefill. `sparse_mla_attention.py`'s override contains: ```python if force_dense or (prefill_max_seq_len <= topk_tokens and not force_masked): return super().forward_mha(...) # the unimplemented base-class path ``` Under chunked prefill with `max-num-batched-tokens=2048`, a 20000-token prompt gets split into chunks no larger than 2048 tokens. If `topk_tokens` (the sparse indexer's per-request top-k budget) is >= 2048, every chunk satisfies `prefill_max_seq_len <= topk_tokens`, permanently routing into the unimplemented path instead of the working masked-MHA implementation a few lines below it. Since all four ranks are running the identical batch shape, they hit this simultaneously. Workers die before producing output; the scheduler's `req_id_to_index` `KeyError` on rank 0 is a consequence of that, not an independent bug, `req_id_to_index` has no guard anywhere against a request going missing because upstream compute failed. ## Reproduction 1. TP=4, GLM-5.2 (or any model on the `sparse_mla_attention.py` code path with a small enough `topk_tokens`), `--max-num-batched-tokens` set low enough that a long prompt's chunk size can be <= `topk_tokens`. 2. Send a prompt whose length exceeds `max-num-batched-tokens`, so chunked prefill kicks in. 3. Every rank raises `NotImplementedError` in `forward_mha` at the same moment; the API returns `500` for all in-flight requests. ## Suggested fix Either implement `forward_mha` in the base class (or raise a clear, catchable, user-facing error instead of a bare `NotImplementedError` that surfaces as an unrelated scheduler `KeyError`), or have `sparse_mla_attention.py`'s branch condition account for cases where the base class path is genuinely unimplemented rather than routing into it unconditionally. Happy to provide full four-rank tracebacks and the exact launch config if useful.", "url": "https://github.com/vllm-project/vllm/issues/52105", "createdAt": "2026-08-13T05:52:25Z", "updatedAt": "2026-08-13T05:52:25Z", "timestamp": "2026-08-13T05:52:25Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9ace529a27a7bdf954aa", "signalId": "github:vllm-project/vllm:pull_request:50897", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50897", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Add lookahead-aware prefix cache hashing for EAGLE-style draft models", "text": "## Purpose Implement https://github.com/vllm-project/vllm/issues/50438 EAGLE-style draft KV at a block boundary depends on both the target prefix through that block and the finalized token immediately following it. The existing cache key proves only the block prefix, so vLLM conservatively drops the last matched unit and recomputes it. That cost is especially visible on hybrid full-attention + Mamba/GDN models whose aligned cache unit can contain hundreds of tokens. The new key includes the successor token and its input identity. A successful match therefore proves that the boundary input consumed by the EAGLE-style drafter is equal, allowing the final matched draft-attention block to be reused without the unconditional drop. ## Design - Maintain the existing target block hashes and an EAGLE-style successor-aware hash list on each request. - Emit an EAGLE-style hash only after the following finalized token is available, so an exact-boundary tail remains pending. - Use successor-aware hashes for EAGLE-style draft-attention cache groups and keep normal hashes for target-only Mamba state groups. - Cap cache publication at the number of successor-aware hashes that are ready. - Skip the legacy EAGLE-style last-block drop only when successor-aware hashing is active. - Preserve the existing conservative drop as the fallback for all unsupported configurations. The mode is enabled through `SpeculativeConfig.use_eagle()`, covering EAGLE, EAGLE3, MTP, DFlash, and DSpark. Successor-aware keys are used for colocated caching and connectors that explicitly advertise compatible lookup and store semantics (currently Mooncake Store). Other connectors keep the conservative last-block fallback. KV-cache events and resumable requests follow the same materialization-aware lifecycle. ## Correctness For a block covering `[s, e)`, the EAGLE-style key proves the regular prefix through `e - 1` plus the finalized token at `e`. An identical successor retains the final safe block, while a different successor naturally falls back to the preceding boundary. Mamba state groups do not contain draft-model layers, so they continue to use their normal target-prefix hashes. This also preserves fine-grained aligned-Mamba partial-state hits when the hash unit is smaller than the physical Mamba page. A successor-aware entry is published only after the worker confirms that the corresponding draft KV has been materialized. Truncation and preemption invalidate readiness past the surviving prefix, including for resumable requests. External connectors opt in through an explicit capability; unsupported connectors preserve the legacy fallback rather than consuming keys they cannot reproduce. Mooncake Store carries the successor hashes through lookup, save, and delayed-finish publication. ## Duplicate-work check I checked the RFC and open PRs before submission. No open implementation PR references #50438 or implements successor-aware prefix-cache hashing. - #44986 uses a phase-based rule to skip the pop during prefill; this PR instead makes reuse conditional on a content-based cache-key proof. - #45614 and #48375 repair or preserve the conservative Mamba backoff; this PR strengthens the key so that backoff is unnecessary only when the boundary token is proven equal. - #39171 addresses hybrid coordinator drop behavior but does not add the successor dependency to the cache key. Commands used for the duplicate check: ```bash gh issue view 50438 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"50438 in:body\" gh pr list --repo vllm-project/vllm --state open --search \"lookahead aware prefix cache EAGLE\" gh pr list --repo vllm-project/vllm --state open --search \"EAGLE prefix cache hash successor\" gh pr list --repo vllm-project/vllm --state open --search \"eagle drop block\" ``` ## Model evaluation Model: `Qwen/Qwen3.5-2B`, `method=mtp`, `num_speculative_tokens=1`, `mamba_cache_mode=align`, greedy sampling, eager execution, prefix caching enabled. The same 1,249-token prompt was submitted three times. The resolved attention and Mamba cache unit was 544 tokens. ```text first_cached_tokens=0 second_cached_tokens=544 third_cached_tokens=1088 outputs_equal=True num_drafts=3 ``` The third request reaches the maximum reusable prefix below the logits token: `floor((1249 - 1) / 544) * 544 = 1088`. All three generated token sequences were identical. ### OffloadingConnector evaluation Model: `Qwen/Qwen3.5-2B`, MTP, real `OffloadingConnector`, GSM8K 200 questions, 10-shot, two passes with APC reset between passes. ```text completion_tokens=49085 first_accuracy=0.515 second_accuracy=0.515 output_mismatches=0 store_bytes=82673664 load_bytes=35356672 num_drafts=25294 num_accepted_tokens=23401 acceptance_length=1.925 ``` ### NixlConnector PD evaluation Model: `meta-llama/Llama-3.2-1B-Instruct` with `nm-testing/Llama3_2_1B_speculator.eagle3`, official vLLM `NixlConnector` P/D topology and toy proxy, HND layout, APC enabled, GSM8K 1,319 questions, 5-shot, two passes. ```text completion_tokens=203004 first_accuracy=0.321456 second_accuracy=0.320697 output_mismatches=385 prefill_second_pass_cache_hit_tokens=776992 decode_second_pass_cache_hit_tokens=776992 decode_second_pass_external_cache_hit_tokens=62810 num_drafts=97783 num_accepted_tokens=105318 acceptance_length=2.077 ``` The two passes are not required to produce byte-identical text; both exceed the repository GSM8K lower bound of 0.23 and their accuracy differs by less than 0.001. ## Tests ```bash HF_HOME=/tmp/codex-vllm-hf-cache \\ HUGGINGFACE_HUB_CACHE=/tmp/codex-vllm-hf-cache/hub \\ .venv/bin/python -m pytest -q \\ tests/v1/core/test_kv_cache_utils.py \\ tests/v1/core/test_prefix_caching.py \\ tests/v1/core/test_scheduler.py \\ tests/v1/worker/test_gpu_model_runner.py \\ tests/v1/kv_connector/unit/test_mooncake_store_connector.py \\ tests/v1/kv_connector/unit/test_mooncake_store_coordinator.py \\ tests/v1/kv_connector/unit/test_mooncake_store_scheduler.py \\ tests/v1/kv_connector/unit/test_mooncake_store_worker.py # 533 passed .venv/bin/pre-commit run --files $(git diff --name-only --diff-filter=ACMR c8602c7906...HEAD) # all applicable hooks passed ``` ## AI assistance AI assistance was used to investigate the issue, implement the change, prepare tests, and draft this PR description. The submitting human reviewed the change and is responsible for understanding and defending it end to end.", "url": "https://github.com/vllm-project/vllm/pull/50897", "createdAt": "2026-08-03T17:09:58Z", "updatedAt": "2026-08-13T05:52:24Z", "timestamp": "2026-08-13T05:52:24Z", "metrics": { "reactions": 4, "comments": 8 }, "labels": [ "ready", "needs-rebase", "kv-connector", "mrv2" ], "author": "ZJY0516", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:412edd0ba54540c6de24", "signalId": "github:vllm-project/vllm:pull_request:51927", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51927", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Refactor] Use common sp utils for Qwen3.5 MoE", "text": "## Summary This PR enables the common sequence-parallel path for Qwen3.5 MoE, keeps sequence-sharded tensors across attention/MoE boundaries, and restores full-token outputs only where required. It also preserves the Qwen3.5 MTP path by normalizing the local shard before gathering the final draft-model output. ## End-to-end flow ```text Qwen3_5ForConditionalGeneration.forward -> Qwen3_5Model.forward -> Qwen3NextDecoderLayer.forward -> Qwen3NextAttention.forward -> sp_shard / sp_reduce_scatter -> Qwen3NextSparseMoeBlock.forward -> sp_all_gather (only when the MoE input must be restored) -> _restore_sequence_parallel_outputs -> one packed sp_all_gather for final and auxiliary states MTP: Qwen3_5MTP.forward -> decoder layer -> RMSNorm on the local sequence shard -> sp_all_gather -> trim sequence padding ``` ## Results Model: `Qwen/Qwen3.5-122B-A10B-FP8`, TP=2, DP=2, EP enabled. ### Non-MTP: main vs. this PR The throughput numbers below use the steady-state runs (main's final run and the mean of this PR's final three runs). | Branch | Output throughput | Total token throughput | Mean TTFT | Mean TPOT | GSM8K 5-shot flexible / strict | |---|---:|---:|---:|---:|---:| | main | 1,853.15 tok/s | 20,384.67 tok/s | 1,013.50 ms | 13.68 ms | Not separately rerun | | this PR | 1,843.41 tok/s | 20,277.53 tok/s | 1,117.85 ms | 13.55 ms | 86.66% / 85.14% | Output throughput differs by **-0.53%**, while mean TPOT improves by **0.94%**. ```bash vllm bench serve \\ --model Qwen/Qwen3.5-122B-A10B-FP8 \\ --host localhost --port 8000 \\ --dataset-name random \\ --random-input-len 10240 \\ --random-output-len 1024 \\ --num-prompts 100 \\ --max-concurrency 32 \\ --temperature 0 --ignore-eos \\ --num-warmups 4 lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-122B-A10B-FP8,base_url=http://127.0.0.1:8000/v1/completions,tokenizer=Qwen/Qwen3.5-122B-A10B-FP8,num_concurrent=32,max_retries=3,tokenized_requests=False,timeout=600 \\ --tasks gsm8k --num_fewshot 5 --batch_size 1 \\ --gen_kwargs temperature=0,max_gen_toks=256 ``` ### MTP: this PR MTP was tested with two speculative tokens. A representative shape-stress run used 127 input tokens, 64 output tokens, 32 prompts, and concurrency 32. | Branch | Output throughput | Mean TTFT | Mean TPOT | MTP acceptance | GSM8K 5-shot flexible / strict | |---|---:|---:|---:|---:|---:| | this PR + MTP | 1,421.63 tok/s | 500.94 ms | 9.20 ms | 78.66% | 86.05% / 84.38% | All 32 requests completed successfully. Additional input lengths `1`, `3`, and `17` were also tested to cover small and non-divisible sequence shapes. ```bash vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\ --tensor-parallel-size 2 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --max-model-len 32768 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2,\"max_model_len\":32768}' vllm bench serve \\ --model Qwen/Qwen3.5-122B-A10B-FP8 \\ --host localhost --port 8000 \\ --dataset-name random \\ --random-input-len 127 \\ --random-output-len 64 \\ --num-prompts 32 \\ --max-concurrency 32 \\ --temperature 0 --ignore-eos lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-122B-A10B-FP8,base_url=http://127.0.0.1:8000/v1/completions,tokenizer=Qwen/Qwen3.5-122B-A10B-FP8,num_concurrent=32,max_retries=3,tokenized_requests=False,timeout=600 \\ --tasks gsm8k --num_fewshot 5 --batch_size 1 \\ --gen_kwargs temperature=0,max_gen_toks=256 ``` non-MTP: (main) ``` local-completions ({'model': 'Qwen/Qwen3.5-122B-A10B-FP8', 'base_url': 'http://127.0.0.1:8000/v1/completions', 'tokenizer': 'Qwen/Qwen3.5-122B-A10B-FP8', 'num_concurrent': 32, 'max_retries': 3, 'tokenized_requests': False, 'timeout': 600}), gen_kwargs: ({'temperature': 0, 'max_gen_toks': 256}), limit: None, num_fewshot: 5, batch_size: 1 |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8650|± |0.0094| | | |strict-match | 5|exact_match|↑ |0.8423|± |0.0100| ``` (This PR) ```text |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8666|± |0.0094| | | |strict-match | 5|exact_match|↑ |0.8514|± |0.0098| ``` MTP: ```text |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8605|± |0.0095| | | |strict-match | 5|exact_match|↑ |0.8438|± |0.0100| ```", "url": "https://github.com/vllm-project/vllm/pull/51927", "createdAt": "2026-08-12T05:20:36Z", "updatedAt": "2026-08-13T05:45:27Z", "timestamp": "2026-08-13T05:45:27Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "qwen" ], "author": "gcanlin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:38f930ae00a466267453", "signalId": "github:vllm-project/vllm:pull_request:48560", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48560", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump fsspec from 2024.12.0 to 2026.6.0", "text": "Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2024.12.0 to 2026.6.0.
Commits
  • a245700 changelog (#2049)
  • 652325d Fix async cat ranges on error (#2045)
  • 073aac8 Forward kwargs from cat_ranges to cat_file (#2044)
  • 2818611 FTP: preserve filenames containing whitespace in _mlsd2 (#2043)
  • c879384 Prevent attribute error by checking for 'forced' attribute before flushing ca...
  • 71b8a6b Reflect async _walk correctly (#2040)
  • 5eec9f9 Fix: Omit threading.get_ident() from cache token for async implementations ...
  • 8643878 Small safety improvements (#2039)
  • a1c16ab Honour simple_templates everywhere in referenceFS (#2029)
  • 85facaa Fix infinite recursion in expand_path when resolving paths with glob magic ch...
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=fsspec&package-manager=pip&previous-version=2024.12.0&new-version=2026.6.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) You can trigger a rebase of this PR by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
> **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days.", "url": "https://github.com/vllm-project/vllm/pull/48560", "createdAt": "2026-07-14T04:58:48Z", "updatedAt": "2026-08-13T05:45:15Z", "timestamp": "2026-08-13T05:45:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ci/build", "nvidia", "dependencies" ], "author": "dependabot[bot]", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bb2c56c1f081a8b192d0", "signalId": "github:vllm-project/vllm:pull_request:52096", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52096", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Reduce AITER MLA FP8 BMM warmup sizes", "text": "## Purpose Reduce ROCm AITER MLA FP8 BMM startup work by precompiling only the real decode sizes that are covered by the configured CUDA graph limit. The previous path precompiled every FP8 BMM size from `1..1024` for every MLA layer. This patch keeps a contiguous warmup range, because `MLAAttention.forward_impl` strips CUDA-graph padding before this BMM path, but caps that range at the graph-covered decode size when CUDA graphs are enabled. Larger batches can still JIT later, but they are already outside the captured fast path. The AITER Triton FP8 BMM kernel specializes on the query-row count. Warming sizes that vLLM will not use for captured decode increases model loading time before the server can handle requests. For the tested DeepSeek-V2-Lite ROCm config, `max_cudagraph_capture_size` is `512`, so this changes each MLA layer's FP8 BMM warmup from `1..1024` to `1..512`. Small odd actual decode sizes, such as batch 7, remain covered because the warmup range is still contiguous. This PR was developed with AI assistance. I reviewed the changed code paths, ran the checks below, and verified the ROCm smoke tests on MI300X. ## Duplicate-work check No linked issue exists for this change. I checked for open duplicate work with: - `gh pr list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` - `gh pr list --repo vllm-project/vllm --state open --search \"rocm aiter mla fp8 bmm\"` - `gh pr list --repo vllm-project/vllm --state open --search \"triton_fp8_bmm mla warmup\"` - `gh issue list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` The only close related open PR I found is #52063, which is also mine. It is related but not a duplicate: #52063 changes the runtime K-side MLA projection path, while this PR changes startup precompile bounds for the AITER FP8 BMM path that remains. If #52063 lands first, this PR may need a small rebase in the same `process_weights_after_loading` area. ## Test Plan Local checks: - `pre-commit run --files vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `ruff check vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `python -m py_compile vllm/model_executor/layers/attention/mla_attention.py tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` - `git diff --check` - `pytest -q tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py` RunPod MI300X smoke tests with `deepseek-ai/DeepSeek-V2-Lite`, ROCm AITER MLA, `max_model_len=1024`, `input_len=128`, `output_len=1`: - batch 1 latency smoke - batch 7 latency smoke, to verify an odd real decode size under the graph limit remains covered ## Test Result Local checks: - `pre-commit run --files ...`: passed - `ruff check ...`: passed - `python -m py_compile ...`: passed - `git diff --check`: passed - `PYTHONPATH=. uv run --no-sync --python 3.12 --with-requirements requirements/common.txt --with pytest --with tblib pytest -q tests/kernels/attention/test_rocm_aiter_mla_fp8_bmm_precompile.py`: passed (`9 passed, 14 warnings`) RunPod MI300X smoke results: - batch 1: FP8 BMM precompile changed to `0/512`; model load `58.38s`; generation completed; avg latency `13.84 ms` - batch 7: FP8 BMM precompile changed to `0/512`; model load `57.30s`; generation completed; avg latency `63.4 ms` No JIT-monitor warnings appeared after inference monitor activation in either smoke test.", "url": "https://github.com/vllm-project/vllm/pull/52096", "createdAt": "2026-08-13T04:31:08Z", "updatedAt": "2026-08-13T05:44:49Z", "timestamp": "2026-08-13T05:44:49Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "rohash123", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:10e25f273801d0f9f111", "signalId": "github:vllm-project/vllm:pull_request:52103", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52103", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Carry request provenance through stored events", "text": "## Purpose Carry request provenance through asynchronous KV-offload completions so locally initiated stores and promotions can emit full `BlockStored` events. The current tracker snapshots full payloads at scheduler lookup/store time and keeps them in a global map. This PR instead keeps a lazy event locator with the request and builds the payload when the raw stored event arrives. ```text Request + KV-group geometry | ReqContext event state | existing async job paths | CPU / FS / OBJ completion | OffloadingEvent(key, ReqContext) | full BlockStored ``` ## Changes - Attach request-scoped event state to `ReqContext`. - Lazily resolve a stored key to its full-chunk or partial-tail boundary. - Carry the exact `ReqContext` through CPU, FS, OBJ, cascade, and promotion completion. - Use context identity, not request ID, when an old completion is finalized. - Generate GPU-equivalent per-block `extra_keys`, including multimodal and prompt-embedding inputs. - Keep placeholders for external work, unsupported cache shapes, and unsafe token-mutating requests. - Keep only detached chunk hashes for the existing CPU removal contract; #49413 PR 2 removes that table. - Preserve queued CPU removals and their detached expansion metadata across reset. - Move context ownership from active work to the raw stored event, then release it after translation; failed submissions and shutdown unwind their owners. ## Scope This PR does not change: - `OffloadKey = (block_hash, group_idx)`; - `prepare_store` admission or batching; - worker transfer metadata; - the legacy expanded-removal contract; or - the public KV-event schema. Single-key admission, k-way ordering, and #44865 remain separate work. This PR is based on the KV-group block-granularity fix merged in #51614. Token-mutating paths affected by #49377 keep placeholder payloads until their hash and offload state are safe. ## Duplicate-work check - #49506 is superseded by this narrower request-scoped design. - #51614 is merged and provides the block-granularity base. - #50087 changes request-level store policy/admission. - #52022 fixes store-threshold accounting. - #51646 only updates event-medium documentation. No open PR above carries request provenance through async stored-event completion. ## Tests All validation ran on the workstation against `3d204dfdaa`: ```text .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_events.py -q 32 passed .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py -q 122 passed .venv/bin/python -m pytest \\ tests/v1/kv_offload/cpu/test_manager.py \\ tests/v1/kv_offload/tiering/test_async_lookup.py \\ tests/v1/kv_offload/tiering/test_fs_tier.py \\ tests/v1/kv_offload/tiering/test_obj_tier.py \\ tests/v1/kv_offload/tiering/test_tiering_offloading.py -q 182 passed pre-commit run ruff-format --files \\ $(git diff --name-only origin/main...HEAD -- '*.py') Passed pre-commit run ruff-check --files \\ $(git diff --name-only origin/main...HEAD -- '*.py') Passed pre-commit run mypy-3.12 --hook-stage manual \\ --files $(git diff --name-only origin/main...HEAD -- vllm) Passed ``` Model evaluation: not applicable. This changes event metadata and lifecycle only, not model execution or output. ## AI assistance AI assistance was used to trace the event lifecycle, implement the change, and draft tests. The human submitter must review every changed line and these test results before submission.", "url": "https://github.com/vllm-project/vllm/pull/52103", "createdAt": "2026-08-13T05:44:02Z", "updatedAt": "2026-08-13T05:44:38Z", "timestamp": "2026-08-13T05:44:38Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "kv-connector" ], "author": "Change72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d76e136c80560d26996b", "signalId": "github:vllm-project/vllm:pull_request:42792", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42792", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[WIP][Model Runner V2] support spec decode + mamba align prefix caching", "text": "## Accuracy Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' \\ --default-chat-template-kwargs '{\"enable_thinking\": false}' ``` **Results** ``` | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |---------|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k_cot| 3|flexible-extract| 8|exact_match|↑ |0.7998|± |0.0110| | | |strict-match | 8|exact_match|↑ |0.7968|± |0.0111| ``` ## Performance Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' ``` **Results** | Metric | #42406 | This PR | Delta | |--------|-----------|-----------|-------| | **General** | | | | | Successful requests | 512 | 512 | — | | Failed requests | 0 | 0 | — | | Max request concurrency | 16 | 16 | — | | Benchmark duration (s) | 119.40 | 113.97 | -4.5% | | Total input tokens | 2,167,806 | 2,271,087 | +4.8% | | Total generated tokens | 262,144 | 262,144 | — | | Request throughput (req/s) | 4.29 | 4.49 | +4.7% | | Output token throughput (tok/s) | 2,195.48 | 2,300.20 | +4.8% | | Peak output token throughput (tok/s) | 1,553.00 | 1,040.00 | -33.0% | | Peak concurrent requests | 28.00 | 28.00 | — | | Total token throughput (tok/s) | 20,351.05 | 22,227.97 | +9.2% | | **Time to First Token** | | | | | Mean TTFT (ms) | 282.49 | 105.66 | -62.6% | | Median TTFT (ms) | 145.19 | 84.13 | -42.1% | | P99 TTFT (ms) | 3,800.32 | 995.73 | -73.8% | | **Time per Output Token** | | | | | Mean TPOT (ms) | 6.64 | 6.68 | +0.6% | | Median TPOT (ms) | 6.59 | 6.61 | +0.3% | | P99 TPOT (ms) | 10.03 | 7.95 | -20.7% | | **Inter-token Latency** | | | | | Mean ITL (ms) | 16.35 | 17.04 | +4.2% | | Median ITL (ms) | 10.58 | 15.83 | +49.6% | | P99 ITL (ms) | 105.52 | 38.90 | -63.1% | | **End-to-end Latency** | | | | | Mean E2EL (ms) | 3,677.74 | 3,521.25 | -4.3% | | Median E2EL (ms) | 3,528.08 | 3,468.17 | -1.7% | | P99 E2EL (ms) | 8,904.19 | 4,462.88 | -49.9% | | **Speculative Decoding** | | | | | Acceptance rate (%) | 73.88 | 78.34 | +4.46pp | | Acceptance length | 2.48 | 2.57 | +3.6% | | Drafts | 105,785 | 102,119 | -3.5% | | Draft tokens | 211,570 | 204,238 | -3.5% | | Accepted tokens | 156,310 | 160,003 | +2.4% | | Position 0 acceptance (%) | 83.86 | 85.68 | +1.82pp | | Position 1 acceptance (%) | 63.90 | 71.00 | +7.10pp |", "url": "https://github.com/vllm-project/vllm/pull/42792", "createdAt": "2026-05-15T22:55:32Z", "updatedAt": "2026-08-13T05:44:30Z", "timestamp": "2026-08-13T05:44:30Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "TheEpicDolphin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9f8c0f88ca302c8713da", "signalId": "github:vllm-project/vllm:pull_request:49506", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49506", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Carry chunk index in OffloadKey", "text": "## Purpose Re-scopes this PR to a single, minimal change agreed with the maintainer in review: carry the chunk index inside the `OffloadKey` so the self-describing KV-event path can recover a chunk's provenance from the key alone. `OffloadKey` becomes: ``` block_hash | group_idx:u32 (big-endian) | chunk_idx:u32 (big-endian) ``` where `chunk_idx` is the absolute, group-relative chunk index. This removes the need to thread a separate `chunk_idx` through the events tracker, so the offloading manager, tiering, and worker layers keep treating the key as an opaque identity. This is the design @orozery proposed in review ([suggestion](https://github.com/vllm-project/vllm/pull/49506#discussion_r3655466611)) and I [agreed to re-scope this PR to](https://github.com/vllm-project/vllm/pull/49506#discussion_r3661314636): coupling `chunk_idx` to the key makes single-key `prepare_store` unnecessary for KV events, so that admission work moves to a follow-up. ### What changes - `vllm/v1/kv_offload/base.py`: `make_offload_key(block_hash, group_idx, chunk_idx)` now requires `chunk_idx` (no default, so no caller can silently pack the wrong position); add `get_offload_chunk_idx()`; `get_offload_block_hash()` / `get_offload_group_idx()` slice the widened key. - `.../offloading/scheduler.py`: `update_offload_keys()` numbers each group's new chunks from its current key count (first pass `0, 1`; a later append continues `2, 3`; every group starts from 0). The events tracker no longer takes a separate `chunk_idx`, so `_maximal_prefix_lookup()` drops its `start_chunk_idx` parameter and `_build_store_jobs()`'s `record_store()` call drops it too. `_build_store_jobs()` otherwise keeps `main`'s batch store logic and still computes an internal `chunk_idx` for GPU block indexing. - `.../offloading/events.py`: `record_store()` / `record_lookup()` decode `chunk_idx` from the key via `get_offload_chunk_idx()`. ### Intentionally unchanged - **KV-event wire format** — `BlockStored` / `BlockRemoved` schema and payloads are unchanged; `chunk_idx` lives in the key, not on the wire. - **Store admission, allocation atomicity, worker transfer** — the CPU/tiering managers are byte-for-byte unchanged from `main`, and `_build_store_jobs()` keeps its batch behavior (batch `prepare_store`, batch allocation/eviction/cascade, existing `pending_primary_stores` pairing); its only change is the `record_store()` call dropping the `chunk_idx` argument. - **Dedup** — unchanged under fixed chunk geometry: a given (hash, group) always maps to the same `chunk_idx`, so keys are stable. - **FS layout** — `FileMapper` still keys file names on `(hash, group)`; a non-zero `chunk_idx` does not change the file name. ### P2P compatibility (breaking) P2P transports the full `OffloadKey` bytes opaquely, so producer and consumer peers must run the **same** key format — deploy in lockstep. This PR does not implement mixed-version P2P compatibility; a key-format/version handshake is a follow-up. ### Out of scope (follow-ups) - Single-key `prepare_store` admission (k-way merge, `OffloadReject`) — @orozery agreed this can follow. - Worker-facing transfer reshape — #44865 (sequenced after this). - Full promotion provenance and key-only removal. Not a duplicate: the nearest open neighbor is #44865 (worker transfer data-model reshape), which the review thread sequences *after* this change and which consumes rather than duplicates it. Other open KV-offload PRs (#48798 tiering metrics, #49850 HIT_PENDING bound) are unrelated. ## Test Plan ``` .venv/bin/python -m pytest \\ tests/v1/kv_offload/ \\ tests/v1/kv_connector/unit/offloading_connector/ ``` New/updated behavior tests: - `OffloadKey` round-trip and same-(hash, group) chunk disambiguation (`tests/v1/kv_offload/test_base.py`). - `update_offload_keys()` per-group numbering: `[0, 1]` then a later append `[2, 3]`, and each KV group numbering independently from 0 (`tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py`). - Events tracker selects chunk metadata by key alone (existing order-independence test) and a placeholder decodes the tail hash / group from an extended key without leaking `chunk_idx` onto the wire (`.../test_events.py`). - FS file name unchanged under a non-zero `chunk_idx` (`tests/v1/kv_offload/test_file_mapper.py`). - P2P opaque full-key round-trip over the ZMQ transport (`tests/v1/kv_offload/tiering/p2p/test_zmq_transport.py`). ## Test Result All offloading suites pass locally on an L4 (torch 2.13.0+cu130, Python 3.12): 498 passed across `tests/v1/kv_offload/` and `tests/v1/kv_connector/unit/offloading_connector/`. `ruff check`, `ruff format`, and `mypy` (3.12) are clean on the changed files. --- AI assistance was used to prepare this PR (Cursor). A human author reviewed every changed line and ran the tests above.", "url": "https://github.com/vllm-project/vllm/pull/49506", "createdAt": "2026-07-23T03:22:48Z", "updatedAt": "2026-08-13T05:43:55Z", "timestamp": "2026-08-13T05:43:55Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "needs-rebase", "v1", "kv-connector" ], "author": "Change72", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:be2ace59704adef9dac3", "signalId": "github:vllm-project/vllm:issue:49413", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:49413", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: KV offload event path refactor — provenance-carrying events and key-only removals", "text": "### Motivation. > Code references are pinned to `main` at > `3d204dfdaaf09d67d49c7855630ef949754e0f8f`; implementation starts from > current `main`. [#48679](https://github.com/vllm-project/vllm/pull/48679) added self-describing KV-offload events as an initial implementation. This RFC finishes the event-path refactor in two PRs. Today, `OffloadKey` contains only `(block_hash, group_idx)` ([base.py:23-41](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L23-L41)), and a raw `OffloadingEvent` has no request context ([base.py:153-159](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L153-L159)). The tracker therefore builds the full event payload at scheduler lookup/store time and keeps it until eviction ([events.py:78-114](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/distributed/kv_transfer/kv_connector/v1/offloading/events.py#L78-L114)). ```text lookup / prepare_store | snapshot full payload in a global map | async store / promotion completion | BlockStored | keep part of the snapshot until removal ``` This has two problems: - Async promotion and secondary-tier completions can lose the request provenance needed to build a full `BlockStored`. Promotions already retain `ReqContext` through `PendingPromotion` and `TransferJob` ([manager.py:62-68](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/tiering/manager.py#L62-L68), [manager.py:469-490](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/tiering/manager.py#L469-L490)), but raw events do not carry it. - Full removal metadata stays in vLLM from store until eviction. That lifetime can be much longer than the request, especially for persistent tiers. ### Proposed Change. #### Scope map | PR | Scope | | --- | --- | | **PR 1 — stored-event provenance** | Carry request context through existing async store, cascade, and promotion paths. Build `BlockStored` only when the raw stored event is translated. | | **PR 2 — key-only removal** | Emit only the chunk key in `BlockRemoved`, remove vLLM's long-lived removal table, and move chunk expansion/owner cleanup to the router. | This is the scope agreed in [#49506](https://github.com/vllm-project/vllm/pull/49506#issuecomment-5263904604). `OffloadKey`, single-key admission, and worker transfer reshaping are not part of this RFC. #### PR 1: stored-event provenance ```text Request + KV-group geometry | ReqContext event state | existing async job paths | CPU / FS / OBJ completion | OffloadingEvent(key, ReqContext) | full BlockStored ``` The implementation will: - Keep `OffloadKey = (block_hash, group_idx)`. - Attach request-scoped event state to `ReqContext`, which already supports typed per-request state ([base.py:90-104](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/v1/kv_offload/base.py#L90-L104)). - Lazily resolve an event key to its full-chunk or partial-tail position when the stored event is translated. - Carry the exact `ReqContext` through CPU, FS, OBJ, cascade, and promotion completion. - Generate per-block `extra_keys`, matching GPU events ([kv_events.py:50-76](https://github.com/vllm-project/vllm/blob/3d204dfdaaf09d67d49c7855630ef949754e0f8f/vllm/distributed/kv_events.py#L50-L76)). - Keep placeholder events for external work, unsupported cache shapes, and token-mutating request paths whose hashes are not safe yet. - Keep only a small detached CPU removal record for PR 2 compatibility. It contains chunk hashes and group index, never `Request` or `ReqContext`. The normal lifetime is bounded by existing owners: ```text RequestOffloadState -> GPU store completion TransferJob / PendingPromotion -> secondary or promotion completion raw OffloadingEvent -> event translation ``` Ownership moves with the work: request state -> pending job -> raw stored event -> event translation. Failure and shutdown release their owned context; reset drains in-flight tier work and preserves any still-valid queued secondary stored event until translation. A synchronous failed tier submission unwinds the job and its primary-tier pin. Each job keeps the exact context object, so request-ID reuse cannot bind old work to a new request. PR 1 does not add a new refcount or retain request state until storage eviction. [#51614](https://github.com/vllm-project/vllm/pull/51614) is now merged and provides the KV-group block-granularity base used here. #### PR 2: key-only removal For one offload chunk: ```text chunk members = [H0, H1] chunk key = H1 vLLM: BlockRemoved([H1]) router: find H1's chunk -> remove that owner from H0 and H1 ``` PR 2 will: - Send only the removed chunk key and group. - Delete vLLM's detached removal-expansion table. - Let the router own `chunk key -> members` and member ownership. - Define CPU reset/restart cleanup for the producer owner scope. - Land with the matching router contract and implementation in [Dynamo #11930](https://github.com/ai-dynamo/dynamo/pull/11930). vLLM and router will move together; no compatibility window is planned. FS/OBJ deletion, TTL cleanup, and shared-storage global invalidation remain out of scope. ### Observable behavior. - PR 1: locally initiated promotions and secondary stores can emit full `BlockStored` payloads instead of placeholders. - PR 1: stored events include per-block `extra_keys`; plain-text events may carry explicit `None` entries. - PR 2: `BlockRemoved` changes from expanded block hashes to one chunk key. PR 1 does not change `OffloadKey`, manager admission, batching, worker transfers, or the public KV-event schema. ### Out of scope. - Single-key `prepare_store`, `OffloadReject`, and k-way admission ordering. - Worker transfer reshaping; [#44865](https://github.com/vllm-project/vllm/pull/44865) owns that work. - Adding `chunk_idx` to `OffloadKey`. - Request-side medium/locality filtering; [#48123](https://github.com/vllm-project/vllm/pull/48123) owns that API. - Fixing token-mutating request paths. [#49377](https://github.com/vllm-project/vllm/issues/49377) tracks stale `Request.block_hashes`; PR 1 keeps their event payloads as placeholders until their hash and offload state are safe. - FS/OBJ deletion events and shared-storage global invalidation. ### Implementation order. 1. PR 1: stored-event provenance, based on current `main`. 2. PR 2: key-only removal plus the matching Dynamo owner-index change. ### Feedback Period. The design discussion is complete. Implementation review should focus on event parity, promotion coverage, context cleanup on every terminal path, and the vLLM/router lockstep contract for PR 2. ### CC List. @orozery @ronensc @jthomson04 ### Any Other Things. | Related work | Relationship | | --- | --- | | [#43468](https://github.com/vllm-project/vllm/pull/43468) | Introduced self-describing offload events and expanded removals. | | [#51614](https://github.com/vllm-project/vllm/pull/51614) | Merged KV-group block-granularity fix used by PR 1. | | [#49377](https://github.com/vllm-project/vllm/issues/49377) | Tracks stale request hashes after token truncation. | | [Dynamo #11930](https://github.com/ai-dynamo/dynamo/pull/11930) | Router-side chunk ownership and key-only removal consumer. | AI assistance was used to trace the event lifecycle, prepare this RFC, and implement tests. All code links were verified against `main` at `3d204dfdaaf09d67d49c7855630ef949754e0f8f`. ### Before submitting a new issue... - [x] I searched for relevant issues and asked the documentation chatbot.", "url": "https://github.com/vllm-project/vllm/issues/49413", "createdAt": "2026-07-22T06:14:12Z", "updatedAt": "2026-08-13T05:43:52Z", "timestamp": "2026-08-13T05:43:52Z", "metrics": { "reactions": 1, "comments": 10 }, "labels": [], "author": "Change72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7815133ba358c2eeed5e", "signalId": "github:vllm-project/vllm:pull_request:51772", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51772", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][MLA] Fuse Kimi-K3 chunked-context K/V packing", "text": "## Purpose The K3 MLA layer delegated chunked-context prefill to `impl._compute_prefill_context`, which per chunk casts `kv_nope` to fp8, casts `k_pe`, concatenates `[k_nope | k_pe]`, and re-quantizes a query the fused new-token epilogue had already quantized. This gives the layer its own context loop so that tail collapses into one kernel per chunk: `fused_kimi_k3_mla_kv_concat{,_quant_fp8}` reads the strided `kv_b_proj` output and the gathered `k_pe` in place -- the latter still in its fp8 cache layout -- and writes a contiguous key plus, on the fp8 path, a contiguous fp8 V. Only the gather and `kv_b_proj` remain. Casts to E4M3 use the native pairwise converters, so the result is bit-identical to `.to(torch.float8_e4m3fn)`. It also wires an `out` tensor through `run_prefill_context_chunk`, reusing the existing `supports_out()` capability. A backend honoring it writes each chunk's partial straight into the accumulating context partial, removing the per-chunk output copy for every non-continuation chunk; a continuation still has to merge with the partial already in place, so it keeps its own buffer. Decode context parallelism is unchanged and still uses `impl._context_parallel_compute_prefill_context`. The second commit fixes a bug in the first, found by serving a stock checkpoint before posting: the fused loop originally replaced the impl's per-chunk cast of the gathered latent with a load-time check that `kv_b_proj` consumes fp8 directly. Cache dtype and `kv_b_proj` dtype are independent -- a plain fp8 cache leaves the gather in fp8, but only an fp8-quantized `kv_b_proj` can take that, and a stock Kimi-K3 checkpoint's is bf16. Every rank died at weight load under `--kv-cache-dtype fp8`: ``` AssertionError: Kimi-K3 with a plain fp8 KV cache needs a kv_b_proj that consumes the fp8 gathered latent directly; this checkpoint's kv_b_proj wants torch.bfloat16. ``` The cast is restored in `run_chunk` and the check dropped. It is a no-op `.to` when `kv_b_proj` already consumes the fp8 latent, so the fused path keeps its win; the fusion itself is untouched. ## Not a duplicate Per the contribution policy, checked before opening: ``` gh pr list --repo vllm-project/vllm --state open --search \"Kimi-K3 chunked context MLA\" gh pr list --repo vllm-project/vllm --state open --search \"kv_b_proj\" gh pr list --repo vllm-project/vllm --state open --search \"fuse chunked context K/V packing\" gh pr list --repo vllm-project/vllm --state all --head agent/kimi-k3-fused-context-kv-pack ``` No open PR touches the K3 fused chunked-context prefill path. The nearest neighbours are different work: #37245 fixes an int8 dtype cast for `kv_c_normed` in the *generic* MLA prefill, and #48051 stops re-quantizing a bf16 MLA `kv_b_proj` on ROCm for GLM MoE DSA. Neither touches `vllm/models/kimi_k3/nvidia/mla.py` or the fused pack kernels. ## Test Plan ``` pytest tests/models/kimi_k3/test_mla_prefill_context.py \\ tests/v1/attention/test_mla_prefill_registry.py \\ tests/v1/attention/test_mla_context_chunks.py \\ tests/kernels/attention/test_kimi_k3_mla_fused_epilogue.py \\ tests/kernels/attention/test_kimi_k3_mla_key_concat_kv_cache.py ``` `tests/models/kimi_k3/test_mla_prefill_context.py` is new here. It asserts the fused loop hands the prefill backend the same `(q, k, v)` per chunk, and returns the same merged partial, as the generic impl -- across bf16/fp8 caches, both `kv_b_proj` weight dtypes, and both `supports_out()` modes. ## Test Result **67 passed** (B300, aarch64, CUDA 13). The two `bf16_kv_b_proj` + `fp8` cache cases fail on the first commit alone and pass with the second: ``` FAILED ...test_fused_context_matches_generic_impl[bf16_kv_b_proj-fp8-copy_out] FAILED ...test_fused_context_matches_generic_impl[bf16_kv_b_proj-fp8-writes_out] 2 failed, 7 passed ``` `pre-commit run --files vllm/models/kimi_k3/nvidia/mla.py tests/models/kimi_k3/test_mla_prefill_context.py` passes (ruff check/format, mypy, SPDX, forbidden-imports). ## Model Evaluation Kimi-K3 served end to end in 2P1D PD disaggregation on 8 x 4 B300 -- prefill 2 x TEP8 (`--enforce-eager`, mns 16), decode DEP16/TP1 (mns 12, hybrid LB), NIXL connector, EAGLE3-3 speculative decoding with `rejection_sample_method` at its default `standard`, `--kv-cache-dtype fp8`. That is exactly the configuration that could not load before the second commit. | Eval | Score | Samples | |---|---|---| | GSM8K, 5-shot, `/v1/completions`, temp 0.6 / top_p 0.95 | **0.9575** flexible-extract / **0.9575** strict-match | 1319, 0 request errors | | OCRBench, thinking at effort high, temp 1.0 / top_p 1.0 | **0.892** +- 0.010 | 1000 | I do not have a pre-change baseline for the same configuration, because a stock checkpoint with an fp8 KV cache could not load at all on the first commit alone. The fp8 packing itself is verified exact against torch's cast in the kernel tests, and the fused loop is verified equal to the generic impl chunk-for-chunk in the new test. ## Note AI assistance was used for the second commit (the `kv_b_proj` cast fix, its test, and this description) and for running the evaluation above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/51772", "createdAt": "2026-08-11T06:28:19Z", "updatedAt": "2026-08-13T05:42:58Z", "timestamp": "2026-08-13T05:42:58Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "ready", "nvidia", "kimi", "k3" ], "author": "zyongye", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a9fb36a111c0f90cc36b", "signalId": "github:vllm-project/vllm:pull_request:52066", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52066", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Fix ragged-N FP8 block-scale GEMM and sparse-MLA metadata sync", "text": "Two XPU-only fixes needed to serve sparse-MLA FP8 MoE models (e.g. GLM-5.2, DeepSeek DSA) on Intel GPUs. 1. Ragged-N FP8 block-scale GEMM (scaled_mm/xpu.py): oneDNN derives the weight block-group width as wei_group_n = N/ceil(N/128), so when the output dim N is not a multiple of 128 it is < 128 and oneDNN cannot create the matmul primitive (\"could not create a primitive descriptor\"). This hits the MLA fused projections whose N is ragged (fused_qkv_a_proj N=2624, kv_a_proj_with_mqa N=576). Pad the weight rows up to the 128 boundary with zeros (which contribute nothing), run the GEMM, and slice the padding off the output; no-op when N is already a multiple of 128. Verified numerically: padded ragged-N matches a block-dequant reference at the same ~0.004 rel-err floor as divisible-N. 2. Sparse-MLA metadata sync (xpu_mla_sparse.py): The shared MLA layer (mla_attention.py::forward_impl) unconditionally reads num_decodes/num_prefills/num_decode_tokens on every MLA metadata; the CUDA sparse backends carry them via SparseMLACommonMetadataBuilder, but the XPU sparse backend built its own metadata without them, so a sparse-MLA run on XPU crashed with \"'XPUMLASparseMetadata' object has no attribute 'num_decode_tokens'\". Add the fields and populate them so all tokens route through the top-k sparse MQA path (num_decode_tokens == num_actual_tokens, num_prefills == 0), which this backend already implements for both prefill and decode; this keeps num_mha_tokens at 0 and avoids the dense-MHA prefill branch (whose prefill-only fields this backend does not provide). ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52066", "createdAt": "2026-08-12T23:02:18Z", "updatedAt": "2026-08-13T05:55:37Z", "timestamp": "2026-08-13T05:55:37Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu" ], "author": "libinta", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:27f966710746a6c01f75", "signalId": "github:vllm-project/vllm:pull_request:51901", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51901", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build] Add warning for unsupported global PTX architecture requests in...", "text": "... CMake configuration. ## Purpose Implements one item from #9129: Warn that PTX builds are not currently supported (post [CI/Build] Per file CUDA Archs (improve wheel size and dev build times) #8845), currently if there is a +PTX in TORCH_CUDA_ARCH_LIST this will be ignored. We should warn when this is the case Users can request PTX through `TORCH_CUDA_ARCH_LIST` values such as `8.0+PTX`. vLLM strips the Torch-provided global `-gencode` flags and rebuilds per-source gencode flags, so the global PTX request is not preserved. This PR surfaces that behavior during CMake configuration instead of silently dropping the request. This branch builds on PR #42375 by: 1. Renaming `clear_cuda_arches` to `clear_cuda_gencode_flags` to clarify that it clears `-gencode` flags from `CMAKE_CUDA_FLAGS`, not the CUDA target architectures list itself. 2. Adding documentation to [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) to explain the CUDA architecture and PTX flags normalization behavior. ## Changes - **Refactor**: Renamed `clear_cuda_arches()` macro to `clear_cuda_gencode_flags()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) for clarity. - **Feat**: Added `warn_if_ptx_arch_requested()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and called it in [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) to warn users when PTX code generation is requested via global CUDA architecture flags. - **Tests**: Added pytest unit test in [tests/test_cmake_utils.py](file:///home/swidanagama/vllm/tests/test_cmake_utils.py) for `clear_cuda_gencode_flags()`. - **Docs**: Added a note in [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) explaining CUDA architecture and PTX flags normalization. ## Test Plan Tested behavior by: 1. Running pytest unit tests in [tests/test_cmake_utils.py](tests/test_cmake_utils.py): ```bash .venv/bin/python -m pytest tests/test_cmake_utils.py -v ``` 2. Running a local install command to verify that the build environment compiles and registers: ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . ``` ## Test Result ### 1. CMake Utility Pytest Verification Output ```text $ .venv/bin/python -m pytest tests/test_cmake_utils.py -v ============================= test session starts ============================== platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/swidanagama/vllm/.venv/bin/python cachedir: .pytest_cache rootdir: /home/swidanagama/vllm configfile: pyproject.toml plugins: asyncio-1.4.0, anyio-4.14.2 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 3 items tests/test_cmake_utils.py::test_exact_family_arch_precedes_generic_family_fallback PASSED [ 33%] tests/test_cmake_utils.py::test_extract_archs_prefers_sass_target_over_corrupted_virtual_arch PASSED [ 66%] tests/test_cmake_utils.py::test_clear_cuda_gencode_flags PASSED [100%] ======================== 3 passed in 0.95s ======================== ``` ### 2. Local Install Result ```text Uninstalled 4 packages in 19ms Installed 164 packages in 939ms ... + vllm==0.26.1rc1.dev652+g066f84253.precompiled (from file:///home/swidanagama/vllm) ``` ## AI assistance disclosure This PR description and code changes were prepared with AI assistance. The submitting contributor has reviewed every changed line and run the relevant validation tests.", "url": "https://github.com/vllm-project/vllm/pull/51901", "createdAt": "2026-08-12T00:11:47Z", "updatedAt": "2026-08-13T05:40:55Z", "timestamp": "2026-08-13T05:40:55Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready", "ci/build", "nvidia" ], "author": "shanewidanagama", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c2bc28594eda5c985267", "signalId": "github:vllm-project/vllm:pull_request:52016", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X dense linear backends", "text": "## Purpose This PR adds opt-in [B12X](https://github.com/local-inference-lab/b12x) dense linear backends for NVIDIA SM120 and SM121 GPUs using vLLM's existing linear backend interfaces. It also provides the shared optional-package, packed-storage reuse, and warmup integration used by the B12X backend family. It does not introduce a new backend abstraction. B12X is an optional dependency installed with `vllm[b12x]` and pinned to `b12x==1.2.4`. It is a pure-Python CuTe DSL library and requires no additional vLLM build step. Supported linear paths include: - Per-tensor FP8. - 128x128 block-scaled FP8. - MXFP8. - NVFP4 and MXFP4. - Explicit `--linear-backend b12x` selection. - Targeted `VLLM_USE_B12X_FP8_GEMM` and `VLLM_USE_B12X_FP4_GEMM` overrides. - Existing fallback behavior for unsupported dense W4A16 layers. The documentation covers installation, selection, supported formats, and fallback behavior. This is the linear and shared-integration component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 target FlashInfer-embedded B12X paths or narrower integrations and do not provide this standalone optional linear backend. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused dense-kernel and warmup coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/model_executor/kernels/test_b12x_mxfp4_linear.py \\ tests/model_executor/kernels/test_b12x_nvfp4_linear.py \\ tests/model_executor/kernels/test_b12x_mxfp8_linear.py \\ tests/model_executor/test_b12x_warmup.py \\ tests/kernels/quantization/test_block_fp8.py \\ -k b12x -q ``` Run pre-commit over every changed file: ```bash mapfile -t changed_files < <(git diff --name-only upstream/main...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check upstream/main...HEAD ``` Benchmark `Qwen/Qwen3.6-27B-FP8` on an RTX PRO 6000 Blackwell Max-Q GPU, holding the attention backend constant and comparing B12X with CUTLASS. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 44 passed, 473 deselected ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | Dense block FP8 | Qwen3.6-27B-FP8 | 1 | CUTLASS | 54.3975 | 56.7518 | +4.33% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52016", "createdAt": "2026-08-12T16:04:49Z", "updatedAt": "2026-08-13T05:36:33Z", "timestamp": "2026-08-13T05:36:33Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:525d4b041f381e33a737", "signalId": "github:vllm-project/vllm:pull_request:52101", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52101", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Distributed][MoonEP] BF16 PoC integration of MoonEP balanced EP backend", "text": "## Purpose First item of the MoonEP integration roadmap RFC #52095: a correctness-first BF16 proof of concept of MoonEP (https://github.com/MoonshotAI/MoonEP) as a vLLM all2all backend. Included: - `\"moonep\"` `All2AllBackend` registration and `has_moonep()` probe - `MoonEPAll2AllManager` owning the cached `moonep.Buffer` - `MoonEPPrepareAndFinalize` conforming to the modular prepare/finalize interface: pad to static capacity → `dispatch` → `prefetch_weight` for the planned redundant experts (the `MoonEPCommPlan` is stashed between prepare and finalize, the same pattern DeepEP-HT uses for its handle) → `combine` in finalize - Replicated `[E+B, …]` BF16 weight-layout helper and a reference segment-loop expert runner over `cu_seqlens` - `tools/moonep/validate_moonep_bf16_poc.py`: distributed validation of the full dispatch/prefetch/compute/combine path against a dense per-token reference MoE Known PoC limitations, tracked as follow-up items in #52095: BF16/unquantized only; eager only; reference segment loop rather than a grouped GEMM; expert weights replicated per rank rather than MoonEP's sharded symmetric-memory ownership; no FusedMoE layer/engine wiring yet (engine use of the backend is not supported by this PR). ## Test Plan On an NVLink symmetric-memory capable node with the `moonep` package installed: ```bash torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py --skew 8.0 torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py \\ --num-tokens 512 --capacity 512 --hidden-size 2048 \\ --intermediate-size 768 --topk 8 --num-experts 64 --skew 4.0 ``` ## Test Result All three configurations pass on 4× GB300 (EP=4, NVLink), every rank `global_ok=True`: | Config | max rel_err | |---|---| | Default (S=128, H=512, top-4 of 32 experts) | 7.7e-3 | | Heavy router skew (`--skew 8`, exercises the redundant-expert planner + prefetch slots) | 5.7e-3 | | Larger shapes (S=512, H=2048, top-8 of 64 experts, skew 4) | 1.2e-2 | All within BF16 tolerance (2e-2) against the dense per-token reference. No model evaluation is included: this backend is opt-in, is not wired into the engine by this PR, and does not affect any existing serving path or model output. ## Notes - Not a duplicate: no open PR or issue in this repo implements a MoonEP backend (searched \"MoonEP\" across PRs/issues; no implementation PRs exist). - AI assistance was used in developing this change; I have reviewed and tested every changed line.", "url": "https://github.com/vllm-project/vllm/pull/52101", "createdAt": "2026-08-13T04:57:26Z", "updatedAt": "2026-08-13T05:25:59Z", "timestamp": "2026-08-13T05:25:59Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "nvidia" ], "author": "kaijunli-infr", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3a5e46ccfc4484d54ef9", "signalId": "github:vllm-project/vllm:pull_request:51959", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51959", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] DeepGEMM pin has no SM120 kernels: family-12 Blackwell cannot run hyperconnections", "text": "## The problem DeepGEMM dispatches the TF32 hyperconnection pre-norm GEMM on the device arch major, in `csrc/apis/hyperconnection.hpp`. At the revision this repository pins, `vllm-project/DeepGEMM` `e21c821`, that dispatch handles `arch_major` 9 and 10 only: ```cpp const auto arch_major = device_runtime->get_arch_major(); if (arch_major == 9) { sm90_tf32_hc_prenorm_gemm(...); } else if (arch_major == 10) { sm100_tf32_hc_prenorm_gemm(...); } else { DG_HOST_UNREACHABLE(\"Unsupported architecture\"); } ``` That revision ships no `sm120_*` implementation at all. `csrc/jit_kernels/impls/` contains `sm90_*` and `sm100_*` only. `tools/install_deepgemm.sh` currently comments that the pin targets nv-dev \"due to sm120 support\", but the pinned tree does not carry it. So every SM120/SM121 device aborts as soon as a model exercises hyperconnections. DeepSeek-V4 Flash is one: it is configured with `hc_mult=4`, so the path is hit on the first forward pass. ## The change Points both pins at `deepseek-ai/DeepGEMM` `a6b593d`, which carries `sm120_bf16_gemm`, `sm120_bmk_bnk_mn`, `sm120_fp8_fp4_gemm_1d1d` and `sm120_tf32_hc_prenorm_gemm`, and whose dispatch has the `arch_major == 12` branch. `cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh` both document that they must stay in sync, so both are updated. To be explicit: only the cmake path is exercised by my build; the `install_deepgemm.sh` edit is the same two values kept in sync, but I have not run that script. ## Testing With this change, DeepSeek-V4-Flash-0731 builds and serves on 2× NVIDIA GB10 (SM121, aarch64, CUDA 13.2, torch 2.13.0+cu132) with `--tensor-parallel-size 2` across two nodes. That is the configuration currently running here, and the fetched tree does contain the four `sm120_*` kernels. The failure itself does not really need a reproduction, since it is visible in the pinned tree: `csrc/apis/hyperconnection.hpp` at `e21c821` has no `arch_major == 12` branch, and `csrc/jit_kernels/impls/sm120_tf32_hc_prenorm_gemm.hpp` does not exist at that revision. For the record, we did hit `DG_HOST_UNREACHABLE(\"Unsupported architecture\")` on the old pin before switching, but I no longer have that log and am not claiming a fresh reproduction. Note it is a runtime abort rather than a build error: DeepGEMM JIT-compiles its kernels, so the build succeeds and the process dies on the first forward pass reaching the dispatch. Recipe, scripts and raw measurements from that setup: https://github.com/Mirrdhyn/dsv4-flash-dgx-spark ## Caveat, and a question for maintainers This is the minimal change that demonstrably fixes SM12x, but it may not be the change you want. The two hyperconnection kernels shared by both trees are byte-identical (`sm90_tf32_hc_prenorm_gemm.hpp` `c17d1b5`, `sm100_tf32_hc_prenorm_gemm.hpp` `0071e2c`), so the hyperconnection path itself is unaffected on SM90 and SM100. Other kernels have diverged, however: `sm90_bf16_gemm.hpp` differs between the two revisions, which suggests the fork carries vLLM-specific work that repointing would drop. I have no SM90 or SM100 hardware and have not tested those architectures. If the fork does carry changes worth keeping, the better fix is to port the `sm120_*` kernels into `vllm-project/DeepGEMM`, or to rebase the fork onto a newer upstream, and leave the pin where it is. I am happy to redo the patch that way. Either path is fine by me; the point of this PR is that SM12x is currently broken and the fix is small.", "url": "https://github.com/vllm-project/vllm/pull/51959", "createdAt": "2026-08-12T08:40:00Z", "updatedAt": "2026-08-13T05:23:46Z", "timestamp": "2026-08-13T05:23:46Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "needs-rebase", "ci/build" ], "author": "Mirrdhyn", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:8be207e606149428f69a", "signalId": "github:vllm-project/vllm:pull_request:51382", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51382", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Build] Pin minimal DeepGEMM SM120 and SITU revision", "text": "## Purpose Addresses #47436 with a minimal DeepGEMM dependency revision that combines: - `a6b593d2826719dcf4892609af7b84ee23aaf32a`, the revision previously used by vLLM for SM120 support. - The SITU change from `f5a76426fa084087169693fd0cd815223576d6e9`, cherry-picked directly on top as `586032eba787dd99e3d1c069f8514ab01ca64906`. - A one-line follow-up importing `Union`, which is referenced by the cherry-picked MegaMoE annotations. The resulting DeepGEMM branch is [`nv_dev+situ+0810`](https://github.com/vllm-project/DeepGEMM/tree/nv_dev%2Bsitu%2B0810), pinned at [`9e8903799beb0b65d88e5ca08940dd5cd712c7d2`](https://github.com/vllm-project/DeepGEMM/commit/9e8903799beb0b65d88e5ca08940dd5cd712c7d2). Both vLLM DeepGEMM pins are updated together. No other vLLM files are changed. ### Duplicate-work check #50796 targets the same high-level SM120 + SITU outcome, but pins the later `nv_dev+situ` merge commit `5f33a180`. This PR instead keeps the smaller `a6b593d` base and the focused SITU cherry-pick. Other open DeepGEMM pin PRs found by the required duplicate search do not use this minimal construction. ## Test plan and results Passed locally in vLLM: ```bash bash -n tools/install_deepgemm.sh .venv/bin/pre-commit run --files cmake/external_projects/deepgemm.cmake tools/install_deepgemm.sh git diff --check ``` The two vLLM pins were checked for exact equality at `9e8903799beb0b65d88e5ca08940dd5cd712c7d2`. Passed on the DeepGEMM branch: ```bash uvx ruff check --select F821 deep_gemm/mega/__init__.py uv run --no-project python -c 'import ast, pathlib; p=pathlib.Path(\"deep_gemm/mega/__init__.py\"); ast.parse(p.read_text(), filename=str(p))' git diff --check ``` Not run locally: CUDA compilation, SM120/B200 hardware validation, or model evaluation. A fresh vLLM CI run is required for the new pin. ## AI assistance AI assistance was used for CI-log analysis, the one-line dependency fix, static validation, and drafting this update. The submitting human must review every changed line and confirm the CUDA/model results before merge.", "url": "https://github.com/vllm-project/vllm/pull/51382", "createdAt": "2026-08-07T09:18:25Z", "updatedAt": "2026-08-13T05:23:31Z", "timestamp": "2026-08-13T05:23:31Z", "metrics": { "reactions": 4, "comments": 11 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "khluu", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:580a1b96734a34507de8", "signalId": "github:vllm-project/vllm:pull_request:50513", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50513", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] update UMD to 26.27", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50513", "createdAt": "2026-07-31T03:22:17Z", "updatedAt": "2026-08-13T05:15:26Z", "timestamp": "2026-08-13T05:15:26Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "intel-gpu", "ci/build" ], "author": "yma11", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:5a80ae813d83623154b4", "signalId": "github:vllm-project/vllm:pull_request:52060", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52060", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Fold scalar KV scales in MiniMax-M3 sparse attention", "text": "> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (incorrect expert count caused silent accuracy loss) > > Base image: `vllm/vllm-openai-rocm:nightly-3ee2df30337a301164c46ae444b76ee67e71c106` ## Purpose MiniMax-M3 Triton sparse attention currently applies scalar FP8 KV scales to every K and V element loaded inside the attention loop. These scales are loop invariant, so the repeated conversions and multiplications add unnecessary work. ## Suggested Fix For scalar scales (`KV_SCALE_MODE == 1`), fold the K scale into the attention score scale and apply the V scale once to the normalized FP32 accumulator. The change covers AMD prefill, common prefill, and common decode. No-scale and per-token/head scale modes remain unchanged through the existing compile-time dispatch. ## Test Plan ```bash pytest -q tests/kernels/test_minimax_m3_sparse_attn_fp8_scale.py ``` Compares scalar and per-token/head FP8 KV-cache outputs against a dequantized BF16 reference for both prefill and decode.
Serve command ```bash export HIP_VISIBLE_DEVICES=0,1,2,3 export VLLM_ROCM_USE_AITER=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER_MOE=0 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 vllm serve EmbeddedLLM/MiniMax-M3-FP8-dynamic \\ --served-model-name minimax-m3 \\ --tensor-parallel-size 4 \\ --block-size 128 \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.92 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 32768 \\ --max-num-seqs 32 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --tool-call-parser minimax_m3 \\ --enable-auto-tool-choice \\ --reasoning-parser minimax_m3 \\ --kv-cache-dtype fp8 \\ --language-model-only \\ --port 8000 \\ --attention-backend ROCM_AITER_UNIFIED_ATTN ```
## Test Result Kernel accuracy: 4/4 passed (scalar + per-token/head, prefill + decode). End-to-end serving on 4x MI325X (gfx942), TP4, `EmbeddedLLM/MiniMax-M3-FP8-dynamic`, FP8 KV cache: | ISL/OSL | Concurrency | Metric | Baseline | This PR | Change | |---------|-------------|--------|----------|---------|--------| | 8k/1k | 4 | Output tok/s | 309.18 | 315.52 | **+2.0%** | | 8k/1k | 4 | TTFT median (ms) | 1683 | 1541 | **-8.5%** | | 8k/1k | 8 | Output tok/s | 470.73 | 485.37 | **+3.1%** | | 8k/1k | 8 | TTFT median (ms) | 2123 | 2041 | **-3.9%** | | 128k/1k | 4 | Output tok/s | 93.48 | 99.85 | **+6.8%** | | 128k/1k | 4 | TTFT median (ms) | 19229 | 17552 | **-8.7%** | | 128k/1k | 8 | Output tok/s | 105.05 | 113.14 | **+7.7%** | | 128k/1k | 8 | TTFT median (ms) | 34297 | 31230 | **-8.9%** | | Task | Filter | n-shot | Baseline | This PR | |------|--------|--------|----------|---------| | GSM8K (full, 1319) | flexible-extract | 8 | 93.18% ± 0.69 | 92.95% ± 0.71 | | GSM8K (full, 1319) | strict-match | 8 | 93.18% ± 0.69 | 92.87% ± 0.71 |", "url": "https://github.com/vllm-project/vllm/pull/52060", "createdAt": "2026-08-12T22:12:24Z", "updatedAt": "2026-08-13T05:12:34Z", "timestamp": "2026-08-13T05:12:34Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "akii96", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dc90ca7be0601091e7e3", "signalId": "github:vllm-project/vllm:pull_request:52102", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52102", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Multimodal] Guard read_frames against empty frame indices", "text": "## Summary `Qwen3VLVideoBackend.compute_frames_index_to_sample()` returns an empty list for a 0-frame video (it lacks the base class's `max(1, ...)` floor), which then crashed `VideoBackend.read_frames()`: - **frame-recovery path** (`_read_frames_with_recovery`): `IndexError: list index out of range` at `next_target_map[frame_indices[-1]] = total_frames` — the same line's neighbor `max_frame_idx = frame_indices[-1] if frame_indices else 0` is guarded a few lines above, but this one is not. - **no-recovery path**: `ValueError: max() iterable argument is empty`. `read_frames` is the single entry point shared by **every** video backend (OpenCV / PyAV / TorchCodec / DeepStream), so the guard protects all of them. This adds an early return for empty frame lists that matches the empty-shaped frames (`np.empty((0, h, w, 3))`) the recovery path already produces for empty input — the same state downstream code already receives today for 0-frame videos through the base-class path. ## Test ```bash .venv/bin/python -m pytest \\ \"tests/multimodal/test_video.py::test_read_frames_handles_empty_frame_indices\" \\ \"tests/multimodal/test_video.py::test_qwen3vl_zero_frame_video_does_not_crash\" -v # 3 passed ``` - `ruff check` / `ruff format` on both changed files: clean. - Full `tests/multimodal/test_video.py` run: 40 passed, 5 skipped; the 25 failures are all `ModuleNotFoundError: No module named 'av'/'torchcodec'` (video decode deps absent in this CPU environment), unrelated to this change. **Model evaluation:** not applicable — this is a crash-only fix for degenerate (0-frame) input; it does not change output, accuracy, or serving behavior for any valid input. ## Duplicate / overlap check No open PR references an empty-frame crash in `read_frames` or the Qwen3VL `compute_frames_index_to_sample` path. - #50990 *\"Validate dynamic video sampling metadata\"* is related in theme but does **not** overlap: it adds validation to `DynamicVideoBackend` only, and does not touch `read_frames`, the Qwen3VL backend, or any backend's `read_frames` boundary. - `bad_words` / YAML-config / JSON-schema empty-input fixes are unrelated code paths. ## AI assistance This change was developed with AI assistance (Claude Code): bug identification, fix, and tests were AI-drafted and reviewed by a human before submission.", "url": "https://github.com/vllm-project/vllm/pull/52102", "createdAt": "2026-08-13T04:59:11Z", "updatedAt": "2026-08-13T05:11:45Z", "timestamp": "2026-08-13T05:11:45Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "multi-modality" ], "author": "zuver-lab", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c1b9e43986d4ff5e5990", "signalId": "github:vllm-project/vllm:issue:51916", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51916", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][DSV4-Flash][DSpark] v0.27 weight loader regressed — KeyError routed_experts.w13_weight_scale (v0.26 gracefully skipped)", "text": "### Your current environment - vLLM: **0.27.1** (pip, `/data1/vllm027` venv) - Python 3.11 - torch 2.13.0+cu130, triton 3.7.1, flashinfer-python 0.6.16.post3, transformers 5.15.0, CUDA 13.0 - Hardware: 8× NVIDIA H20-3e (141 GB), TP=8 + expert parallel - (Full `collect_env.py` available on request.) ### How to reproduce Serve DeepSeek-V4-Flash-0731 with DSpark speculative decoding on v0.27.1 (the actual flags used): ``` vllm serve \\ --trust-remote-code \\ --tensor-parallel-size 8 --enable-expert-parallel \\ --gpu-memory-utilization 0.85 --max-model-len 524288 \\ --max-num-batched-tokens 16384 --max-num-seqs 64 \\ --kv-cache-dtype fp8 --block-size 256 --dtype auto \\ --enable-prefix-caching --enable-chunked-prefill \\ --kv-offloading-size 512 --kv-offloading-backend native \\ --speculative-config.method dspark \\ --speculative-config.model \\ --speculative-config.num_speculative_tokens 5 \\ --enable-auto-tool-choice --tool-call-parser deepseek_v4 \\ --reasoning-parser deepseek_v4 --disable-custom-all-reduce ``` **Pre-req to reach this code path on Python ≤ 3.11**: flashinfer 0.6.16's `flashinfer/comm/fd_exchange.py:55` has a return annotation `tuple[..., array.array[int]]` that raises `TypeError: type 'array.array' is not subscriptable` at import time (`array.array` gains `__class_getitem__` only in 3.12). That crash happens *earlier* and masks this bug. To reproduce **this** bug, either run on Python 3.12, or add `from __future__ import annotations` to flashinfer's `fd_exchange.py`. ### Error During draft-model weight load at startup (all TP/EP ranks fail identically): ``` File \".../vllm/v1/worker/gpu/spec_decode/speculator.py\", line 156, in load_model self.model = self.load_draft_model(target_model, target_attn_layer_names) File \".../vllm/v1/worker/gpu/spec_decode/dspark/speculator.py\", line 81, in load_draft_model model = load_dspark_model(target_model, self.vllm_config) File \".../vllm/v1/worker/gpu/spec_decode/dspark/utils.py\", line 45, in load_dspark_model draft_model = get_model(...) ... File \".../vllm/models/deepseek_v4/nvidia/dspark.py\", line 449, in load_weights param = params_dict[name_mapped] ~~~~~~~~~~~^^^^^^^^^^^^^ KeyError: 'model.layers.0.ffn.experts.routed_experts.w13_weight_scale' ``` → `WorkerProc failed to start` → engine init failed. ### Root cause v0.27's DSpark draft weight loader (`dspark.py:449` `load_weights`) looks up the mapped name with a direct subscript `params_dict[name_mapped]`. The DSpark draft module does not register `model.layers.0.ffn.experts.routed_experts.w13_weight_scale` (a target-model routed-experts weight the draft does not carry), so the lookup raises. **This is a regression from v0.26**: v0.26's loader skipped unknown draft weights with a `\"Skipping unknown DStorm weight ...\"` log line (graceful). v0.27 dropped that guard → hard `KeyError`. ### Suggested fix Restore the skip guard at `dspark.py:449`: ```python if name_mapped not in params_dict: continue param = params_dict[name_mapped] ``` Safe because spec-decoding draft weights are validated by the target model; an unmapped draft weight should be skipped, not fatal. Restores v0.26 behavior. ### Additional context Two further issues block v0.27.1 on this exact stack (DSV4-Flash + DSpark + Hopper + Py3.11), tracked separately: 1. **flashinfer 0.6.16 `fd_exchange.py` PEP-585 annotation crash on Py ≤ 3.11** (the pre-req above) — flashinfer side; one-line `from __future__ import annotations` fix. 2. After both fixes, the **V1 engine hangs in warmup** (`shm_broadcast: No available shared memory broadcast block found in 60s` repeating 7+ min, all 8 GPUs at 0% util). Root cause not yet isolated (may be a knock-on of the patched loader); not reporting yet pending a clean repro. Happy to provide the full raw traceback / `collect_env.py` / test a fix on this 8×H20-3e stack.", "url": "https://github.com/vllm-project/vllm/issues/51916", "createdAt": "2026-08-12T03:07:04Z", "updatedAt": "2026-08-13T05:07:44Z", "timestamp": "2026-08-13T05:07:44Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "xiongyidxx", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8feb88be5d736a0886b4", "signalId": "github:vllm-project/vllm:pull_request:51664", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51664", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Helm] Fix chart resource references", "text": "Assisted-by: Codex ## Purpose Fix inconsistent Helm chart resource references when custom labels and autoscaling are enabled. Before this change: - The Service selector used configured labels, while the Deployment selector and Pod labels were hard-coded to `test/test`. - The HPA targeted a non-existent Deployment named `vllm`. After this change, the Deployment selector, Pod labels, Service selector, and HPA target consistently reference the release-specific resources and configured labels. This does not duplicate an existing PR: [fill in after running the duplicate checks; e.g. “No overlapping open PR was found.”] ## Test Plan ```bash helm lint . helm unittest . helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/service.yaml helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/deployment.yaml helm template vllm-test . \\ --namespace vllm-test \\ -f /tmp/compare-values.yaml \\ --show-only templates/hpa.yaml ``` ## Test Result helm lint . 1 chart(s) linted, 0 chart(s) failed helm unittest . Charts: 1 passed, 1 total Test Suites: 5 passed, 5 total Tests: 10 passed, 10 total Before rendering with custom labels showed: - Service selector: production/qwen-serving - Deployment selector and Pod labels: test/test - HPA target: vllm After rendering with the same values showed: - Service selector, Deployment selector, and Pod labels: production/qwen-serving - HPA target: vllm-test-deployment-vllm ### Rendered manifest comparison #### Before **Service selector — already uses configured labels** \"스크린샷 **Deployment selector and Pod labels — incorrectly hard-coded to test/test** \"스크린샷 **HPA target — incorrectly points to vllm** \"스크린샷 #### After **Service selector — configured labels** \"스크린샷 **Deployment selector and Pod labels — now match the Service** \"스크린샷 **HPA target — now points to the release-specific Deployment** \"스크린샷 No model evaluation was run because this Helm-template-only change does not affect model output, accuracy, or serving behavior. - [x] The purpose of the PR is described. - [x] Test commands are provided. - [x] Test results and before/after comparison are provided. - [ ] Documentation update is not needed. This does not duplicate an existing PR: no overlapping open PR was found.", "url": "https://github.com/vllm-project/vllm/pull/51664", "createdAt": "2026-08-10T11:35:20Z", "updatedAt": "2026-08-13T05:03:03Z", "timestamp": "2026-08-13T05:03:03Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "documentation", "performance", "rocm", "ci/build", "multi-modality", "tool-calling", "qwen", "cpu", "kv-connector", "nvidia", "quantization" ], "author": "iwannagotobed", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b5e59ffe393f0f298ef7", "signalId": "github:vllm-project/vllm:pull_request:52001", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52001", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kimi-K3] Use MTP draft model config", "text": "## Bug Kimi-K3 constructs its MTP model with the target model's `hf_text_config` instead of the speculative draft model's config. The target config can legitimately have `num_nextn_predict_layers = 0`, while the MTP draft config has the actual draft layers (5 in our setup). Kimi therefore initializes `self.num_mtp_layers` to zero. During the profile run, MTP step selection executes: ```python spec_step_idx % self.num_mtp_layers ``` and startup fails with: ```text ZeroDivisionError: integer modulo by zero ``` ## Fix Read `hf_text_config` from `speculative_config.draft_model_config` in the AMD and NVIDIA Kimi-K3 MTP constructors. This PR intentionally changes only: - `vllm/models/kimi_k3/amd/mtp.py` - `vllm/models/kimi_k3/nvidia/mtp.py` - one focused regression test It does not change URLs, test asset paths, the generic EAGLE loader, or local-argmax behavior. ## Reproduction and A/B test The regression test creates: - target config: `num_nextn_predict_layers = 0` - draft config: `num_nextn_predict_layers = 5` Before the fix: - Kimi selects the target config - `0 % 0` raises `ZeroDivisionError` After the fix: - Kimi selects the draft config - `0 % 5 == 0` Run: ```bash pytest -q tests/models/kimi_k3/test_mtp.py ``` Result: `2 passed` (AMD and NVIDIA). Also checked Ruff formatting/lint and `git diff --check`.", "url": "https://github.com/vllm-project/vllm/pull/52001", "createdAt": "2026-08-12T14:15:50Z", "updatedAt": "2026-08-13T05:02:40Z", "timestamp": "2026-08-13T05:02:40Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "rocm", "speculative-decoding", "ci/build", "multi-modality", "deepseek", "cpu", "kv-connector", "mrv2", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:72ab5251063365d71876", "signalId": "github:vllm-project/vllm:pull_request:50599", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50599", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Fix dead class references in MoE design docs", "text": "## Purpose Six symbol references in the two MoE design docs no longer resolve against main. `docs/design/moe_kernel_features.md`: - The all2all backend table lists a `naive` backend whose Subclass column is an autoref to `vllm.model_executor.layers.fused_moe.layer.FusedMoE`. That class no longer exists, so the cross-reference cannot resolve. `naive` is also gone as a backend: `ParallelConfig.__post_init__` logs \"The '%s' all2all backend has been removed. Falling back to 'allgather_reducescatter'.\" for both `naive` and `pplx` (see #33728, #36294), and `allgather_reducescatter` is not in the table at all despite being the fallback. The row is renamed to `allgather_reducescatter` and pointed at `MoEPrepareAndFinalizeNaiveDPEPModular`, which is what that path constructs (`all2all_utils.py`, `use_ag_rs_all2all_kernels` branch). - The intro prose still calls the layer `FusedMoE`; it is `RoutedExperts`. - Two cross-reference labels are misspelled relative to their targets, which are already correct: `CutlasBatchedExpertsFp8` becomes `CutlassBatchedExpertsFp8`, and `TrtLlmNvfp4ExpertsModular` becomes `TrtLlmNvFp4ExpertsModular`. `docs/design/fused_moe_modular_kernel.md`: - `TopKWeightAndReduceNoOp` becomes `TopKWeightAndReduceNoOP`, since the class name ends in a capitalised `OP`. - `FusedMoEPremuteExpertsUnpermute` has never existed under that name; the surrounding text already uses `FusedMoEExpertsModular`. Only the cells I could verify against the code were changed. The quantization-type columns on the renamed row are left exactly as they were. Two further staleness items in these files are deliberately not touched, because fixing them correctly needs a judgment call from someone who owns this code: 1. The \"Modular Kernel families\" table has a `flashinfer` row naming `FlashInferCutlassMoEPrepareAndFinalize`. That class is gone and `flashinfer` is not in the `All2AllBackend` literal; the real classes are `FlashInferNVLinkOneSidedPrepareAndFinalize` and `FlashInferNVLinkTwoSidedPrepareAndFinalize`. Splitting the row needs a decision on which side pairs with `FlashInferExperts`, given one-sided also dispatches bf16 and mxfp8. 2. In `fused_moe_modular_kernel.md`, the `select_gemm_impl` and `maybe_make_prepare_finalize` sections list `Fp8MoEMethod`, `CompressedTensorsW8A8Fp8MoEMethod`, `CompressedTensorsW8A8Fp8MoECutlassMethod` (which does not exist) and `ModelOptNvFp4FusedMoE`. None of those define `select_gemm_impl` today, and `ModelOptNvFp4FusedMoE.maybe_make_prepare_finalize` now raises `ValueError(\"uses the new modular kernel initialization logic\")`, the opposite of what the doc describes. Happy to fold either in if a reviewer tells me the intended answer. Docs only, no behavior change. ## Test Plan - `pre-commit run --files docs/design/moe_kernel_features.md docs/design/fused_moe_modular_kernel.md` - Resolve all mkdocs `][vllm...]` autoref targets across `docs/` against an AST index of modules and class members, before and after. - Confirm each replacement name exists, and each removed name is absent, by grepping for the class definitions under `vllm/` and `tests/`. - Read the `naive` and `pplx` removal in `vllm/config/parallel.py` and the `use_ag_rs_all2all_kernels` branch in `vllm/model_executor/layers/fused_moe/all2all_utils.py`. ## Test Result - `pre-commit run`: `typos`, `markdownlint-cli2`, `Check for spaces in all filenames`, `Update Dockerfile dependency graph` and the config validation hook all Passed. - Autoref resolution: `vllm.model_executor.layers.fused_moe.layer.FusedMoE` was the only genuinely unresolvable target in these two files, and it resolves after the change. - All six replacement names found at their definition sites: `MoEPrepareAndFinalizeNaiveDPEPModular` in `prepare_finalize/naive_dp_ep.py`, `CutlassBatchedExpertsFp8` in `experts/cutlass_moe.py`, `TrtLlmNvFp4ExpertsModular` in `experts/trtllm_nvfp4_moe.py`, `TopKWeightAndReduceNoOP` in `topk_weight_and_reduce.py`, `FusedMoEExpertsModular` in `modular_kernel.py`, and `RoutedExperts` in `routed_experts.py`. - All removed names: no matches anywhere under `vllm/` or `tests/`, under either spelling. I do not have GPU access, so this was verified statically against the source rather than by running MoE kernels.", "url": "https://github.com/vllm-project/vllm/pull/50599", "createdAt": "2026-07-31T16:58:58Z", "updatedAt": "2026-08-13T05:00:56Z", "timestamp": "2026-08-13T05:00:56Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready" ], "author": "UgaTheDev", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0c9e5d98190ce0e0a936", "signalId": "github:vllm-project/vllm:pull_request:52100", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52100", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Preserve TP sharding with explicit EP", "text": "## Summary - preserve physical TP size/rank inside experts when explicit EP is configured - use the fixed-lane EP group for expert mapping and pre-I/O weight filtering - keep AG/RS dispatch/combine over EP and the existing late TP all-reduce - add coverage for TP4/EP2 config, Kimi expert ownership, loader filtering, final TP reduction, and the ROCm latent tail Stacked on #52099. The topology commit disappears once #52099 merges. Kimi-K3 has 896 routed experts. With EP2, each process owns 448 experts with TP4-sharded weights; four ranks in a TP group own distinct shards of the same 448-expert set. No Kimi production-code special case is required: the existing ROCm latent runner consumes the preserved physical TP size. ## 8-GPU validation - topology: TP4/DP2/EP2, AG/RS, target-only, 100K input + 1K output, concurrency 1 - startup log: `Local/global number of experts: 448/896`; weight filter loads `448/896` - execution log: `MoEPrepareAndFinalizeNaiveDPEPModular` + `AiterExperts` - numerical gate: GSM8K **100/100 strict**, **100/100 flexible**, zero errors/malformed - TPOT over 3 runs: **27.296 ms mean** (27.107–27.447 ms); correctness passes, but this topology is not the C1 performance winner - non-MTP performance gate is met by TP8/DP1: **18.059 ms mean TPOT** over 3 runs (18.056–18.062 ms) ## Test plan - [x] topology + generic execution suite — 13 passed - [x] Kimi ROCm latent runner + hybrid MoE suite — 19 passed on 8 GPUs - [x] Ruff check and format check on changed files - [x] `git diff --check` - [x] 8-GPU TP4/DP2/EP2 Kimi-K3 serving and GSM8K validation", "url": "https://github.com/vllm-project/vllm/pull/52100", "createdAt": "2026-08-13T04:54:49Z", "updatedAt": "2026-08-13T05:00:15Z", "timestamp": "2026-08-13T05:00:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2e257dc43557d28e5c74", "signalId": "github:vllm-project/vllm:pull_request:52099", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52099", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][MoE] Add explicit expert-parallel topology", "text": "## Summary - add an experimental `--expert-parallel-size` independent of tensor parallel size - construct fixed-TP-lane EP groups, e.g. TP4/DP2 gives `[0,4]`, `[1,5]`, `[2,6]`, `[3,7]` - preserve existing flattened EP behavior when the option is unset - fail closed for PP/PCP, EPLB, elastic EP, non-AG/RS backends, and EP sizes other than DP This is the topology foundation for experts that remain tensor-sharded while ownership is distributed across DP ranks. MoE execution support is stacked in #52100. ## Test plan - [x] `pytest -q tests/config/test_explicit_expert_parallel.py` — 9 passed - [x] Ruff check and format check on changed files - [x] `git diff --check`", "url": "https://github.com/vllm-project/vllm/pull/52099", "createdAt": "2026-08-13T04:52:10Z", "updatedAt": "2026-08-13T05:00:13Z", "timestamp": "2026-08-13T05:00:13Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c2fd38b5c3ed7c3e31ac", "signalId": "github:vllm-project/vllm:issue:52095", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52095", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "RFC: Kimi K3 MoonEP integration roadmap", "text": "## Motivation Track the **MoonEP integration** item under Large-scale Serving in #50001 as a small set of independently reviewable PRs. MoonEP (https://github.com/MoonshotAI/MoonEP) is the balanced EP communication library used by Kimi K3: dynamic redundant experts keep every EP rank at exactly `S × K` tokens regardless of router skew, with static shapes and zero-copy dispatch/combine. ## Proposed change - [ ] **BF16 PoC: backend registration + dispatch/combine path + validation** — #52101 `\"moonep\"` `All2AllBackend`, `MoonEPAll2AllManager`, `MoonEPPrepareAndFinalize` (pad → dispatch → `prefetch_weight` → combine, plan stashed between prepare/finalize like DeepEP-HT's handle), replicated `[E+B, …]` BF16 weight-layout helper, reference expert runner over `cu_seqlens`, torchrun validation script. Validated on 4× GB300 (EP=4, NVLink): passes at default config, heavy router skew, and larger shapes (S=512, H=2048, top-8 of 64 experts); rel_err within BF16 tolerance. - [ ] **Grouped-GEMM expert implementation over `cu_seqlens`** — PR TBD Modular-kernel experts class consuming MoonEP's expert-grouped `[NvS, H]` activation format. Open question for maintainers: `torch._grouped_mm` vs Triton vs DeepGEMM contiguous-layout kernels. - [ ] **FusedMoE layer + engine wiring (BF16, eager)** — PR TBD Attach the weight layout after weight loading; config validation and docs. Exit criterion: a small real MoE (e.g. Qwen3-30B-A3B) serves end-to-end with `--all2all-backend moonep` with sane evals. - [ ] **Tests + benchmarks** — PR TBD Multi-GPU correctness test under `tests/kernels/moe/`; dispatch/combine benchmark vs `deepep_*` backends under `benchmarks/kernels/`, sweeping router imbalance. - [ ] **Sharded symmetric-memory expert weight ownership** — PR TBD Replace per-rank weight replication with MoonEP's production layout: one contiguous VMM tensor per projection where rows `[0, E)` alias each home rank's parameter memory. Memory-viability requirement for Kimi-K3 scale. Design question: allocate expert weights inside the MoonEP buffer before weight loading vs. post-load remap. - [ ] **CUDA graph support** — PR TBD MoonEP's fully static `NvS` shapes should make capture tractable. - [ ] **Quantized dispatch + weights (fp8 first)** — PR TBD Kimi-K3-scale serving will not be BF16; needs MoonEP-side support scoping as well. - [ ] **EPLB interaction** — PR TBD MoonEP's dynamic redundant experts overlap with the EPLB item in the same #50001 section; define compose-or-exclude semantics. - [ ] **Async / DBO support** — PR TBD `async_finish=True` + MoonEP's comm stream for prepare_async/finalize_async and dual-batch overlap. ## Scope This issue is a tracking RFC only. Each item should remain independently reviewable. The first four items establish a correctness-first BF16/eager path; the sharded weight-ownership and EPLB items carry the design decisions where maintainer input is requested before implementation.", "url": "https://github.com/vllm-project/vllm/issues/52095", "createdAt": "2026-08-13T04:22:08Z", "updatedAt": "2026-08-13T04:59:56Z", "timestamp": "2026-08-13T04:59:56Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "quantization", "kimi", "k3" ], "author": "kaijunli-infr", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:74669e896e02a4f4b5b2", "signalId": "github:vllm-project/vllm:pull_request:52078", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52078", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Avoid redundant mask compute in GDN metadata build", "text": "## Purpose The spec decode detection in GDNAttentionMetadataBuilder.build() computed tensors multiple times: 1. computed the `num_decode_draft_tokens_cpu >= 0` mask twice in line 192 and 200. Each >= 0 creates a fresh boolean tensor. 2. computed `~spec_sequence_masks_cpu` 4 times This PR avoid redundant compute: 1. Compute `num_decode_draft_tokens_cpu` only once 2. Only compute `num_decode_draft_tokens_cpu[spec_sequence_masks_cpu].sum().item()` iff `num_spec_decodes == 0` is false 3. Compute `~spec_sequence_masks_cpu` once into `non_spec_sequence_masks_cpu` 4. Behavior is preserved exactly ## Profiling Main: \"Screenshot PR: \"Screenshot Main: `aten::ge` launched twice. PR: `aten::ge` launched once. ## Benchmark ``` vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"z-lab/Qwen3.6-35B-A3B-DFlash\",\"method\":\"dflash\",\"num_speculative_tokens\":8}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 84.82 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.71 Output token throughput (tok/s): 212.22 Peak output token throughput (tok/s): 62.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 405.69 ---------------Time to First Token---------------- Mean TTFT (ms): 113.86 Median TTFT (ms): 131.82 P99 TTFT (ms): 142.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.35 Median TPOT (ms): 4.45 P99 TPOT (ms): 5.99 ---------------Inter-token Latency---------------- Mean ITL (ms): 16.15 Median ITL (ms): 16.15 P99 ITL (ms): 16.64 ---------------Speculative Decoding--------------- Acceptance rate (%): 34.34 Acceptance length: 3.75 Drafts: 4827 Draft tokens: 38616 Accepted tokens: 13260 Per-position acceptance (%): Position 0: 75.08 Position 1: 55.13 Position 2: 41.06 Position 3: 31.32 Position 4: 24.72 Position 5: 19.23 Position 6: 15.66 Position 7: 12.51 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 206.67 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.65 Output token throughput (tok/s): 1393.52 Peak output token throughput (tok/s): 585.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2494.53 ---------------Time to First Token---------------- Mean TTFT (ms): 209.36 Median TTFT (ms): 193.29 P99 TTFT (ms): 414.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.76 Median TPOT (ms): 10.80 P99 TPOT (ms): 17.09 ---------------Inter-token Latency---------------- Mean ITL (ms): 38.40 Median ITL (ms): 22.08 P99 ITL (ms): 156.83 ---------------Speculative Decoding--------------- Acceptance rate (%): 32.45 Acceptance length: 3.60 Drafts: 80447 Draft tokens: 643576 Accepted tokens: 208834 Per-position acceptance (%): Position 0: 74.47 Position 1: 52.20 Position 2: 37.66 Position 3: 28.59 Position 4: 22.55 Position 5: 17.95 Position 6: 14.43 Position 7: 11.74 ================================================== ``` PR: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 84.28 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.71 Output token throughput (tok/s): 213.58 Peak output token throughput (tok/s): 63.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 408.29 ---------------Time to First Token---------------- Mean TTFT (ms): 112.45 Median TTFT (ms): 130.62 P99 TTFT (ms): 138.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.32 Median TPOT (ms): 4.42 P99 TPOT (ms): 5.97 ---------------Inter-token Latency---------------- Mean ITL (ms): 16.06 Median ITL (ms): 16.05 P99 ITL (ms): 16.51 ---------------Speculative Decoding--------------- Acceptance rate (%): 34.34 Acceptance length: 3.75 Drafts: 4827 Draft tokens: 38616 Accepted tokens: 13260 Per-position acceptance (%): Position 0: 75.08 Position 1: 55.13 Position 2: 41.06 Position 3: 31.32 Position 4: 24.72 Position 5: 19.23 Position 6: 15.66 Position 7: 12.51 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 204.94 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.68 Output token throughput (tok/s): 1405.30 Peak output token throughput (tok/s): 619.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2515.61 ---------------Time to First Token---------------- Mean TTFT (ms): 206.68 Median TTFT (ms): 190.26 P99 TTFT (ms): 410.25 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.67 Median TPOT (ms): 10.75 P99 TPOT (ms): 16.93 ---------------Inter-token Latency---------------- Mean ITL (ms): 38.09 Median ITL (ms): 22.18 P99 ITL (ms): 153.25 ---------------Speculative Decoding--------------- Acceptance rate (%): 32.45 Acceptance length: 3.60 Drafts: 80447 Draft tokens: 643576 Accepted tokens: 208834 Per-position acceptance (%): Position 0: 74.47 Position 1: 52.20 Position 2: 37.66 Position 3: 28.59 Position 4: 22.55 Position 5: 17.95 Position 6: 14.43 Position 7: 11.74 ================================================== ``` ## Accuracy Testing ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.6-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3533|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3290|± |0.0129| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3563|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3321|± |0.0130| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52078", "createdAt": "2026-08-13T01:02:20Z", "updatedAt": "2026-08-13T04:57:50Z", "timestamp": "2026-08-13T04:57:50Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "xyang16", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a3fc3a3c739e2c548b08", "signalId": "github:vllm-project/vllm:issue:51788", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51788", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Suffix decoding on GPU (`suffix_gpu`) — GPU-resident suffix decoding compatible with async scheduling", "text": "## Summary Suffix decoding gets the best acceptance lengths among model-free drafters on repetitive and agentic workloads, but the in-tree implementation (`method=\"suffix\"`, #25784, from Arctic Inference) is CPU-side and rejected by the async-scheduling whitelist. This RFC proposes `suffix_gpu`: a GPU-resident suffix drafter that follows the same device-state drafter contract as `ngram_gpu` (#29184), so suffix decoding composes with async scheduling — no host sync on the draft path, drafts come from the previous step's device sampled ids. The in-tree diff is small (config + a proposer wrapper + runner wiring + unit tests, ~500 lines). The drafter core (suffix automaton over device tensors, cross-request global index with background rebuilds, fused Triton kernels, CUDA-graph capture) lives in a standalone package ([SuffixGPU](https://github.com/zip95297/SuffixGPU)), imported lazily — vLLM works without it installed, same pattern as the CPU suffix method's `arctic-inference` dependency. ## Motivation - #38069 asked for n-gram/suffix speculative decoding in the new runner under the condition that it stays compatible with async scheduling; it went stale, but the gap is still there. - Async scheduling is on the way to being the default. `method=\"suffix\"` forces `--no-async-scheduling` today (`config/vllm.py` whitelist: \"async scheduling is only supported with EAGLE/MTP/Draft Model/NGram GPU/DSpark\"), giving up CPU/GPU overlap exactly where spec decode helps most: high-concurrency TPOT. - The reason is structural, not incidental: the CPU suffix tree needs the committed host token ids each step, and under async scheduling the scheduler runs one step ahead of the worker, so those ids do not exist on the host at schedule time. A GPU-state drafter sidesteps this by drafting from the previous step's sampled ids that are already on device — the mechanism #29184 built for `ngram_gpu`. So: keep suffix decoding's draft quality, get async scheduling's overlap, by moving the drafter's state and compute onto the GPU. ## Design `suffix_gpu` plugs into the existing `ngram_gpu` machinery; no scheduler changes. - **Config** (`config/speculative.py`): `method=\"suffix_gpu\"`, reusing the `suffix_decoding_*` knobs (max_tree_depth, max_spec_factor, min_token_prob, max_cached_requests) plus GPU-specific ones (`suffix_gpu_global_capacity`, `suffix_gpu_delta_capacity`, `suffix_gpu_max_occurrences`, `suffix_gpu_use_cuda_graph`, `suffix_gpu_ingest_chunk`). A `use_gpu_state_drafter()` helper groups `ngram_gpu` + `suffix_gpu` for the shared device-state paths; the async whitelist admits `suffix_gpu`. - **Proposer** (`v1/spec_decode/suffix_proposer_gpu.py`): wraps the SuffixGPU drafter behind the `ngram_gpu` contract — same `update_token_ids_ngram` device bookkeeping (borrowed verbatim), same `propose(k, num_tokens_no_spec, token_ids, sampled, counts) -> (drafts [B,k], valid counts [B])`, same async D2H of per-request valid draft counts feeding the existing worker-side trim (`update_scheduler_for_invalid_drafts`). Two suffix-specific additions: - a per-request local matcher plus a cross-request global suffix index (the analogue of the CPU method's global tree / `max_cached_requests`), fed off the critical path: in-flight responses are ingested on a side stream, finished requests are final-flushed before their persistent batch rows are reused; - the whole draft chain is captured into CUDA graphs, bucketized by batch size, pre-captured at engine warmup so the first serving step pays no Triton JIT / capture latency (eager Triton fallback if capture fails or `suffix_gpu_use_cuda_graph=0`). - **Runner** (`v1/worker/gpu_model_runner.py`): the `ngram_gpu`-only gates (resident token buffers, optimistic-accept bookkeeping + rejection correction, valid-count D2H, scheduler_output shallow copy) become `use_gpu_state_drafter()` gates; suffix adds the two ingest hooks and graph pre-capture. No new scheduler interplay. Packaging follows the `suffix` / `arctic-inference` precedent: external optional dependency, lazy import, config validation fails with a clear install hint. (Open question below.) ## Correctness All numbers below: vLLM branch commit `f646af65c`, 1x NVIDIA L20 48GB, torch 2.13.0+cu130, drafter package suffix-gpu 0.1.1 (PyPI, tag `v0.1.1`; runs used the equivalent package code at repo commit `1ac9238`, installed editable — https://github.com/zip95297/SuffixGPU). - **Unit tests** (`tests/v1/spec_decode/test_suffix_gpu.py`): draft correctness on repetitive history, CUDA-graph vs eager draft agreement, graph pre-capture at warmup, JIT warmup with graphs disabled, global index cross-request drafting. Skips cleanly without CUDA or the package. Result: 6/6 passed (L20). - **E2E GSM8K** (in-tree pattern, same as ngram_gpu's async test): `test_suffix_gpu_with_async_scheduling` — Llama-3.1-8B-Instruct, suffix_gpu k=16 under async scheduling, 1319 questions 5-shot: accuracy 0.762 (threshold 0.70), passed. - **Acceptance e2e** (`tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance`): `test_suffix_gpu_acceptance` mirrors the CPU suffix acceptance test — 10 warm-up rounds over the same 100 prompts, suffix_gpu k=16 + async, pinned `suffix_gpu_ingest_chunk=1`, `num_backoff=8`, `max_occurrences=128`, prefix caching off, V1 runner. Warm-up series: round-0 rate 0.428 / AL 2.03 -> round-9 rate 0.857 / AL 5.70; the end rate clears the CPU test's 0.80 floor (0.79-0.86 across reruns — the final-round delta is ~1.4k drafted tokens, greedy near-ties move it a few points). Both tests pass together (`2 passed`); both — including the pre-existing CPU one — pin `enable_prefix_caching=False`, since warm-round prefix-cache hits change decode batch composition and depress the measured rate below the floor for the CPU method too. - **Greedy consistency**: spec on/off must produce token-identical greedy outputs. Run under `VLLM_BATCH_INVARIANT=1` + enforce_eager (a bitwise gate is meaningless otherwise: spec decode changes batch composition, and batch-dependent FP reduction order flips near-tie argmax even between two no-spec runs — measured on this box before enabling the invariant mode). Legs: async no-spec vs async suffix_gpu, plus suffix_gpu-eager / ngram_gpu / sync suffix CPU discriminators, 16 prompts x 256 tokens. Result: all legs token-identical (0/16 mismatches each; graph replay output == eager output). - **Drafter-level equivalence and latency vs the CPU suffix tree** (no engine, deterministic Spec-Bench replay): 221 tests pass including fuzz-equivalence against arctic on unambiguous corpora; replay tokens/step within -7%..0% of the CPU tree cold and parity to +3% warm; <= 532 MB reserved VRAM in the largest tested configuration. On drafting cost, the CPU tree wins small batches but its sequential per-request walk grows linearly with batch size, while graph-mode drafting is one flat batched launch: crossover at B64-128, and beyond it CUDA-graph mode is 2.1-2.7x faster per step even in the CPU's best case (B=256: 1.85 vs 4.98 ms; B=512: 5.15 vs 10.68 ms); under replay conditions that include per-step tree updates the GPU drafter is already faster from B32 (CPU 1.0-3.4 ms at B=58-80 vs a flat ~0.5 ms). This is what makes the drafter viable exactly in the high-concurrency regime async scheduling targets. Full tables and repro commands: https://github.com/zip95297/SuffixGPU/blob/main/RESULTS.md ## Performance Setup mirrors #25784 (Spec-Bench, k x concurrency grid, TPOT + drafted / accepted tokens) and #29184 (async-vs-sync same-method comparison): meta-llama/Llama-3.1-8B-Instruct, bf16, TP=1, 1x NVIDIA L20 48GB, `vllm bench serve --dataset-name spec_bench --spec-bench-output-len 256 --no-oversample`, concurrency 1/4/16/64/128/192/256, k in {5,16}, prefix caching off, all variants on model runner v1. Every variant gets `--max-num-batched-tokens 8192 --max-num-seqs 320` (spec decode schedules (k+1) padded token slots per running request before invalid ones are trimmed, so the serve-context defaults silently cap effective concurrency at the top grid points) and an identical 64-prompt warmup pass before measurement (cold CUDA graphs / Triton JIT / suffix index otherwise land in the first grid point; drafted/accepted numbers are the per-run deltas `vllm bench serve` records in each result json, so warmup traffic is excluded by construction). Reproduction script: `benchmarks/bench_pr_matrix.sh` (attached to the PR). Variants: async no-spec baseline; `suffix` + sync scheduling (what you must run today); `ngram_gpu` + async (in-tree GPU drafter baseline); `suffix_gpu` + async (this RFC). **k=5 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 58 (17.2) | 245 (15.4) | 754 (19.8) | 1297 (45.5) | 1343 (86.0) | 1375 (127.2) | 1373 (169.5) | | ngram_gpu_async | 46 (21.6) | 168 (23.3) | 515 (30.3) | 1126 (52.7) | 1341 (89.2) | 1389 (128.5) | 1396 (172.3) | | suffix_gpu_async | 57 (17.6) | 239 (15.6) | 744 (19.3) | 1316 (43.2) | 1410 (80.5) | 1461 (117.6) | 1467 (154.5) | **k=16 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 55 (18.2) | 239 (15.9) | 755 (19.5) | 1273 (46.3) | 1322 (86.0) | 1357 (125.8) | 1363 (167.2) | | ngram_gpu_async | 43 (23.0) | 161 (24.3) | 519 (29.9) | 1156 (52.2) | 1339 (89.6) | 1387 (131.0) | 1405 (170.6) | | suffix_gpu_async* | 52 (19.2) | 230 (16.2) | 736 (19.5) | 1276 (44.8) | 1349 (84.3) | 1402 (121.5) | 1417 (154.6) | \\* suffix_gpu k=16 ran at `gpu_memory_utilization=0.85` (other rows 0.9): at 0.9 the device-resident drafter state plus the (k+1)-slot verification batch OOMed the EngineCore at c256 on the 46GB L20 (see Known limits). **Drafted / accepted tokens per point (AL, acceptance rate)** | variant | k | c1 | c16 | c64 | c256 | |---|---|---|---|---|---| | suffix_sync | 5 | 79832/22918 (1.54, 28.7%) | 86405/33544 (1.94, 38.8%) | 86516/32697 (1.89, 37.8%) | 86240/31708 (1.85, 36.8%) | | suffix_gpu_async | 5 | 79742/22332 (1.52, 28.0%) | 87458/33453 (1.93, 38.3%) | 88350/35551 (2.04, 40.2%) | 88444/38485 (2.23, 43.5%) | | ngram_gpu_async | 5 | 33913/9467 (2.40, 27.9%) | 34863/9738 (2.40, 27.9%) | 34527/9332 (2.35, 27.0%) | 34358/9667 (2.41, 28.1%) | | suffix_sync | 16 | 83232/22867 (1.54, 27.5%) | 95070/34453 (1.98, 36.2%) | 93340/33505 (1.94, 35.9%) | 92211/32536 (1.89, 35.3%) | | suffix_gpu_async | 16 | 82552/22582 (1.53, 27.4%) | 95486/34166 (2.00, 35.8%) | 98323/36571 (2.13, 37.2%) | 105420/40513 (2.33, 38.4%) | | ngram_gpu_async | 16 | 35468/9978 (2.41, 28.1%) | 34844/9586 (2.38, 27.5%) | 33685/9643 (2.43, 28.6%) | 34316/9778 (2.42, 28.5%) | **Per-position acceptance rate — suffix CPU vs suffix_gpu, c=1 only** Measured at concurrency 1 (the c1 grid point, after the identical warmup pass), where scheduling noise is minimal, so this is the cleanest drafter-quality comparison. Each row also lists that run's drafts / draft tokens / accepted tokens / AL / overall acceptance rate. k=5, c=1: | variant | drafts | draft toks | accepted toks | AL | rate | p0 | p1 | p2 | p3 | p4 | |---|---|---|---|---|---|---|---|---|---|---| | suffix_sync | 42793 | 79832 | 22918 | 1.54 | 28.7% | 0.336 | 0.120 | 0.049 | 0.019 | 0.012 | | suffix_gpu_async | 42945 | 79742 | 22332 | 1.52 | 28.0% | 0.330 | 0.114 | 0.046 | 0.019 | 0.012 | k=16, c=1: | variant | drafts | draft toks | accepted toks | AL | rate | p0 | p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | p10 | p11 | p12 | p13 | p14 | p15 | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | suffix_sync | 42677 | 83232 | 22867 | 1.54 | 27.5% | 0.329 | 0.115 | 0.046 | 0.018 | 0.010 | 0.005 | 0.004 | 0.002 | 0.002 | 0.002 | 0.001 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 | | suffix_gpu_async | 42293 | 82552 | 22582 | 1.53 | 27.4% | 0.328 | 0.113 | 0.045 | 0.018 | 0.011 | 0.005 | 0.004 | 0.002 | 0.002 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.000 | The two per-position curves are essentially identical at both k values: the GPU drafter reproduces the CPU suffix tree's draft quality request-locally. Its aggregate advantage appears only as concurrency grows (AL 1.53 -> 2.33 from c1 to c256 at k=16, vs 1.98 -> 1.89 for the CPU tree), i.e. it comes from the cross-request global index compounding under concurrent traffic, not from a different per-draft profile. What the numbers show: 1. `suffix_gpu` + async beats `suffix` + sync from c64 up (+7% at k=5 saturation, 1467 vs 1373 tok/s; +4% at k=16, 1417 vs 1363; lower TPOT), and matches it at c1-c16 — resolving the suffix x async conflict is a real win once batches grow, exactly where async overlap matters. 2. `suffix_gpu` + async beats `ngram_gpu` + async at every grid point (up to +44%, k=5 c16: 744 vs 515 tok/s) — it is the stronger GPU drafter on this workload. 3. Against async no-spec, spec decoding wins through c64 (TPOT -31% at c4, -27% at c16, -8% at c64) but loses beyond c128 at saturation on Spec-Bench's mixed categories (1467 vs 1654 tok/s at k=5 c256) — true for all three drafters here (suffix_sync 1373, ngram_gpu 1396), see Known limits. ## Known limits - Saturation: beyond ~c128 on Spec-Bench's mixed categories, async no-spec out-throughputs every drafter here (k=5 c256: no-spec 1654 vs suffix_gpu 1467, suffix_sync 1373, ngram_gpu 1396 tok/s) — once the GPU is compute-bound, verification FLOPs for rejected drafts cost more than acceptance saves. On repetition-heavy traffic the crossover moves right; measured, not hidden. (Contrary to the usual GPU-drafter expectation, c=1 does not regress on this workload: 57-58 tok/s vs 47 no-spec, the warm suffix index pays for the drafter cost.) - Memory headroom at k=16: the device-resident drafter state plus the (k+1)-slot verification batch OOMed the EngineCore at c256 with the default `gpu_memory_utilization=0.9` on a 46GB L20; the k=16 row was measured at 0.85. Large-k + high-concurrency deployments need to budget for the drafter's device state. - Spec-decode acceptance metrics for GPU-state drafters counted scheduler-padded slots in the denominator (affects `ngram_gpu` equally). This PR includes the worker-side fix — per-request trim counts ride on `ModelRunnerOutput` and are subtracted in `make_spec_decoding_stats` — with method-neutral naming; mechanism-wise it matches the stale open ngram-only PR #44056 (happy to rebase or split if maintainers prefer; #50518 covers the scheduler-side view for uniform batches). ## Proposed change Land `suffix_gpu` as a speculative decoding method: config surface, `SuffixProposerGPU`, runner wiring behind `use_gpu_state_drafter()`, unit tests. Branch: `zip95297/suffix_gpu` on https://github.com/zip95297/vllm-dev. Docs plan (same shape as #25784's docs addition): a `suffix_gpu` subsection in `docs/features/spec_decode.md` — when to pick it over `suffix`/`ngram_gpu`, the `suffix_gpu_*` knobs and their defaults, the async-scheduling compatibility note, the memory-headroom guidance for large k, and the package install hint. Lands with this PR or as an immediate follow-up once the packaging question below is settled (in-tree vs external changes the install section). ## Open question Keep the drafter core as an external optional dependency (arctic-inference precedent; small in-tree diff, kernels iterate on their own cadence) or port it in-tree under `v1/spec_decode/` (~1.5 kLOC pure Python + Triton, torch-only deps; one source of truth, CI coverage)? I lean in-tree for a decode-hot-path feature but either works; the wiring in this PR is the same either way. ## Future work Variable draft-length scheduling for GPU-state drafters (Dynamic SD #32374 integration and per-request draft lengths) is deliberately out of scope; it applies to `ngram_gpu` and `suffix_gpu` equally and deserves its own design discussion.", "url": "https://github.com/vllm-project/vllm/issues/51788", "createdAt": "2026-08-11T09:12:06Z", "updatedAt": "2026-08-13T04:53:39Z", "timestamp": "2026-08-13T04:53:39Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "RFC" ], "author": "zip95297", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3fd7016d2629c8b9f53b", "signalId": "github:vllm-project/vllm:pull_request:50554", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50554", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3][ROCm] Fuse KDA upper-triangle zeroing", "text": "Avoid a separate full `Akk` buffer memset by writing the required zeros in the existing fused solve kernel. ## Purpose Kimi-K3's ROCm KDA prefill path allocates `Akk` with `torch.zeros`, even though the fused inter/solve Triton kernel already writes its lower triangle. This change makes the fused kernel explicitly write the required upper-triangle zeros, allowing `Akk` to use `torch.empty` and avoiding a separate full-buffer memset. It supports: - Fixed-length and variable-length inputs - Safe-gate and standard-gate paths - Partial chunks with boundary masking No documentation update is required because this is an internal performance optimization with no API or model-support changes. ## Test Plan Run the targeted ROCm tests: python3 -m pytest -q tests/models/kimi_k3/test_amd_kda.py Run Ruff checks: ruff check \\ vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py \\ tests/models/kimi_k3/test_amd_kda.py ruff format --check \\ vllm/models/kimi_k3/amd/ops/third_party/kda/chunk_intra.py \\ tests/models/kimi_k3/test_amd_kda.py Validate performance using Kimi-K3 TP8 on 8 × MI355X: - Input length: 100,000 tokens - Output length: 20 tokens - Concurrency: 1 - Prefix caching: disabled - Five measured requests per run - Two independent baseline and patched runs ## Test Result ### Correctness Targeted ROCm tests: 4 passed The tests cover: - Fixed-length input with `safe_gate=False` - Fixed-length input with `safe_gate=True` - Variable-length input with `safe_gate=False` - Variable-length input with `safe_gate=True` Additional A/B validation confirmed: - `Akk` matched the original implementation bit-for-bit - End-to-end KDA output matched bit-for-bit - The upper triangle remained zero for every tested sequence and partial chunk - Ruff lint and formatting checks passed ### Accuracy / Numerical Parity The optimization does not change the KDA arithmetic. It only moves the required upper-triangle zero writes from a standalone `torch.zeros` initialization into the existing fused solve kernel. A direct baseline-versus-patched kernel comparison was performed with identical inputs. Results: - `Akk` matched the original zero-initialized implementation bit-for-bit. - The complete KDA output matched bit-for-bit. - Sequence lengths tested: `64`, `65`, `127`, `128`, `129`, and `1024`. - Partial chunks and chunk-boundary cases were included. - Fixed-length and variable-length layouts were tested. - Both `safe_gate=True` and `safe_gate=False` produced an exactly zero upper triangle. - No NaNs or numerical regressions were observed. - All TP8 serving requests completed successfully. The 100K serving environment was not deterministic at the generated-text level: repeated runs of the same patched server also produced different generated text. Therefore, generated-text equality was not used as numerical-parity evidence. Kernel-level tensor comparison was used instead and showed bit-for-bit output parity. ### Kernel Performance Same-GPU alternating A/B using Kimi-K3 TP8 per-rank shapes: Tokens Baseline median Patched median Improvement 8,192 0.1614 ms 0.1580 ms 2.08% 32,768 0.5193 ms 0.5157 ms 0.70% 100,000 1.5398 ms 1.5250 ms 0.96% ### End-to-End Performance Kimi-K3 TP8, 100K input, concurrency 1: Baseline mean TTFT, run 1: 7223.33 ms Baseline mean TTFT, run 2: 7221.91 ms Patched mean TTFT, run 1: 7206.20 ms Patched mean TTFT, run 2: 7210.31 ms Baseline average: 7222.62 ms Patched average: 7208.26 ms Improvement: 14.36 ms (0.20%) All serving requests succeeded. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR has been described. - [x] The test plan includes the relevant test commands. - [x] Correctness and performance results have been provided. - [x] Documentation impact was considered; no update is required.
", "url": "https://github.com/vllm-project/vllm/pull/50554", "createdAt": "2026-07-31T08:59:11Z", "updatedAt": "2026-08-13T04:49:52Z", "timestamp": "2026-08-13T04:49:52Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:648cd8a56f1bcc072e27", "signalId": "github:vllm-project/vllm:pull_request:51119", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51119", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] Use saturated Gluon split bucket for CUDA graphs", "text": "## Summary This PR changes the default `AiterMLADecodeMetadata.min_kv_seq_len` from `1` to `16384`. The change is intentionally small, but the reason is specific to the interaction between vLLM FULL CUDA Graph capture and AITER's host-side Gluon split selection: - AITER uses `min_kv_seq_len` to choose the split-KV launch topology in its Python wrapper. - vLLM currently leaves this field at its default value. - FULL CUDA Graph capture runs the wrapper once with dummy decode metadata and then replays captured GPU work without running the wrapper again. - With the old default, the captured graph is permanently specialized to the one-split path, even when the real request has a long KV sequence. This PR only fixes that vLLM integration issue. Split-policy tuning and stage-1/stage-2 kernel optimization remain AITER responsibilities. ## How the issue was found I first found this from Kimi-K3 TP8 end-to-end serving performance on gfx950. The 100K-input / 1K-output, concurrency-1 workload measured roughly 87–90 ms TPOT, while an isolated 12-head Gluon configuration with sufficient split parallelism indicated that the decode path should be much faster. The initial investigation focused on AITER. [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) tuned the 12-head BF16 Gluon split policy and tested context buckets for the Kimi-K3 TP8 shape. Reducing the split count also reduced the cost of the existing serial stage-2 reduction, so the first apparent solution was an AITER split-policy change. During the review of [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450), [@Dewei-Wang-sh](https://github.com/Dewei-Wang-sh) and I spent an afternoon tracing this behavior from the AITER wrapper back into vLLM. His key review direction was to [pass the split-sizing information from the caller](https://github.com/ROCm/aiter/pull/4450#discussion_r3710954584). He explained that the existing AITER interface follows the Moonshot integration: Moonshot obtains a meaningful minimum KV length in its runtime and passes it to AITER. In that contract, `1` is a fallback default; it is not intended to describe every real long-context decode request. This exposed the integration mismatch. vLLM was leaving the fallback at `1`. More importantly, when FULL CUDA Graph was enabled, vLLM performed warmup/capture with a dummy sequence length of 1. Dewei helped me follow the execution path and confirm that real decode replay did not run the host wrapper again. Therefore, even an attempted runtime assignment could not reach the Python code that selects the launch topology. This discussion and the subsequent call-chain tracing moved the fix from AITER to the vLLM caller. Thank you to Dewei for the detailed review and for helping identify the actual boundary of the problem. The relevant AITER host calculation is in [`mla_gluon.py`](https://github.com/ROCm/aiter/blob/d1898b7af894bda10310a631963771782e5aebc4/aiter/ops/triton/gluon/mla_gluon.py#L1042-L1048): ```python NUM_KV_SPLITS = max( 1, min( 256 // (batch_size * qlen * NUM_M_BLOCKS), triton.cdiv(min_kv_seq_len, BLOCK_N), ), ) ``` For the `BLOCK_N=64` path, capture with `min_kv_seq_len=1` gives `cdiv(1, 64)=1`, so the host calculation collapses to one KV split. The captured graph then permanently records that one-split launch. With the vLLM capture hint set to `16384`, the same term becomes `cdiv(16384, 64)=256`; it no longer forces the policy to one split, and AITER can apply its workgroup budget and small-head policy to capture a saturated, non-one-split topology. The real sequence length is still read from device metadata during replay; `16384` is only the host-side capture hint. In vLLM, the field was declared with a default of `1`, and `_build_decode()` constructed `AiterMLADecodeMetadata` without assigning it. `forward_mqa()` then passed that unchanged value to `mla_gluon()`: - [`min_kv_seq_len` metadata default](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L129-L133) - [`_build_decode()` metadata construction](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L691-L703) - [`forward_mqa()` passing the host hint to AITER](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/attention/backends/mla/rocm_aiter_mla.py#L1091-L1106) Therefore, even a real 10K or 100K request reached the AITER wrapper with `min_kv_seq_len=1` in normal execution. ## Why assigning the real runtime length did not fix FULL CUDA Graph I next tested assigning the real value while constructing decode metadata: ```python min_kv_seq_len=int(seq_lens_for_kernel.min()) ``` For an actual request this computes the expected value, for example approximately 10K or 100K. It works conceptually in eager execution because the AITER Python wrapper runs for every call and can select a new split count. It does not work for FULL CUDA Graph capture. For pure decode, vLLM's dummy run has query length 1. The capture path consequently builds dummy sequence lengths from `max_query_len`, which is also 1: - [pure-decode dummy batch construction](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/worker/gpu_model_runner.py#L5888-L5902) - [dummy attention sequence length selection](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/v1/worker/gpu_model_runner.py#L5988-L6015) The resulting sequence is: ```text capture startup: max_query_len = 1 dummy seq_len = 1 min_kv_seq_len = 1 AITER wrapper selects NUM_KV_SPLITS = 1 CUDA Graph records the one-split grid and the no-stage-2 fast path real request: device metadata is updated to the real KV length FULL graph replay starts AITER Python wrapper is not called again captured split count, grid, workspace, and stage topology remain unchanged ``` ## Instrumented FULL CUDA Graph trace To verify this rather than infer it from performance, I instrumented an isolated vLLM/AITER copy and ran one Kimi-K3 TP8 request with 100K input tokens and 4 output tokens. Logging was restricted to TP rank 0. The complete capture call chain was: ```text worker_busy_loop -> compile_or_warm_up_model -> capture_model -> _capture_cudagraphs -> _warmup_and_capture -> _dummy_run -> self.model -> BreakableCUDAGraphWrapper._capture -> Kimi-K3 MLA layer -> unified_mla_attention_with_output -> forward_mqa -> mla_gluon ``` The following excerpts are sanitized to retain only the events relevant to MLA graph capture and replay.
Eager warmup ```text MLA_TRACE event=dummy_metadata mode=NONE seq_lens=1 MLA_TRACE event=aiter_metadata_build seq_lens=[1] static_seq_info_ptr=137855956567552 MLA_TRACE event=forward_mqa_call stream_capturing=False host_min_kv_seq_len=1 MLA_TRACE event=mla_wrapper_split stream_capturing=False block_n=64 num_kv_splits=1 MLA_TRACE event=mla_stage1_launch stream_capturing=False grid=(1, 1, 1) logits_shape=(1, 1, 12, 1, 512) mid_lse_shape=None MLA_TRACE event=mla_stage2_skipped stream_capturing=False reason=num_kv_splits_is_1 ```
FULL graph capture ```text MLA_TRACE event=dummy_metadata mode=FULL graph_capture_request=True seq_lens=1 MLA_TRACE event=model_capture_enter mode=FULL MLA_TRACE event=forward_mqa_call stream_capturing=True host_min_kv_seq_len=1 seq_info_ptr=137855956567552 MLA_TRACE event=mla_wrapper_split stream_capturing=True block_n=64 num_kv_splits=1 MLA_TRACE event=mla_stage1_launch stream_capturing=True grid=(1, 1, 1) logits_shape=(1, 1, 12, 1, 512) mid_lse_shape=None MLA_TRACE event=mla_stage2_skipped stream_capturing=True reason=num_kv_splits_is_1 MLA_TRACE event=model_capture_exit mode=FULL graph_segments=1 eager_breaks=0 ``` At this point capture has fixed all of the host-selected topology: - `NUM_KV_SPLITS=1` - stage-1 grid `(1, 1, 1)` - workspace/logits shape `(1, 1, 12, 1, 512)` - `mid_lse=None` - no stage-2 GPU node exists in the graph
Real 100K decode replay ```text MLA_TRACE event=runtime_dispatch mode=FULL computed_before_step=[100000] MLA_TRACE event=aiter_metadata_build seq_lens=[100001] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=1 segments=1 eager_breaks=0 MLA_TRACE event=aiter_metadata_build seq_lens=[100002] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=2 segments=1 eager_breaks=0 MLA_TRACE event=aiter_metadata_build seq_lens=[100003] static_seq_info_ptr=137855956567552 MLA_TRACE event=model_graph_replay mode=FULL count=3 segments=1 eager_breaks=0 ``` The capture-time `seq_info_ptr` and all three runtime `static_seq_info_ptr` values are the same: `137855956567552`. This proves that the real metadata was copied into the same static GPU buffer used by the graph. However, there were no `forward_mqa_call`, `mla_wrapper_split`, or Python stage-launch events between the three replay steps.
### Why the wrapper is not re-entered This is not a JIT-cache effect. During capture, `BreakableCUDAGraphWrapper` executes the model and records the graph. On later calls, its [capture-versus-replay dispatch](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L310-L333) selects `_replay()`, which [calls only `entry.capture.replay()`](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L406-L424). The capture artifact then [replays its saved segments](https://github.com/vllm-project/vllm/blob/7f97cd901bc32f4d8240a10ea131281a0eac0626/vllm/compilation/breakable_cudagraph.py#L210-L214). This run reported `graph_segments=1, eager_breaks=0`, so the only saved segment was GPU graph replay; it was not a Python model, attention, or AITER-wrapper invocation. A PIECEWISE graph can re-enter Python when attention is registered as an eager break, but this FULL graph had no such break. JIT kernel caching also does not skip the Python wrapper and should not be confused with graph replay. The distinction is therefore: - **Dynamic during replay:** the real sequence length, because the captured kernel reads the latest values from the static `seq_info` GPU buffer. - **Frozen at capture:** `NUM_KV_SPLITS`, workspace shape, stage-1 grid, `mid_lse` allocation, and whether stage 2 exists, because those are host-side decisions made before the graph is recorded. This directly explains why a host-only split policy still regressed to approximately 87–88 ms: FULL capture recorded the `split=1` topology and replay could not change it. > **Diagnostic timing note:** this instrumented 100K-to-4-token run measured 131.33 ms TPOT, but the instrumentation added GPU-to-CPU synchronization, stack collection, and synchronous printing. That number is only call-chain evidence and must not be compared with the uninstrumented 23.49 ms result below. The raw logs are approximately 640 KB and contain unrelated runtime configuration, so only the focused, sanitized trace is included here. The runtime assignment also requires a device-to-host synchronization for `.min()`/`.item()`. It increased the measured TPOT from approximately 87–88 ms to approximately 89–90 ms while still replaying the one-split graph. It was therefore both insufficient and slower. ## Why use 16,384 for capture Changing vLLM to capture a separate graph for every context bucket would be a much larger architectural change. Moving split activation back to the device is an AITER policy decision and conflicts with the reviewer-requested host-side interface. Per-request device-to-host synchronization is not viable on the decode path. The smallest vLLM-side integration is therefore to provide one representative, saturated capture hint. `16384` was selected so capture does not fall into the one-split topology, while avoiding a context-length graph ladder or runtime synchronization. The value is a capture hint, not an assertion that every request has a 16K context. The kernel still reads the real per-request sequence length from `seq_info` during replay. The hint determines only the host-selected split topology that CUDA Graph records. The target workload matrix includes 8K/1K, 60K/1K, and 100K/1K cases. The fixed-hint end-to-end rerun completed so far is the 100K/1K case; the shorter fixed-hint cases have not yet been rerun end to end. ## Runtime-owner feedback and related AITER work The AITER runtime discussion clarified that two separate problems had been mixed together: 1. **vLLM integration correctness/performance:** the caller leaves the host hint at 1 and FULL graph capture freezes that topology. 2. **AITER kernel efficiency:** with a valid large split count, the original stage-2 reduction becomes serial and expensive. The related AITER PRs explore different kernel-side solutions: - [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) started the 12-head split-policy investigation and production-like scattered-page validation. - [ROCm/aiter#4507](https://github.com/ROCm/aiter/pull/4507) derives host launch sizing from page-table shape rather than the optional hint. - [ROCm/aiter#4509](https://github.com/ROCm/aiter/pull/4509) adds split-major stage-1 scheduling and a more parallel stage-2 reduction on top of [ROCm/aiter#4507](https://github.com/ROCm/aiter/pull/4507). - [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555), from the AITER runtime team, directly optimizes stage-2 by loading and reducing a block of partial outputs at once instead of walking one split at a time. The runtime-owner measurement makes the separation important: | Configuration | Splits | Stage 1 | Stage 2 | Total | |---|---:|---:|---:|---:| | Original serial reduction | 256 | 34.7 us | 74.3 us | 109.0 us | | Split-bucket workaround | 96 | 43.1 us | 26.1 us | 69.2 us | | [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) blocked reduction | 256 | 34.7 us | 10.7 us | 46.9 us | This shows that reducing splits helped partly because it avoided an inefficient stage-2 implementation. [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) is the better kernel optimization: it preserves enough stage-1 workgroups to fill the CUs and fixes stage-2 itself. The AITER runtime team will own that optimization. Accordingly, this PR does not claim that 48 or 96 splits is the final AITER policy. It only ensures that vLLM does not silently force every host-side policy to the pathological one-split capture. ## End-to-end validation Kimi-K3 TP8 on 8x MI355X/gfx950, BF16 KV, FULL CUDA Graph, 100K input / 1K output, concurrency 1: | Configuration | TPOT | |---|---:| | Previous host-side capture with default `1` | approximately 87.8 ms | | Runtime `min()` metadata experiment | approximately 89.2 ms | | This PR, fixed 16K capture hint | 23.49 ms | | Device-side runtime-policy reference | approximately 23.0 ms | The fixed capture hint removes the regression without changing graph dispatch or adding decode-path synchronization. ## Scope question for maintainers The remaining concern is short-context behavior. This change assumes that the small-head Gluon path is primarily valuable for the long-context workloads for which split parallelism is required. A fixed 16K hint may over-split very short requests. AITER has empty-split guards, and [ROCm/aiter#4450](https://github.com/ROCm/aiter/pull/4450) validated explicit split counts larger than sequence lengths at boundaries including sequence length 1, so this is expected to be correct. However, the short-context performance trade-off still deserves explicit confirmation. Would maintainers prefer: 1. this single representative capture hint for the Gluon path; 2. a more narrowly scoped hint for gfx950/small-head deployments; 3. a configurable vLLM capture hint; or 4. relying on an AITER-side solution such as page-table sizing or device-side active split selection? Given the runtime-owner direction toward [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) for kernel performance, my preference is to keep this PR narrowly focused on preventing `min_kv_seq_len=1` from being frozen by vLLM FULL CUDA Graph capture. ## Change ```diff - min_kv_seq_len: int = 1 + min_kv_seq_len: int = 16384 ``` No graph-dispatch, replay-path, or AITER kernel code is changed. ## Test plan - [x] `python3 -m py_compile vllm/v1/attention/backends/mla/rocm_aiter_mla.py` - [x] Kimi-K3 TP8, gfx950, 100K input / 1K output / concurrency 1 - [x] Confirmed FULL capture enters `forward_mqa()`/`mla_gluon()` once and runtime decode uses graph replay without re-entering either wrapper - [x] Confirmed runtime sequence metadata updates the same static GPU `seq_info` buffer used by the captured graph - [ ] Re-run the fixed capture hint at 8K/1K and 60K/1K to quantify short/mid-context performance", "url": "https://github.com/vllm-project/vllm/pull/51119", "createdAt": "2026-08-05T05:55:47Z", "updatedAt": "2026-08-13T04:46:13Z", "timestamp": "2026-08-13T04:46:13Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "nvidia" ], "author": "LiuYinfeng01", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f2dabb91eec45d920a3b", "signalId": "github:vllm-project/vllm:pull_request:52063", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52063", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Avoid AITER FP8 BMM for MLA K projection", "text": "## Purpose This changes the ROCm AITER FP8 MLA decode path so the K-side up-projection stays dense after the post-load transform, while the V-side up-projection continues to use the AITER FP8 BMM. The K-side MLA projection is a skinny batched matmul. Requantizing the transformed K weight and routing it through AITER FP8 BMM adds overhead for this shape and adds another quantization step. Keeping K dense reuses the existing `torch.bmm` path, keeps the K-side math tied directly to the dequantized model weight, and leaves the V-side AITER path unchanged. What changed: - remove the AITER FP8 runtime branch for the K-side MLA projection - keep `W_UK_T` dense in `process_weights_after_loading` for the AITER FP8 path - keep quantizing/precompiling `W_V` for the V-side AITER FP8 BMM - add a unit test that checks both the post-load storage contract and the K-side projection math This PR was developed with AI assistance. I reviewed the changed code paths and ran the checks below. ## Duplicate-work check No linked issue exists for this change. I checked for open duplicate work with: - `gh pr list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` - `gh pr list --repo vllm-project/vllm --state open --search \"rocm aiter mla fp8 bmm\"` - `gh pr list --repo vllm-project/vllm --state open --search \"triton_fp8_bmm mla warmup\"` - `gh issue list --repo vllm-project/vllm --state open --search \"AITER MLA FP8 BMM warmup\"` The closest related open PR is #52096, which is also mine. It is related but not a duplicate: #52096 changes startup precompile bounds for the AITER FP8 BMM path, while this PR changes the runtime K-side MLA projection path. If one lands first, the other may need a small rebase in the same `process_weights_after_loading` area. ## Test Plan Local focused correctness: ```bash PYTHONPATH=. uv run --no-sync --python 3.12 \\ --with-requirements requirements/common.txt \\ --with pytest --with torch --with tblib \\ pytest -q --confcutdir=tests/v1/attention \\ tests/v1/attention/test_mla_backends.py::test_mla_kv_cache_spec_uses_layer_cache_dtype \\ tests/v1/attention/test_mla_backends.py::test_mla_post_load_preserves_runtime_weight_addresses \\ tests/v1/attention/test_mla_backends.py::test_mla_aiter_fp8_post_load_keeps_k_side_dense ``` Local lint: ```bash pre-commit run --files \\ vllm/model_executor/layers/attention/mla_attention.py \\ tests/v1/attention/test_mla_backends.py ``` MI300X focused correctness and microbench remain from the earlier draft run. I have not rerun full serve/generation on this branch after rebasing to current `origin/main`. ## Test Result Local focused correctness: `4 passed, 14 warnings`. Local lint: `pre-commit run --files ...` passed. Earlier MI300X focused correctness: `4 passed, 14 warnings`. Earlier MI300X isolated K-side projection results: | tokens | old AITER FP8 K BMM median us | dense K `torch.bmm` median us | median speedup | |---:|---:|---:|---:| | 1 | 35.205 | 14.536 | +142.2% | | 2 | 34.709 | 14.630 | +137.2% | | 4 | 34.777 | 14.795 | +135.1% | | 8 | 34.813 | 14.665 | +137.4% | | 16 | 34.778 | 14.692 | +136.7% | | 32 | 34.852 | 14.800 | +135.5% | | 64 | 34.728 | 13.852 | +150.7% | | 128 | 34.272 | 13.934 | +146.0% | | 256 | 34.517 | 13.887 | +148.6% | Caveat: I could not complete a full editable vLLM ROCm build on the current RunPod image because it ships PyTorch `2.10.0+rocm7.1.1`, while current vLLM's ROCm `_C_stable_libtorch` extension expects a newer PyTorch stable C++ API. The build failed before this patch was involved, at `_C_stable_libtorch` with missing `torch::stable::Tensor::layout`. A full serve/generation check should be run in the official vLLM ROCm dev image or another environment with the current supported ROCm/PyTorch stack.", "url": "https://github.com/vllm-project/vllm/pull/52063", "createdAt": "2026-08-12T22:53:37Z", "updatedAt": "2026-08-13T04:39:55Z", "timestamp": "2026-08-13T04:39:55Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "rohash123", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0bec7a94132dc1cd5488", "signalId": "github:vllm-project/vllm:pull_request:52097", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52097", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Spec Decode] Add suffix_gpu drafter for Async Scheduling", "text": "## Purpose Suffix decoding (`method=\"suffix\"`, #25784) has the best acceptance lengths among model-free drafters on repetitive/agentic traffic, but it is CPU-side and rejected by the async-scheduling whitelist, so it gives up CPU/GPU overlap exactly where spec decode helps most (high-concurrency TPOT). This PR adds `method=\"suffix_gpu\"`: a GPU-resident suffix drafter following the same device-state drafter contract as `ngram_gpu` (#29184) — state on device, drafts from the previous step's GPU sampled ids, no host sync on the draft path — which makes suffix decoding compatible with async scheduling. RFC: #51788. In-tree changes (~590 lines outside tests, ~300 lines of tests): - `config/speculative.py`: `suffix_gpu` method; reuses `suffix_decoding_*` knobs, adds `suffix_gpu_*` GPU knobs; `use_gpu_state_drafter()` groups ngram_gpu + suffix_gpu; async-scheduling whitelist admits `suffix_gpu`. - `v1/spec_decode/suffix_proposer_gpu.py`: `SuffixProposerGPU` wrapper — borrows `update_token_ids_ngram` from `NgramProposerGPU`, same `propose()` signature and valid-count async D2H; adds cross-request global-index ingest (side stream, off the critical path) and a bucketized CUDA graph for the draft chain, pre-captured at warmup. - `v1/worker/gpu_model_runner.py`: `ngram_gpu`-only gates generalized to `use_gpu_state_drafter()` (`is_ngram_gpu` local renamed to `is_gpu_state_drafter`); suffix-specific ingest hooks and draft-graph pre-capture. - `tests/v1/spec_decode/test_suffix_gpu.py`: proposer unit tests. - `tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py`: `test_suffix_gpu_with_async_scheduling` (GSM8K under async, mirroring the ngram_gpu async test) and `test_suffix_gpu_acceptance` (acceptance improves as the global index warms, same pattern as the suffix CPU acceptance test; pins `suffix_gpu_ingest_chunk=1`, `num_backoff=8`, `max_occurrences=128`). Both acceptance tests — including the pre-existing CPU one — now pin `enable_prefix_caching=False` and the V1 model runner: warm rounds replay identical prompts, and prefix-cache hits change decode batch composition enough to depress the measured rate below the 0.80 floor for the CPU method too - Metrics accounting for GPU-state drafters (dedicated merge): `update_scheduler_for_invalid_drafts` returns per-request padded-slot trim counts, the worker reports them via `ModelRunnerOutput.num_invalid_spec_tokens`, and `make_spec_decoding_stats` subtracts them (fully-padded drafts are skipped instead of being recorded as zero-token draft events). Without this, padded-k scheduling deflates the reported acceptance rate for ngram_gpu and suffix_gpu alike and makes GPU and CPU drafter stats incomparable. The drafter core lives in a standalone package (https://github.com/zip95297/SuffixGPU), lazily imported — same pattern as `suffix`'s `arctic-inference` dependency; vLLM works without it installed. All results in this PR were produced with suffix-gpu 0.1.1 (PyPI, tag `v0.1.1`; the runs used the equivalent package code at repo commit `1ac9238`, installed editable — the only delta is constructor defaults, which the vLLM config passes explicitly). Not duplicating existing work (searched 2026-08-12): no open PR wires suffix decoding into the async path — open-PR search for \"suffix\" in titles returns docs/loader/KV items only, \"suffix_gpu\" returns none; #25784 is CPU+sync-only, #29184 is ngram-only. The metrics accounting part overlaps in mechanism with open PR #44056 (no activity since May) and is required here for suffix_gpu's acceptance test and honest benchmark numbers; this PR uses method-neutral naming (`num_invalid_spec_tokens`), also fixes ngram_gpu, and I am happy to rebase onto #44056 if it lands first or split this part out if maintainers prefer. #50518 covers the scheduler-side view for uniform spec batches and is unaffected. ## Test Plan - Unit tests: `pytest tests/v1/spec_decode/test_suffix_gpu.py -v` (skips without CUDA / suffix_gpu package -> CI-safe) - Acceptance e2e (in-tree pattern; suffix_gpu + async and the pre-existing CPU suffix test as a regression check): `pytest tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance -v` - Greedy consistency (spec decode must not change greedy outputs): `VLLM_BATCH_INVARIANT=1 python benchmarks/greedy_consistency.py 16` async no-spec vs async suffix_gpu (+ suffix_gpu eager / ngram_gpu / sync suffix CPU legs), 16 prompts x 256 tokens, token-identical required. Batch-invariant kernels + enforce_eager are required for a bitwise gate: spec decode changes batch composition, and batch-dependent FP reduction order (FlashAttention split heuristics) flips near-tie greedy argmax even between two no-spec runs. - Serving benchmark mirroring #25784's setup (Spec-Bench, TPOT + drafted/ accepted totals, k x concurrency grid) and #29184's async-vs-sync comparison: Llama-3.1-8B-Instruct bf16 TP=1 on 1x L20 48GB, k in {5,16}, concurrency 1/4/16/64/128/192/256, prefix caching off, max_num_batched_tokens=8192 and max_num_seqs=320 for every variant (spec decode schedules (k+1) padded token slots per running request before invalid ones are trimmed, so the serve-context defaults silently cap effective concurrency at the top grid points). Script: benchmarks/bench_pr_matrix.sh (attached). - Drafter-level correctness/latency/VRAM vs the arctic CPU suffix tree (deterministic Spec-Bench replay, 221 tests): equivalence on unambiguous corpora; per-step drafting cost crosses over at B~64-128 — beyond that, CUDA-graph mode is 2.1-2.7x faster than the CPU tree's sequential walk (B=256: 1.85 vs 4.98 ms), which is what makes the drafter viable in the high-concurrency regime async scheduling targets. Full tables: https://github.com/zip95297/SuffixGPU/blob/main/RESULTS.md ## Test Result ### Unit tests ``` $ pytest tests/v1/spec_decode/test_suffix_gpu.py -v 6 passed in 5.93s (L20, torch 2.13.0+cu130, branch commit f646af65c) test_propose_drafts_repetition[False/True], test_graph_and_eager_agree, test_capture_draft_graph_at_warmup, test_capture_draft_graph_warms_up_without_graph, test_ingest_and_cross_request_draft ``` ### E2E GSM8K (in-tree pattern, mirrors ngram_gpu's async test) ``` $ pytest \"tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py::test_suffix_gpu_with_async_scheduling\" -v 1 passed — GSM8K accuracy: 0.762 (threshold 0.70) Llama-3.1-8B-Instruct, suffix_gpu k=16, async_scheduling=True, 1319 questions 5-shot ``` ### Acceptance e2e (in-tree pattern, mirrors the CPU suffix acceptance test) ``` $ pytest tests/v1/e2e/spec_decode/ngram_suffix/test_ngram_suffix.py -k acceptance -v 2 passed test_suffix_gpu_acceptance (suffix_gpu k=16, async scheduling) test_suffix_decoding_acceptance (CPU suffix, unchanged-behavior check) ``` suffix_gpu warm-up on the shared workload (10 rounds x 100 prompts, per-round rate/AL from an identical-config probe, prefix caching off): round 0 rate 0.428 / AL 2.03 -> round 9 rate 0.857 / AL 5.70. The final-round delta is only ~1.4k drafted tokens, so the end rate moves a few points between reruns (0.79-0.86 observed; greedy near-ties under varying async batch composition); it clears the same 0.80 floor the CPU test uses. ### Greedy consistency ``` $ VLLM_BATCH_INVARIANT=1 python benchmarks/greedy_consistency.py 16 16 prompts x 256 tokens, k=16, Llama-3.1-8B-Instruct, enforce_eager: suffix_gpu (graph) vs async no-spec : 0/16 mismatches suffix_gpu (eager) vs async no-spec : 0/16 suffix_gpu eager vs graph replay : 0/16 (graph does not change outputs) ngram_gpu vs async no-spec : 0/16 suffix CPU (sync) vs async no-spec : 0/16 identical: true ``` ### Spec-Bench serving (Llama-3.1-8B-Instruct, 1x L20) 480 Spec-Bench prompts per point, output len 256, prefix caching off, `--max-num-batched-tokens 8192 --max-num-seqs 320` for every variant, identical 64-prompt warmup per server before measurement. Full script attached below; raw results under `benchmarks/results/`. **k=5 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 58 (17.2) | 245 (15.4) | 754 (19.8) | 1297 (45.5) | 1343 (86.0) | 1375 (127.2) | 1373 (169.5) | | ngram_gpu_async | 46 (21.6) | 168 (23.3) | 515 (30.3) | 1126 (52.7) | 1341 (89.2) | 1389 (128.5) | 1396 (172.3) | | suffix_gpu_async | 57 (17.6) | 239 (15.6) | 744 (19.3) | 1316 (43.2) | 1410 (80.5) | 1461 (117.6) | 1467 (154.5) | **k=16 — output tok/s (mean TPOT ms)** | variant | c1 | c4 | c16 | c64 | c128 | c192 | c256 | |---|---|---|---|---|---|---|---| | async_nospec | 47 (21.0) | 172 (22.7) | 578 (26.4) | 1286 (47.0) | 1566 (75.6) | 1659 (109.6) | 1654 (147.1) | | suffix_sync | 55 (18.2) | 239 (15.9) | 755 (19.5) | 1273 (46.3) | 1322 (86.0) | 1357 (125.8) | 1363 (167.2) | | ngram_gpu_async | 43 (23.0) | 161 (24.3) | 519 (29.9) | 1156 (52.2) | 1339 (89.6) | 1387 (131.0) | 1405 (170.6) | | suffix_gpu_async* | 52 (19.2) | 230 (16.2) | 736 (19.5) | 1276 (44.8) | 1349 (84.3) | 1402 (121.5) | 1417 (154.6) | **Drafted / accepted tokens per point (AL, acceptance rate)** | variant | k | c1 | c16 | c64 | c256 | |---|---|---|---|---|---| | suffix_sync | 5 | 79832/22918 (1.54, 28.7%) | 86405/33544 (1.94, 38.8%) | 86516/32697 (1.89, 37.8%) | 86240/31708 (1.85, 36.8%) | | suffix_gpu_async | 5 | 79742/22332 (1.52, 28.0%) | 87458/33453 (1.93, 38.3%) | 88350/35551 (2.04, 40.2%) | 88444/38485 (2.23, 43.5%) | | ngram_gpu_async | 5 | 33913/9467 (2.40, 27.9%) | 34863/9738 (2.40, 27.9%) | 34527/9332 (2.35, 27.0%) | 34358/9667 (2.41, 28.1%) | | suffix_sync | 16 | 83232/22867 (1.54, 27.5%) | 95070/34453 (1.98, 36.2%) | 93340/33505 (1.94, 35.9%) | 92211/32536 (1.89, 35.3%) | | suffix_gpu_async | 16 | 82552/22582 (1.53, 27.4%) | 95486/34166 (2.00, 35.8%) | 98323/36571 (2.13, 37.2%) | 105420/40513 (2.33, 38.4%) | | ngram_gpu_async | 16 | 35468/9978 (2.41, 28.1%) | 34844/9586 (2.38, 27.5%) | 33685/9643 (2.43, 28.6%) | 34316/9778 (2.42, 28.5%) | Findings: 1. **suffix_gpu + async vs suffix + sync** (the conflict this PR resolves): parity at c1-c16, ahead from c64 up — +7% at k=5 saturation (1467 vs 1373 tok/s), +4% at k=16 (1417 vs 1363), with lower TPOT (154.5 vs 169.5 ms at k=5 c256). 2. **vs ngram_gpu + async** (in-tree GPU drafter): ahead at every grid point, up to +44% (k=5 c16: 744 vs 515 tok/s). 3. **vs async no-spec**: ahead through c64 (TPOT -8% at c64, -27% at c16, -31% at c4), behind at c128+ on this mixed workload (1467 vs 1654 at k=5 c256) — at saturation, verification compute outweighs acceptance on Spec-Bench's non-repetitive categories. Same regime behavior as the other drafters (suffix_sync 1373, ngram_gpu 1396). 4. **Acceptance scales with concurrency for suffix_gpu only**: AL 1.52 -> 2.23 (k=5) and 1.53 -> 2.33 (k=16) from c1 to c256 while the CPU suffix tree stays flat (1.94 -> 1.85, 1.98 -> 1.89) — the cross-request global index compounds under concurrent traffic. 5. Per-position acceptance at c=1 is essentially identical to the CPU suffix tree (k=16: 0.328/0.113/0.045... vs 0.329/0.115/0.046...) — the GPU drafter reproduces the CPU method's draft quality; full table in the RFC. --- - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) Documentation update — planned (see RFC \"Docs plan\"): `suffix_gpu` subsection in `docs/features/spec_decode.md` (when to pick it over suffix/ngram_gpu, `suffix_gpu_*` knobs, async-scheduling note, large-k memory-headroom guidance, install hint); lands with this PR or as an immediate follow-up once packaging (in-tree vs external) is settled. AI assistance was used for code analysis, benchmark scripts; all changed lines were reviewed and validated by the author, and all tests above were run by the author. Signed-off-by: zip95297 ---
Benchmark reproduction script (bench_pr_matrix.sh) ```bash #!/bin/bash # PR benchmark matrix for suffix_gpu (mirrors vllm-project/vllm#25784 setup). # # Grid: 4 variants x k {5,16} x concurrency {1,4,16,64,128,192,256} # async_nospec : async scheduling, no spec (baseline, k-independent) # suffix_sync : method=suffix (CPU tree, arctic), sync scheduling # (async rejects it -- that is the point of this PR) # ngram_gpu_async : method=ngram_gpu, async scheduling (in-tree GPU drafter) # suffix_gpu_async : method=suffix_gpu, async scheduling (this PR) # # Model: meta-llama/Llama-3.1-8B-Instruct (bf16, TP=1), 1x NVIDIA L20 48GB # Dataset: Spec-Bench question.jsonl (md5 0c39ae23e6f213549c66d6d691c99034) # Suffix knobs shared with the CPU suffix method: max_tree_depth=24, # max_cached_requests=1000. ngram uses prompt_lookup 3-5 (the stronger # baseline config from #25784). set -x export HF_HUB_OFFLINE=1 # pyarrow's embedded jemalloc background thread SIGSEGVs in the EngineCore # subprocess on this box; disable it. export JE_ARROW_MALLOC_CONF=background_thread:false # All variants forced onto model runner v1 for a fair comparison # (suffix/ngram_gpu/suffix_gpu fall back to v1 anyway). export VLLM_USE_V2_MODEL_RUNNER=0 BASE=/home/admin/workspace/aop_lab/app_source/zjb/suffix_async/pr_to_vllm_final REPO=${BASE}/vllm MODEL=meta-llama/Llama-3.1-8B-Instruct DATA=${BASE}/benchmarks/question.jsonl OUT=${BASE}/benchmarks/results PATH=\"${REPO}/.venv/bin:${PATH}\" # libcuda 580 shim (kernel driver 580 / userland libcuda 12.2 mismatch) export LD_LIBRARY_PATH=${BASE}/.cudafix:${LD_LIBRARY_PATH:-} export HF_HUB_CACHE=${BASE}/hf_cache mkdir -p \"${OUT}\" cd \"${BASE}/benchmarks\" [ -f \"${DATA}\" ] || cp /home/admin/question.jsonl \"${DATA}\" echo \"$(cd ${REPO} && git rev-parse HEAD) $(cd ${REPO} && git status --short | wc -l) dirty-files\" \\ > \"${OUT}/run_manifest.txt\" echo \"suffix-gpu $(git -C ${BASE}/../SuffixGPU rev-parse --short HEAD) $(git -C ${BASE}/../SuffixGPU status --short | wc -l) dirty-files\" \\ >> \"${OUT}/run_manifest.txt\" date >> \"${OUT}/run_manifest.txt\" wait_server() { local pid=$1 i=0 until curl -s http://localhost:8000/v1/models >/dev/null; do kill -0 \"$pid\" 2>/dev/null || { echo \"SERVER PROCESS DIED\"; return 1; } sleep 5 i=$((i + 5)) if [ \"$i\" -ge 900 ]; then echo \"SERVER START TIMEOUT\"; return 1; fi done } for k in 5 16; do suffix_cfg=$(cat < \"${OUT}/server_${name}_k${k}.log\" 2>&1 & pid=$! if ! wait_server ${pid}; then echo \"SKIP ${name} k${k}: server failed to start\" | tee -a \"${OUT}/failures.txt\" kill ${pid} 2>/dev/null; wait ${pid} 2>/dev/null; sleep 10 continue fi # Same warmup for every variant before measuring: model/drafter CUDA # graphs, Triton JIT and the suffix global index all start cold. # Snapshot the cumulative spec counters afterwards so the first # measured point's delta excludes warmup traffic. vllm bench serve \\ --model ${MODEL} \\ --dataset-name spec_bench \\ --dataset-path \"${DATA}\" \\ --spec-bench-output-len 256 \\ --num-prompts 64 \\ --max-concurrency 16 \\ --no-oversample curl -s http://localhost:8000/metrics \\ | grep -E \"spec_decode\" > \"${OUT}/metrics_${name}_k${k}_warmup.txt\" || true for concurrency in 1 4 16 64 128 192 256; do vllm bench serve \\ --model ${MODEL} \\ --dataset-name spec_bench \\ --dataset-path \"${DATA}\" \\ --spec-bench-output-len 256 \\ --max-concurrency ${concurrency} \\ --no-oversample \\ --save-result \\ --result-filename \"${OUT}/bench_${name}_k${k}_c${concurrency}.json\" # Per-concurrency spec metrics delta comes from the cumulative # counters; snapshot after each client run. curl -s http://localhost:8000/metrics \\ | grep -E \"spec_decode\" > \"${OUT}/metrics_${name}_k${k}_c${concurrency}.txt\" || true done kill ${pid}; wait ${pid} 2>/dev/null sleep 10 done done echo \"BENCHMARK ALL DONE\" ```
Greedy-consistency gate (greedy_consistency.py) ```python # Greedy-consistency gate for suffix_gpu. # # Speculative decoding must not change greedy outputs: run the same # prompts with (a) async scheduling + no spec and (b) async scheduling + # suffix_gpu, and require token-identical outputs. Additional legs # (suffix_gpu eager, ngram_gpu, CPU suffix) localize any divergence. # # Bitwise comparability requires batch-invariant kernels: spec decode # changes batch composition, and FlashAttention's split heuristics make # FP reduction order batch-dependent, flipping near-tie greedy argmax # (verified: without this, async no-spec disagrees with itself across # duplicated prompts in one batch). So we set VLLM_BATCH_INVARIANT=1 and # enforce_eager=True (batch-invariant aten overrides must not be # bypassed by compile/cudagraphs; on SM<90 vLLM cannot guarantee them # under compiled paths). # # Each engine runs in its own subprocess so GPU memory is fully # released between runs (spawn context; worker must be module-level). import json import os import sys os.environ.setdefault(\"VLLM_BATCH_INVARIANT\", \"1\") from vllm import LLM, SamplingParams MODEL = \"meta-llama/Llama-3.1-8B-Instruct\" K = int(sys.argv[1]) if len(sys.argv) > 1 else 16 SUFFIX_GPU_SPEC = { \"method\": \"suffix_gpu\", \"num_speculative_tokens\": K, \"suffix_decoding_max_cached_requests\": 1000, \"suffix_decoding_max_tree_depth\": 24, } SUFFIX_CPU_SPEC = { \"method\": \"suffix\", \"num_speculative_tokens\": K, \"suffix_decoding_max_cached_requests\": 1000, \"suffix_decoding_max_tree_depth\": 24, } # High-repetition prompts (exercise the accept path) + regular prompts # (exercise the empty-draft path), repeated for multi-request scheduling. PROMPTS = [ \"Repeat the following sentence 20 times: the quick brown fox jumps over the lazy dog.\", \"Count from 1 to 100, separated by commas.\", \"Explain the theory of relativity in one paragraph.\", 'List the JSON object {\"a\": 1, \"b\": 2} fifteen times, one per line.', ] * 4 SP = SamplingParams(temperature=0.0, max_tokens=256, ignore_eos=True) def run(mode: str) -> list[list[int]]: kwargs = { \"model\": MODEL, \"enable_prefix_caching\": False, \"gpu_memory_utilization\": 0.8, # Batch-invariant aten overrides must not be bypassed by # compile/cudagraphs (SM8.9 cannot guarantee them otherwise). \"enforce_eager\": True, } if mode == \"async_nospec\": kwargs[\"async_scheduling\"] = True elif mode == \"async_suffix_gpu\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = SUFFIX_GPU_SPEC elif mode == \"async_suffix_gpu_eager\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = { **SUFFIX_GPU_SPEC, \"suffix_gpu_use_cuda_graph\": False, } elif mode == \"async_ngram_gpu\": kwargs[\"async_scheduling\"] = True kwargs[\"speculative_config\"] = { \"method\": \"ngram_gpu\", \"num_speculative_tokens\": K, \"prompt_lookup_min\": 3, \"prompt_lookup_max\": 5, } elif mode == \"sync_suffix_cpu\": kwargs[\"async_scheduling\"] = False kwargs[\"speculative_config\"] = SUFFIX_CPU_SPEC else: raise ValueError(mode) llm = LLM(**kwargs) outs = llm.generate(PROMPTS, SP) token_ids = [list(o.outputs[0].token_ids) for o in outs] del llm return token_ids def _worker(q, mode): q.put(run(mode)) def run_in_subprocess(mode: str) -> list[list[int]]: import multiprocessing as mp ctx = mp.get_context(\"spawn\") q = ctx.Queue() for _attempt in range(3): p = ctx.Process(target=_worker, args=(q, mode)) p.start() try: ids = q.get(timeout=1800) p.join() return ids except Exception: p.kill() p.join() import time time.sleep(20) raise RuntimeError(f\"engine failed 3 times for mode={mode}\") def diff(ref: list[list[int]], got: list[list[int]]) -> list[dict]: mismatches = [] for i, (r, g) in enumerate(zip(ref, got)): if r != g: j = next( k for k in range(min(len(r), len(g)) + 1) if k >= min(len(r), len(g)) or r[k] != g[k] ) mismatches.append( { \"prompt_idx\": i, \"first_diff_pos\": j, \"ref_len\": len(r), \"got_len\": len(g), } ) return mismatches if __name__ == \"__main__\": import os prev = f\"greedy_consistency_k{K}.json\" if os.path.exists(prev): cached = json.load(open(prev)) nospec_ids = cached[\"nospec_ids\"] sgpu_ids = cached[\"suffix_gpu_ids\"] scpu_ids = cached[\"suffix_cpu_ids\"] print(\"reusing cached nospec/suffix_gpu/suffix_cpu ids\") else: nospec_ids = run_in_subprocess(\"async_nospec\") sgpu_ids = run_in_subprocess(\"async_suffix_gpu\") scpu_ids = run_in_subprocess(\"sync_suffix_cpu\") sgpu_eager_ids = run_in_subprocess(\"async_suffix_gpu_eager\") ngram_ids = run_in_subprocess(\"async_ngram_gpu\") result = { \"k\": K, \"num_prompts\": len(PROMPTS), \"suffix_gpu_vs_nospec\": diff(nospec_ids, sgpu_ids), \"suffix_gpu_eager_vs_nospec\": diff(nospec_ids, sgpu_eager_ids), \"suffix_gpu_eager_vs_graph\": diff(sgpu_ids, sgpu_eager_ids), \"ngram_gpu_vs_nospec\": diff(nospec_ids, ngram_ids), \"suffix_cpu_vs_nospec\": diff(nospec_ids, scpu_ids), } result[\"identical\"] = not result[\"suffix_gpu_vs_nospec\"] print(json.dumps(result, indent=2)) with open(f\"greedy_consistency_k{K}_full.json\", \"w\") as f: json.dump( { \"result\": result, \"nospec_ids\": nospec_ids, \"suffix_gpu_ids\": sgpu_ids, \"suffix_gpu_eager_ids\": sgpu_eager_ids, \"ngram_gpu_ids\": ngram_ids, \"suffix_cpu_ids\": scpu_ids, }, f, ) ```
", "url": "https://github.com/vllm-project/vllm/pull/52097", "createdAt": "2026-08-13T04:34:29Z", "updatedAt": "2026-08-13T04:35:07Z", "timestamp": "2026-08-13T04:35:07Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "speculative-decoding" ], "author": "zip95297", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0adac514f2e144f299ad", "signalId": "github:vllm-project/vllm:pull_request:51795", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51795", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Reject negative token ids as out-of-vocabulary", "text": "## Purpose `InputProcessor._validate_model_input` validates that caller-supplied token ids are within the vocabulary, but it only checks the upper bound, so a negative token id passes validation. A token id is used as an index downstream, and a negative index is never valid input. This PR adds a symmetric lower-bound check beside the existing upper-bound check. A negative id is now rejected with the same \"out of vocabulary\" error (HTTP 400). The check sits on the shared input-validation path, so it covers generate, embedding, and pooling requests. ## Test Plan ```bash pytest -q tests/entrypoints/openai/completion/test_prompt_validation.py::test_negative_token_ids ``` ## Test Result The new test passes: a `/v1/completions` request with `prompt=[-1]` returns HTTP 400 with an \"out of vocabulary\" message (previously it passed validation). The existing `test_out_of_vocab_token_ids` (upper-bound) continues to pass. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51795", "createdAt": "2026-08-11T09:48:47Z", "updatedAt": "2026-08-13T04:34:56Z", "timestamp": "2026-08-13T04:34:56Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "bug", "ready" ], "author": "ECMGit", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:44c51dde3a2bbcb0fb61", "signalId": "github:vllm-project/vllm:pull_request:52091", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Auto-ping Cohere on related issues", "text": "## Purpose As discussed offline ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52091", "createdAt": "2026-08-13T03:37:59Z", "updatedAt": "2026-08-13T04:31:17Z", "timestamp": "2026-08-13T04:31:17Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ci/build" ], "author": "DarkLight1337", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:20c5d29f67b6cff6caa9", "signalId": "github:vllm-project/vllm:pull_request:51852", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51852", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Give CPU attention layers their own metadata when head counts differ", "text": "## Purpose The CPU backend sizes one scheduler metadata blob per KV cache group from the model-wide query head count, so models that vary it per layer (e.g. Laguna) overrun the split-KV scratchpad and either segfault or hang. Build metadata per distinct head count and let each layer select the one matching its own. Change-Id: I671eadc2b5601f1a3af12f39391657a11df1c9f7 ## Test Plan - `pytest tests/v1/attention/test_group_head_counts.py` — new unit test, mirroring `test_group_sliding_window.py`: uniform heads, Laguna-style alternating heads, and a default that is not the most common count. - A tiny Laguna config with per-layer heads `[8, 16, 8, 16]` against 2 KV heads and random weights (`load_format=dummy`, so no checkpoint is needed), decoding 512 tokens so split-KV reduction engages. Run both on unpatched `main` and with this change. - Laguna-XS-2.1-INT4 end to end on CPU with TP=2: a 300-token decode, plus GSM8K 5-shot over the full 1319 prompts via lm_eval.
Reproducer ```python import json, os, tempfile from vllm import LLM, SamplingParams CONFIG = { \"architectures\": [\"LagunaForCausalLM\"], \"model_type\": \"laguna\", \"hidden_size\": 512, \"intermediate_size\": 1024, \"num_hidden_layers\": 4, \"num_attention_heads\": 8, \"num_attention_heads_per_layer\": [8, 16, 8, 16], \"num_key_value_heads\": 2, \"head_dim\": 64, \"max_position_embeddings\": 4096, \"rms_norm_eps\": 1e-6, \"rope_theta\": 500000.0, \"tie_word_embeddings\": False, \"torch_dtype\": \"bfloat16\", \"mlp_only_layers\": [0, 1, 2, 3], \"num_experts\": 4, \"num_experts_per_tok\": 2, \"moe_intermediate_size\": 128, \"shared_expert_intermediate_size\": 128, } model_dir = tempfile.mkdtemp() with open(os.path.join(model_dir, \"config.json\"), \"w\") as f: json.dump(CONFIG, f) llm = LLM(model=model_dir, tokenizer=\"\", load_format=\"dummy\", dtype=\"bfloat16\", max_model_len=2048, enforce_eager=True) out = llm.generate([\"context \" * 64], SamplingParams(max_tokens=512, temperature=0.0, ignore_eos=True)) print(\"SURVIVED\", len(out[0].outputs[0].token_ids)) ```
## Test Result - Unit tests: 6 passed (3 new, plus the 3 existing sliding-window cases). - Reproducer: hangs on current `main` and is killed at the timeout; passes in 1.5s with this change. - Laguna-XS 300-token decode: previously segfaulted or hung, now completes in 47s with coherent output. - GSM8K 5-shot, full 1319 prompts. Before this change the same run died with an `execute_model` timeout. | Filter | exact_match | stderr | | --- | --- | --- | | flexible-extract | 0.9014 | ±0.0082 | | strict-match | 0.8908 | ±0.0086 | Instrumenting the scratchpad bounds shows the overrun directly on unpatched `main`: the region holds 6336 bytes per KV head while the split-KV write path needs 8384, because sizing assumes 6 query heads per KV head (48 heads / 8 KV heads) and the 64-head layers address it with a stride of 8. ``` [SCRATCHPAD-SIZING] split_num=2 max_tile=6 q_heads=24 kv_heads=4 q_per_kv=6 per_kv_head=6336 [SCRATCHPAD-OVERFLOW] per_head_need=8384 per_kv_head=6336 q_head_tile_size=8 ``` Models whose layers share one head count keep the existing single-blob path, so there is no extra allocation or lookup for them. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51852", "createdAt": "2026-08-11T17:18:34Z", "updatedAt": "2026-08-13T04:21:57Z", "timestamp": "2026-08-13T04:21:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "ci/build", "cpu", "verified" ], "author": "ganeshr10", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1b1400897061dd97d828", "signalId": "github:vllm-project/vllm:pull_request:43327", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43327", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Add per-call decode budget to sparse-MLA indexer", "text": "## Purpose The ROCm AITER sparse-MLA indexer decode path allocates a fresh fp32 logits tensor on every call to `rocm_fp8_paged_mqa_logits`: `(heads, B*next_n, max_model_len)` on the stage1 3D path, or `(B*next_n, max_model_len)` on the gfx942/gfx950 fused 2D path. With long context and a large running batch the per-call working set is multi-GiB; across ~60 layers per decode forward the caching allocator fragments and OOMs. Prefill had the same problem and got `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` plus chunking in #36178. This is the decode mirror. Two pieces, decode-only, ROCm-only: - the fp32 logits buffer is now drawn from a per-process workspace keyed by (device, shape) and refilled with `-inf` between calls, so no per-layer alloc/free of multi-GiB tensors; - `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` (default 4096) bounds the per-call working set. When a single call would exceed the budget the batch is split into sub-batches that fit and copied into the full output. The aiter kernel does not consume `schedule_metadata` on ROCm, so the metadata doesn't need to be re-derived per chunk. Set to 0 to restore the legacy unbounded behaviour. Prefill is not touched. The CUDA path is not touched (this file only runs on ROCm via the AITER sparse-MLA backend). Both `_ON_GFX942` and `_ON_GFX950` are covered, plus the stage1 3D else-branch that older arches still hit. Related: #36178 is the prefill-side mirror (same shape of fix, opposite forward-pass phase). #41002 is in flight in the same area, happy to rebase on top of it after it merges. ## Test Plan 1. Numeric equivalence test added at `tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py`. Skipped on non-ROCm and when the AITER `paged_mqa_logits` module is not importable, so it sits with the rest of the ROCm-only kernel tests. On a ROCm gate it asserts the chunked path returns bit-identical logits to the full-batch call. ```bash pytest -s -v tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py ``` 2. Synthetic micro-bench on a single MI355X (gfx950), 60 back-to-back calls into `rocm_fp8_paged_mqa_logits` with a pre-allocated paged kv at `B=1024, next_n=1, heads=64, head_dim=128, block_size=64, max_model_len=131072`. Stress shape, picked so the 3D per-call working set is well above any reasonable budget. Measured peak `torch.cuda.max_memory_allocated()` minus the paged-kv baseline, sweeping `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` against the legacy code. 3. End-to-end re-run of the GLM-5.1-FP8 customer repro that motivated this: ```bash vllm bench serve --random-input-len 1 --random-output-len 2 --num-prompts 1024 \\ --ignore-eos --temperature 0 --seed 0 \\ --gpu-memory-utilization 0.9 -tp 8 --enforce-eager ``` ## Test Result Indexer working set at the high-water mark, 60-layer loop: | setting | indexer working set @ peak | | -------------------------------------------------- | -------------------------: | | legacy (no patch) | 33792 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` unset (= 4096) | ~4160 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=2048` | ~2080 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=1024` | ~1040 MiB | The 33.8 GiB legacy delta is exactly `heads * B * next_n * max_model_len * 4` bytes for the 3D stage1 buffer; chunking brings it down to `chunk_b * heads * next_n * max_model_len * 4` plus a one-shot 2D accumulator. Wall time across all four settings stays between 1.09 and 1.24 s for the 60-call loop, the only overhead is one `copy_` per chunk into the persistent output buffer. End-to-end on GLM-5.1-FP8, the decode-time HIP OOM that originally fired at ~24 GiB per layer no longer occurs; 1017/1024 prompts succeed, the 7 remaining failures are bench-client side (\"never received a valid chunk\") and clear with `--max-concurrency 256`.", "url": "https://github.com/vllm-project/vllm/pull/43327", "createdAt": "2026-05-21T15:29:18Z", "updatedAt": "2026-08-13T04:19:36Z", "timestamp": "2026-08-13T04:19:36Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "rocm", "needs-rebase", "v1" ], "author": "reger-men", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b60c669bc515683189e2", "signalId": "github:vllm-project/vllm:issue:51758", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51758", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: upgrade vllm from 0.26.0 to 0.27.0 run deepseek v4 flash error", "text": "### Your current environment
The output of python collect_env.py ```text python collect_env.py Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : 18.1.3 (1ubuntu1) CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by conda-forge | (main, Mar 5 2026, 16:50:00) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-6.8.0-137-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.3.73 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 1: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 2: NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 3: NVIDIA RTX PRO 6000 Blackwell Server Edition Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8488C CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 21% CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user Virtualization: VT-x L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 192 MiB (96 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchcodec==0.15.0 [pip3] torchvision==0.28.0 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] flashinfer-python 0.6.16.post3 pypi_0 pypi [conda] nccl4py 0.3.1 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.1.3 pypi_0 pypi [conda] nvidia-cuda-cccl 13.3.3.4.1 pypi_0 pypi [conda] nvidia-cuda-crt 13.3.73 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvcc 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvdisasm 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.20.0.48 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.27.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-core 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu12 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu13 4.6.0 pypi_0 pypi [conda] nvidia-ml-py 13.610.43 pypi_0 pypi [conda] nvidia-nccl-cu13 2.29.7 pypi_0 pypi [conda] nvidia-nvjitlink 13.3.33 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] nvidia-nvvm 13.3.73 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] tokenspeed-triton 3.8.10.post20260721 pypi_0 pypi [conda] torch 2.13.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchcodec 0.15.0 pypi_0 pypi [conda] torchvision 0.28.0 pypi_0 pypi [conda] transformers 5.15.0 pypi_0 pypi [conda] triton 3.7.1 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.0 vLLM Build Flags: CUDA Archs: Blackwell; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE NODE 48-95,144-191 1 N/A GPU1 NODE X NODE NODE 48-95,144-191 1 N/A GPU2 NODE NODE X NODE 48-95,144-191 1 N/A GPU3 NODE NODE NODE X 48-95,144-191 1 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== TORCH_CUDA_ARCH_LIST=Blackwell LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda-12.8/lib64 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_yafeng_wang ```
### 🐛 Describe the bug [bugreport.txt](https://github.com/user-attachments/files/30921135/bugreport.txt) ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51758", "createdAt": "2026-08-11T03:42:26Z", "updatedAt": "2026-08-13T04:16:50Z", "timestamp": "2026-08-13T04:16:50Z", "metrics": { "reactions": 0, "comments": 17 }, "labels": [ "bug" ], "author": "WangHHY19931001", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d1de0b2392fb7d1f5c95", "signalId": "github:vllm-project/vllm:pull_request:52093", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52093", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Perf] Narrow DeepSeek V4 eager CUDA graph region\" (#51430)", "text": "## Summary - Revert [#51430](https://github.com/vllm-project/vllm/pull/51430) and restore the wider DeepSeek-V4 eager attention region. - Preserve the later ROCm `_fused_wqa_wkv_gemm` override point while resolving the revert against current main. - This is the prerequisite for safely restoring MRV1 piecewise graphs in the stacked follow-up. ## Why On gfx950 TP8, the narrowed eager region corrupts output under the MRV1 piecewise graph path. MRV2 avoids that corruption but regresses 8k/1k concurrency-4 throughput. Restoring the wider eager region makes the legacy path correct and recovers the attention performance needed by the follow-up runner revert. This revisits closed PR [#51750](https://github.com/vllm-project/vllm/pull/51750) with new ROCm correctness and performance measurements. No open PR currently reverts [#51430](https://github.com/vllm-project/vllm/pull/51430). ## Test plan Configuration: DeepSeek-V4-Pro, gfx950, TP=8, DP=1, prefix caching disabled, `max_num_batched_tokens=8192`, AITER MoE, FP8 KV cache, `FULL_AND_PIECEWISE` graphs. Combined with the stacked [#51768](https://github.com/vllm-project/vllm/pull/51768) revert: - Correctness smoke: `1 + 1` returned `2`. - InferenceX 8k/1k, concurrency 4: 40/40 successful requests. - Output throughput: **185.66 tok/s**. - Mean TPOT: **20.53 ms**. - Median ITL: **19.47 ms**. - Mean TTFT: **466.34 ms**. - Historical reference: 186.99 tok/s, 20.36 ms TPOT, 19.23 ms median ITL. Pre-commit hooks run during commit passed, including Ruff, formatting, mypy, SPDX, and configuration validation. ## Contribution notes - Duplicate check: [#51750](https://github.com/vllm-project/vllm/pull/51750) is closed; no open PR reverting [#51430](https://github.com/vllm-project/vllm/pull/51430) was found. - AI assistance was used for regression isolation, implementation, testing, and PR preparation. - The human submitter must review every changed line and independently validate the results before merge.", "url": "https://github.com/vllm-project/vllm/pull/52093", "createdAt": "2026-08-13T04:06:16Z", "updatedAt": "2026-08-13T04:15:31Z", "timestamp": "2026-08-13T04:15:31Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "deepseek", "nvidia" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:54f461b611b6a336566a", "signalId": "github:vllm-project/vllm:pull_request:51406", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51406", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Enable fused QK-norm+RoPE+gate Triton kernel for Qwen3-Next/Qwen3.5", "text": "## Purpose `Qwen3NextAttention` fuses the gated QKV split + QK-RMSNorm + (partial) NeoX RoPE + gate copy into a single Triton kernel (`fused_qk_rmsnorm_rope_gate`, added in https://github.com/vllm-project/vllm/pull/44176). The kernel is pure Triton and backend-agnostic, but it is gated behind `current_platform.is_cuda()`, so ROCm falls back to the slow multi-op eager path, even though the kernel runs correctly and fast on AMD GPUs. This PR relaxes that one-line guard to `is_cuda_alike()` so ROCm gets the same fusion. It affects the whole `Qwen3NextAttention` family: Qwen3-Next, Qwen3.5, Qwen3.5-MoE Measured on MI325X (gfx942), bf16, against the model's real eager branch (`GemmaRMSNorm` + the `rotary_emb` custom op), 300 timed iters after 50 warmup: | Seq len (tokens) | Eager (µs) | Fused (µs) | Speedup | |---|---|---|---| | 1 (decode) | 279.4 | 33.0 | 8.47× | | 8 | 279.3 | 33.6 | 8.32× | | 64 | 281.7 | 33.9 | 8.32× | | 512 | 277.4 | 33.7 | 8.22× | | 4096 | 562.3 | 138.7 | 4.05× | Note: https://github.com/vllm-project/vllm/pull/47719 is an opt-in AITER path that additionally fuses the KV-cache write. ## Test Plan Add a test case in `tests/kernels/core/test_fused_qk_norm_rope.py` ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51406", "createdAt": "2026-08-07T15:04:43Z", "updatedAt": "2026-08-13T04:09:32Z", "timestamp": "2026-08-13T04:09:32Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm", "qwen" ], "author": "xuebwang-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9597eeccdc301397cbe8", "signalId": "github:vllm-project/vllm:pull_request:51915", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51915", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness", "text": "## Purpose Enables GLM-5.2 (`GlmMoeDsaForCausalLM`) end-to-end on `vllm/models/deepseek_v32/amd/` for gfx942/gfx950. Routing is opt-in via `--model-class-overrides`; the registry entry is unchanged, so the default path for GLM-5.2 and DeepSeek-V3.2 is untouched. The following issues were also fixed as a result of this overall enablement since it surfaced dormant bugs from a prior deepseek_v32 porting work: - **`launch_pdl` forwarded to a Triton runtime that doesn't accept it** — the shared `common/kernels.py` passed this CUDA-only kwarg unconditionally; ROCm's Triton rejects it by name even when `False`, so the engine died at startup with `KeyError`. Now passed only when the platform actually supports PDL. - **fp4 bmm invoked with mismatched positional args** — the output tensor landed in the kernel's `dtype` slot (`Expected 'Optional[int]' for 'dtype' but found Tensor`). Routed through the existing `rocm_aiter_ops.batched_gemm_a16wfp4` wrapper rather than calling aiter directly, so the argument contract lives in one place. - **fp4 MLA bmm silently wrong for non-power-of-two K** — `batched_gemm_a16wfp4` only supports power-of-two K, and returns wrong values without erroring otherwise. GLM-5.2's `qk_nope_head_dim=192` lands in the unsupported set; DeepSeek-V3.2's 128 does not. Guarded on both `qk_nope_head_dim` and `kv_lora_rank` to fall back to the fp8 bmm, with a warning. `W_K`/`W_K_scale` are also made contiguous after the transpose, which otherwise produced NaN. - **Indexer-K cache written in the plain layout while its reader expects the shuffled one** — affects DeepSeek-V3.2 identically, not just GLM-5.2. The AMD path writes this cache from the shared `fused_norm_rope`, which only implemented `[n_blocks, blk, head_dim]`, while the reader selects `SHUFFLE` above block size 1: the write kept storing row-major, but the reader expected shuffled, so the indexer scored garbage K and output degenerated once a sequence exceeded `index_topk=2048`, with no error raised. Refactored so the shared kernel stays platform-agnostic — it writes whichever layout the cache reports via a new `uses_shuffled_layout` property — and the AMD-specific shuffled write (`[n_blocks, blk/16, head_dim/16, 16, 16]`, used above block size 1) lives in the AMD backend. The base class returns `False`, so CUDA is unchanged. - **Sparse-MLA block sizes narrowed instead of extended** — the ported backends replaced the inherited `get_supported_kernel_block_sizes()` with `[16, 32]`, which made `select_common_block_size` silently downgrade a requested 64 to 32 via its largest-divisor fallback. Now concatenated with the base list. - **Kernel tests gated on `is_cuda()` and running against dims that matched neither model** — the tests were skipped entirely on CDNA. The gate is now `supports_fp8()`, so the suite runs on gfx950. The hardcoded attention dims were written as if one set served both GLM-5.2 and DeepSeek-V3.2, which it did not; they are now taken from each model's published config and parametrized over both. ## Test Plan Server: VLLM_ROCM_USE_AITER=1 \\ HF_HUB_OFFLINE=1 SAFETENSORS_FAST_GPU=1 \\ vllm serve /path-to-your/GLM-5.2-MXFP4 \\ --model-class-overrides '{\"GlmMoeDsaForCausalLM\": \"vllm.models.deepseek_v32:DeepseekV32ForCausalLM\"}' \\ --tensor-parallel-size 8 \\ --kv-cache-dtype auto \\ --max-model-len 65536 \\ --no-enable-prefix-caching \\ --trust-remote-code \\ --port 8000 Variants: --kv-cache-dtype fp8 for the fp8 KV row; --block-size {16,64} for the block-size matrix. Drop --model-class-overrides to run the legacy deepseek_v2 path for comparison. Accuracy: lm_eval --model local-completions \\ --model_args \"model=/models/GLM-5.2-MXFP4,base_url=http://localhost:8000/v1/completions,num_concurrent=32,tokenized_requests=False,max_retries=3,timeout=3600\" \\ --tasks gsm8k --num_fewshot 5 --gen_kwargs max_gen_toks=2048 --limit 200 Throughput: vllm bench serve --backend vllm --model /shareddata/models/GLM-5.2-MXFP4 \\ --dataset-name random --random-input-len 4096 --random-output-len 1024 \\ --num-prompts 200 --max-concurrency 32 --ignore-eos --port 8000 \\ --save-result --result-filename glm52_v32amd_bench.json Kernel unit tests: pytest tests/kernels/test_fused_deepseek_v32_norm_rope.py -v ## Test Result **gsm8k** — GLM-5.2 200 samples, 5-shot, `exact_match`, `deepseek_v32` AMD path | KV cache | Before (flexible / strict) | After (flexible / strict) | |-------------|----------------------------|------------------------------| | auto (bf16) | 0.0000 / 0.0000 | **0.9400 / 0.9400** ± 0.0168 | | fp8 | 0.0000 / 0.0000 \\* | **0.9350 / 0.9350** ± 0.0175 | \\* Not separately measured. fp8 KV goes through the same absorbed-weight bmm where the bug was, so it fails identically; only the bf16 run was evaluated pre-fix.", "url": "https://github.com/vllm-project/vllm/pull/51915", "createdAt": "2026-08-12T02:59:15Z", "updatedAt": "2026-08-13T04:08:44Z", "timestamp": "2026-08-13T04:08:44Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "bug", "rocm", "needs-rebase", "deepseek" ], "author": "jhu960213", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3322928c37526afe9374", "signalId": "github:vllm-project/vllm:pull_request:52094", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52094", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Bugfix] Guard DeepSeek V4 MRV1 piecewise CUDA graphs\" (#51768)", "text": "## Summary - Restore Model Runner V1 as the DeepSeek-V4 default. - Remove the MRV1 piecewise-graph rejection added by [#51768](https://github.com/vllm-project/vllm/pull/51768). - Stack this change on [#52093](https://github.com/vllm-project/vllm/pull/52093), which first restores the wider eager attention region required for correctness. ## Dependency Stacked on [#52093](https://github.com/vllm-project/vllm/pull/52093). Do not merge this PR first: reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) alone would re-enable the known-broken MRV1 piecewise path while [#51430](https://github.com/vllm-project/vllm/pull/51430) remains active. Until [#52093](https://github.com/vllm-project/vllm/pull/52093) merges, this PR intentionally contains both revert commits; afterward its diff reduces to the config/test revert. ## Why On gfx950 TP8, MRV2 preserved correctness but regressed steady decode performance. With the prerequisite eager-region revert, MRV1 piecewise execution is correct and restores long-context throughput. No open PR reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) was found. ## Test plan Configuration: DeepSeek-V4-Pro, gfx950, TP=8, DP=1, prefix caching disabled, `max_num_batched_tokens=8192`, AITER MoE, FP8 KV cache, `FULL_AND_PIECEWISE` graphs. | Configuration | Output throughput | Mean TPOT | Median ITL | | --- | ---: | ---: | ---: | | MRV2 before reverts | 154.81 tok/s | 24.70 ms | 23.65 ms | | MRV1 + [#52093](https://github.com/vllm-project/vllm/pull/52093) | **185.66 tok/s** | **20.53 ms** | **19.47 ms** | | Historical reference | 186.99 tok/s | 20.36 ms | 19.23 ms | Additional results: - 40/40 successful InferenceX 8k/1k concurrency-4 requests. - Mean TTFT: 466.34 ms. - Correctness smoke: `1 + 1` returned `2`. - Two selective ROCm runs measured 185.16 and 184.94 tok/s, corroborating the rollback result. Pre-commit hooks run during commit passed, including Ruff, formatting, mypy, SPDX, and configuration validation. ## Contribution notes - Duplicate check: no open PR reverting [#51768](https://github.com/vllm-project/vllm/pull/51768) was found. - AI assistance was used for regression isolation, implementation, testing, and PR preparation. - The human submitter must review every changed line and independently validate the results before merge.", "url": "https://github.com/vllm-project/vllm/pull/52094", "createdAt": "2026-08-13T04:07:21Z", "updatedAt": "2026-08-13T04:08:00Z", "timestamp": "2026-08-13T04:08:00Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "deepseek", "nvidia" ], "author": "Fangzhou-Ai", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1931712a8a214ffb9f26", "signalId": "github:vllm-project/vllm:pull_request:40704", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:40704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunner V2] Speculative Decoding NGram GPU Implementations", "text": "## Purpose 1. Added a new NGram GPU speculator. The main feature is a new implementation at: `vllm/v1/worker/gpu/spec_decode/ngram/speculator.py`, similiar to https://github.com/vllm-project/vllm/pull/29184. 2. Updated request state storage for NGram GPU. `vllm/v1/worker/gpu/states.py` `vllm/v1/worker/gpu/model_runner.py` This changes how RequestState is initialized so that all_token_ids can stay densely resident on GPU instead of defaulting to UVA when ngram_gpu is active. As discussed in #29184, the new n-gram speculator repeatedly scans active request token history, doing that from GPU-resident dense storage is much more appropriate than pulling through UVA-backed memory, this is a performance-oriented architectural change supporting the new feature. The `model_runner.py` also injects req_states into speculators that need direct access to the persistent token store. 3. Added variable-length draft token plumbing Several files were updated to support draft proposals where different requests may have different numbers of valid draft tokens: `vllm/v1/outputs.py` `vllm/v1/worker/gpu/spec_decode/utils.py` `vllm/v1/core/sched/scheduler.py` `vllm/v1/engine/core.py` `vllm/v1/worker/gpu/model_runner.py` DraftTokenIds now includes: `num_valid_draft_tokens: list[int] | None`. Scheduler logic now truncates speculative tokens based on `num_valid_draft_tokens`. `EngineCore adds _maybe_update_async_draft_token_ids()` to consume draft metadata from async execution and update scheduler state at the right time. ## Test Plan vllm bench cmd: ``` vllm bench serve \\ --port 8000 \\ --backend vllm \\ --model Qwen3-8B \\ --endpoint /v1/completions \\ --dataset-name sonnet \\ --dataset-path sonnet.txt \\ --max-concurrency 128 \\ --sonnet-input-len 128 \\ --sonnet-output-len 100 \\ --sonnet-prefix-len 10 \\ --num-prompts 256 \\ --ignore-eos \\ --percentile-metrics \"ttft,tpot,itl,e2el\" \\ --seed 1234 ``` ## Test Result Async NGram GPU V1 results: ``` ============ Serving Benchmark Result ============ Successful requests: 256 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 10.42 Total input tokens: 31693 Total generated tokens: 25600 Request throughput (req/s): 24.56 Output token throughput (tok/s): 2456.33 Peak output token throughput (tok/s): 4277.00 Peak concurrent requests: 186.00 Total token throughput (tok/s): 5497.29 ---------------Time to First Token---------------- Mean TTFT (ms): 1348.49 Median TTFT (ms): 1083.36 P99 TTFT (ms): 2778.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 34.86 Median TPOT (ms): 34.45 P99 TPOT (ms): 51.03 ---------------Inter-token Latency---------------- Mean ITL (ms): 40.09 Median ITL (ms): 30.44 P99 ITL (ms): 236.84 ----------------End-to-end Latency---------------- Mean E2EL (ms): 4799.33 Median E2EL (ms): 4419.59 P99 E2EL (ms): 7136.50 ---------------Speculative Decoding--------------- Acceptance rate (%): 5.16 Acceptance length: 1.15 Drafts: 22038 Draft tokens: 66114 Accepted tokens: 3409 Per-position acceptance (%): Position 0: 6.27 Position 1: 5.34 Position 2: 3.86 ================================================== ``` Async NGram GPU V2 results: ``` ============ Serving Benchmark Result ============ Successful requests: 256 Failed requests: 0 Maximum request concurrency: 128 Benchmark duration (s): 8.41 Total input tokens: 31693 Total generated tokens: 25600 Request throughput (req/s): 30.43 Output token throughput (tok/s): 3042.78 Peak output token throughput (tok/s): 4471.00 Peak concurrent requests: 187.00 Total token throughput (tok/s): 6809.77 ---------------Time to First Token---------------- Mean TTFT (ms): 980.36 Median TTFT (ms): 713.87 P99 TTFT (ms): 1685.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 31.17 Median TPOT (ms): 31.34 P99 TPOT (ms): 42.63 ---------------Inter-token Latency---------------- Mean ITL (ms): 35.77 Median ITL (ms): 28.55 P99 ITL (ms): 154.67 ----------------End-to-end Latency---------------- Mean E2EL (ms): 4065.85 Median E2EL (ms): 3954.04 P99 E2EL (ms): 5862.69 ---------------Speculative Decoding--------------- Acceptance rate (%): 52.64 Acceptance length: 2.58 Drafts: 2136 Draft tokens: 6408 Accepted tokens: 3373 Per-position acceptance (%): Position 0: 64.04 Position 1: 54.63 Position 2: 39.23 ================================================== ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/40704", "createdAt": "2026-04-23T12:42:36Z", "updatedAt": "2026-08-13T03:58:56Z", "timestamp": "2026-08-13T03:58:56Z", "metrics": { "reactions": 3, "comments": 21 }, "labels": [ "speculative-decoding", "ready", "ci/build", "v1", "mrv2" ], "author": "PatchouliTIS", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b28c6089174b928bf07b", "signalId": "github:vllm-project/vllm:pull_request:50021", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50021", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Bound accepted-token state lookups in GDN/KDA spec decode", "text": "## Summary On a hybrid GDN model (Qwen3.5 / Qwen3.6) with MTP speculative decoding and prefix caching (`--mamba-cache-mode align`), the engine dies with `CUDA error: unspecified launch failure` within 7-10 requests of agent-shaped traffic. The GPU faults, not the runtime: the signature is an SM address exception (`Xid 13`, `ESR 0x404000`) or an MMU fault (`Xid 31`), depending on whether the wild address happens to be mapped. This is the crash half of the problems reported around hybrid-Mamba + MTP. It is distinct from the prefix-cache corruption in #43559 (the coordinator-level EAGLE cache-peek gating for Mamba), which is already handled on current `main`. This PR does not touch that path and does not claim to fix #43559; it fixes a separate GPU fault in two kernels downstream. ## Root cause: unchecked accepted-count-derived indices Speculative decoding produces a per-request accepted-token count. Multiple GPU state consumers turn that count into an array index without bounding it, so a count that is stale, zero, or too large indexes outside its tensor and yields a wild address the GPU then dereferences. **Site 1** `fused_recurrent_gated_delta_rule_fwd_kernel` (`fused_recurrent.py`): ```python i_t = tl.load(num_accepted_tokens + i_n).to(tl.int64) - 1 # unbounded state_idx = tl.load(ssm_state_indices + i_n * stride_indices_seq + i_t) if state_idx <= 0: return # positive garbage passes p_h0 = h0 + state_idx * stride_init_state_token # dereferenced ``` `i_t` (= count - 1) is unbounded against a `stride_indices_seq`-column tensor. A zero accepted count gives `i_t == -1`, a read before this request's row (before the tensor for `i_n == 0`); a stale or too-large count reads past the row. The `state_idx <= 0` guard only rejects non-positive values, so an out-of-range read that returns a garbage positive int flows into the address math and faults the SM. **Site 2** `_copy_mamba_state_block` (`mamba_utils.py`): the block-table columns derive from the same count and index the per-request block-table row unbounded; the loaded block id becomes `state_base_addr + block_id * state_block_stride`, which is then read and written. The two sites fire under different loads. A light decode load exercises only Site 1; a heavy cache-transition load (long prefills, A->B->A prefix reuse) also drives Site 2. Both must be bounded. ## The fix Both loads are masked to the valid range, so an out-of-range index falls into the existing invalid-state path instead of producing an address. No stream-ordering change, no device sync, no measurable throughput cost. ## Test plan RTX 5090 (sm_120), Qwen3.6-27B NVFP4, TP=1, `--enforce-eager --max-model-len 16384`, MTP 3 (`qwen3_5_mtp`), prefix caching on, `align` mode. | build | crash (Xid) | corruption (A->B->A probe) | |---|---|---| | stock | dies at 7-10 requests, Xid every run | 0/N reproduced (already fixed on main) | | Site 1 fix alone | survives light load; still crashes on the first heavy probe (`Xid 31`) | 0/N reproduced | | **both fixes** | **68 heavy probes + a 39-minute soak, 0 new Xid** | **0/68 reproduced** | The A->B->A probe issues an agent-shaped sequence (long prefill, prefix reuse, 20 tool schemas) and reports whether a poisoned prefix reappears; 68/68 returned a clean verdict, which also confirms the out-of-range early-return does not drop a needed state copy. Warm throughput and MTP acceptance length (3.98-4.00 of 4) are unchanged. ## Note: a separate, still-open livelock (#49203) Independently of this crash, the same stack can occasionally hang: engine alive, `/v1/models` answering, but the in-flight request stuck at 0% GPU util with no Xid. It is rare and timing-variable (seen once, then not across the 68 probes here) and matches open issue #49203. This PR does not address it and does not claim to; the 68/68 clean-verdict result rules out this change as a cause. ## Follow-up bounds audit A follow-up audit expanded the same fail-closed rule to the remaining consumers in this path: - Both FLA wrappers now zero rejected output deterministically instead of returning with new_empty storage visible downstream. - _causal_conv1d_update_kernel now bounds the accepted-count offset before state address math; invalid active rows produce zero output and leave state unchanged. - GPU regressions cover too-small/too-large accepted counts, NULL block IDs, source/destination columns crossing the block-table row, and temporal-bias overflow. Validation after this follow-up: full pre-commit passed; on an RTX 5090, 186 FLA/causal-conv kernel tests and all 21 fused Mamba postprocess tests passed (207 total). ## Kimi K3 KDA expansion The same accepted-count-derived state selection pattern also existed in Kimi K3 KDA fused recurrent decode, in both the NVIDIA and AMD vendored kernels. This PR now masks that initial state-index load to the request row, zeroes all invalid-count output tokens, preserves state on invalid counts, and releases NVIDIA PDL dependents before the new empty/invalid early returns. Additional validation on mini-beast RTX 5090: - Patched KDA invalid-count test: 4/4 passed across NVIDIA and AMD implementations (`num_accepted` 0 and 4). - Existing KDA spec-decode correctness plus the new invalid-count target: 12/12 passed. - Negative control at previous PR head `e7f66b199` with only the new test added: 4/4 failed on old KDA for NVIDIA and AMD, proving the regression is non-vacuous. ## AI assistance disclosure OpenAI Codex assisted with the follow-up bounds audit, implementation, and regression-test generation. The submitter owns the conclusions and final review. ## Mamba2 selective-state expansion The Mamba2 selective-state-update kernel also derives its initial state-slot lookup from `num_accepted_tokens - 1`. It already clamps the lower side, but had no upper row bound. This update preserves that zero-count behavior and makes an oversized count fail closed: it writes zero output and returns before reading or writing state. RTX 5090 red/green regression: a count one past a three-column state row made the prior source select an adjacent row and emit nonzero output; the new kernel passes by emitting zero output and preserving the complete state tensor. Pre-commit passes for both changed files.", "url": "https://github.com/vllm-project/vllm/pull/50021", "createdAt": "2026-07-27T17:29:26Z", "updatedAt": "2026-08-13T03:58:49Z", "timestamp": "2026-08-13T03:58:49Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "bug", "v1", "nvidia", "kimi", "k3" ], "author": "amittell", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:df59eb951ccac79bdbb8", "signalId": "github:vllm-project/vllm:pull_request:51157", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51157", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Pad SigLIP text prompts to the trained sequence length", "text": "## Purpose SigLIP is trained with `padding=\"max_length\"` and without an attention mask, so padding tokens are part of the input and the pooled embedding is taken from the last position. Text embeddings computed without padding are not aligned with the image embeddings, which makes image-text similarity unusable. The existing test only passes because it opts into padding explicitly: ```python # tests/models/multimodal/pooling/test_siglip.py tokenization_kwargs={ \"padding\": \"max_length\", \"max_length\": 64, }, # siglip2 was trained with this padding setting. ``` Callers that cannot pass tokenization kwargs get silently wrong results. The request still returns 200 with a correctly shaped vector; only the values are wrong, so the failure is easy to miss in production. Two entry points need covering, and they take different routes: | entry point | source of `TokenizeParams` | inherits model defaults | |---|---|---| | `LLM.embed` | `renderer.default_cmpl_tok_params` | yes | | `/v1/embeddings` (text and chat) | `request.build_tok_params()` | **no** | `entrypoints/pooling/base/protocol.py:97-116` builds `TokenizeParams` from scratch, so a model-level default alone does not reach the OpenAI-compatible server. Note that padding text-only prompts inside `SiglipMultiModalProcessor` does **not** work: `inputs/preprocess.py:169-181` routes text-only prompts straight to `_tokenize_prompt` and never reaches the multi-modal processor. I verified this by instrumenting `apply()` — during the text test it is only ever called with dummy profiling data. ## Fix Two small pieces. **1. The model declares its requirement** through the existing `BaseProcessingInfo.get_default_tok_params` extension point, which ten models already override (`paligemma`, `gemma4_mm`, `ovis`, `ovis2_5`, `whisper`, `ultravox`, `cohere_asr`, `lfm2_vl`, `nano_nemotron_vl`, `nemotron_parse`): ```python def get_default_tok_params(self) -> \"TokenizeParams\": tok_params = super().get_default_tok_params() return tok_params.with_kwargs(pad_prompt_tokens=self.get_text_max_length()) ``` **2. The renderer applies that default** when a frontend builds `TokenizeParams` without one, in `_tokenize_singleton_prompt` and its async twin: ```python if params.pad_prompt_tokens is not None or prompt.get(\"multi_modal_data\"): return params ``` Both guards matter. Explicitly requested padding wins. Multi-modal prompts are skipped because `renderers/base.py:786` tokenizes the prompt *before* multi-modal processing on the online path, so without the guard image requests would be padded too. The padding machinery itself already exists (`TokenizeParams.pad_prompt_tokens`, `_token_padding`); this only turns it on for a model that needs it. Models that declare nothing get `None` and the helper returns the params unchanged. ### Known trade-off A caller who deliberately passes `padding=False` for SigLIP will still get the model default reapplied, so there is no opt-out. Note this is not caused by the fallback alone. `TokenizeParams.with_kwargs` guards the padding mapping with a truthiness check: ```python if padding := tokenization_kwargs.pop(\"padding\", None): if padding == \"max_length\": pad_prompt_tokens = max_length elif padding in (False, \"do_not_pad\"): # unreachable pad_prompt_tokens = None ``` `padding=False` is falsy, so the key is popped and the `elif` never runs — `padding=False` is already a no-op on `main` for any model whose default sets `pad_prompt_tokens`. Consequently \"disabled\" and \"unspecified\" are indistinguishable to the fallback, and it reapplies the default. I judged that acceptable because unpadded SigLIP is simply incorrect, but I am happy to add an explicit sentinel if maintainers prefer to keep the opt-out. Fixing the dead `elif` branch itself looks like a separate change; happy to send one if that is wanted. ### Alternative considered Setting `pad_prompt_tokens` inside `_build_pooling_tok_params` would avoid touching the renderer, but it only fixes frontends that build `TokenizeParams` and puts a model-specific rule in the entrypoints layer. Happy to move the fix there if preferred. ## Test plan `test_models_text` is parametrized over `explicit_padding`. Both cases compare against the same HF reference computed with padding, so `explicit_padding=False` fails on `main`. `tests/renderers/test_default_padding.py` covers the server route by building `TokenizeParams` exactly the way the pooling endpoints do (no padding field). It runs on CPU in seconds. ```bash .venv/bin/python -m pytest tests/models/multimodal/pooling/test_siglip.py -v .venv/bin/python -m pytest tests/renderers/ -q ``` ## Test result All three models in `MODELS`, on an NVIDIA GB10 (sm_121, aarch64, CUDA 13.0): ``` $ .venv/bin/python -m pytest tests/models/multimodal/pooling/test_siglip.py \\ tests/renderers/test_default_padding.py -v test_models_text[True-float-google/siglip-base-patch16-224] PASSED test_models_text[True-float-google/siglip2-base-patch16-224] PASSED test_models_text[True-float-google/siglip2-giant-opt-patch16-384] PASSED test_models_text[False-float-google/siglip-base-patch16-224] PASSED test_models_text[False-float-google/siglip2-base-patch16-224] PASSED test_models_text[False-float-google/siglip2-giant-opt-patch16-384] PASSED test_models_image[float-google/siglip-base-patch16-224] PASSED test_models_image[float-google/siglip2-base-patch16-224] PASSED test_models_image[float-google/siglip2-giant-opt-patch16-384] PASSED test_models_text_image_no_crash[float-google/siglip-base-patch16-224] PASSED test_models_text_image_no_crash[float-google/siglip2-base-patch16-224] PASSED test_models_text_image_no_crash[float-google/siglip2-giant-opt-patch16-384] PASSED test_default_padding::test_model_default_is_applied PASSED test_default_padding::test_explicit_padding_is_not_overridden PASSED test_default_padding::test_multimodal_prompts_are_not_padded PASSED 15 passed in 2085.08s (0:34:45) ``` The same suite passes on an RTX 4070 SUPER (sm_89, x86_64, CUDA 12.8, vLLM built from source) for the two base models — 8 passed, giant skipped there for disk reasons. Full renderer suite on that machine, to check the shared `renderers/base.py` change against every other model: ``` $ .venv/bin/python -m pytest tests/renderers/ -q 1 failed, 428 passed in 469.25s ``` The single failure is `test_hf.py::test_resolve_content_format_fallbacks[facebook/chameleon-7b-string]`, which fails identically on unpatched `main` — unrelated to this change. ### Reverse check Reverting only the changed source files and rerunning: | test | unpatched `main` | with this change | |---|---|---| | `test_models_text[False-...siglip-base-patch16-224]` | **FAIL**, cosine 0.5006 vs HF | PASS | | `test_models_text[True-...siglip-base-patch16-224]` | PASS | PASS | | `test_default_padding::test_model_default_is_applied` | **FAIL**, `assert 9 == 64` | PASS | | `test_default_padding::test_explicit_padding_is_not_overridden` | PASS | PASS | | `test_default_padding::test_multimodal_prompts_are_not_padded` | PASS | PASS | ### Lint `pre-commit run --files ` passes, including `mypy` for Python 3.10 and `mypy-3.12 --hook-stage manual`. ## Model evaluation Zero-shot classification through `LLM.embed` on `google/siglip-base-patch16-224`, two images from `vllm.assets.image` against three labels. Cosine similarity, argmax picks the label. Before (unpatched `main`): | image | \"a photo of a stop sign\" | \"a photo of a cherry blossom\" | \"a scanned tax invoice\" | picked | |---|---|---|---|---| | stop sign | −0.0245 | −0.0166 | −0.0422 | cherry blossom (wrong) | | cherry blossom | −0.0684 | −0.0348 | −0.0488 | cherry blossom | After: | image | \"a photo of a stop sign\" | \"a photo of a cherry blossom\" | \"a scanned tax invoice\" | picked | |---|---|---|---|---| | stop sign | **0.1567** | 0.0114 | −0.0974 | stop sign | | cherry blossom | −0.0089 | **0.1033** | −0.0942 | cherry blossom | The accuracy change (1/2 to 2/2) understates it: before the change every similarity is negative and near zero, so there is no signal at all and the one correct pick is accidental. SigLIP's `logit_scale` (~118) and `logit_bias` (~−12.7) put the 0.5 probability threshold at a cosine of about 0.107, which only the patched matching pairs clear. Image-image cosine is 0.5879 in both runs, so the vision tower is untouched and only the cross-modal alignment changed. ## Duplicate check Checked 2026-08-05, no overlapping work found: ```bash gh pr list --repo vllm-project/vllm --state open --search \"siglip\" gh issue list --repo vllm-project/vllm --state open --search \"siglip padding\" ``` Open PRs mentioning siglip are unrelated (MiniCPM-RobotTrack, a Gemma 4 fp16 overflow fix, PaddleOCR-VL CUDA graph support, DeepSeek-VL2). Open issues mentioning siglip are unrelated. \\#29794 added `tokenization_kwargs` to the offline API and is already merged; this PR covers the case where the caller cannot pass them. ## AI assistance This change was prepared with AI assistance. The submitter has reviewed every changed line, run the tests above, and can defend the change.", "url": "https://github.com/vllm-project/vllm/pull/51157", "createdAt": "2026-08-05T12:12:34Z", "updatedAt": "2026-08-13T03:58:03Z", "timestamp": "2026-08-13T03:58:03Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "bug", "multi-modality" ], "author": "Hert4", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:56249f74dd7acba23f4d", "signalId": "github:vllm-project/vllm:pull_request:49540", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49540", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MM] Profile the multimodal embedding merge during memory profiling", "text": "Automated draft from oss-work.", "url": "https://github.com/vllm-project/vllm/pull/49540", "createdAt": "2026-07-23T06:59:19Z", "updatedAt": "2026-08-13T03:54:51Z", "timestamp": "2026-08-13T03:54:51Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "v1", "qwen", "nvidia", "mrv2" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:54e380b178adec60e2d4", "signalId": "github:vllm-project/vllm:pull_request:49539", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49539", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Quantization] Keep unquantized routed experts unquantized under AutoGPTQ", "text": "# Issue #49141 — Fused_moe dimension mismatch for Qwen MXFP4 model on ROCm ## 1. Root cause The issue has two reports. Both are the same underlying defect: **a MoE layer whose expert weights are stored *unquantized* in the checkpoint is still given a *quantized* MoE method**, so the allocated parameters do not match the checkpoint tensors. The trigger in both cases is the MTP (speculative-decoding drafter) layer. AMD's Quark and community GPTQ quantizations of `Qwen3.5-35B-A3B` quantize the 40 transformer layers but leave the MTP layer's experts in bf16. The main model loads fine; only the drafter fails. * **Original MXFP4/Quark report** (`RuntimeError: The size of tensor a (256) must match the size of tensor b (512) at non-singleton dimension 1`, in `routed_experts.py::_load_w2`). `hidden_size=2048`, `moe_intermediate_size=512`. Quark MXFP4 allocates `w2_weight` as `(E, hidden, intermediate // 2)` = `(E, 2048, 256)` (fp4 packs two values per byte), while the checkpoint's MTP `down_proj.weight` is the unquantized `(2048, 512)`. `amd/Qwen3.5-35B-A3B-MXFP4`'s `quantization_config.exclude` lists `mtp.layers.0.mlp.experts..{gate,up,down}_proj` for all 256 experts, but `should_ignore_layer()` only compared the *parent* layer name `mtp.layers.0.mlp.experts` against those *child* entries. **This half is already fixed on main** by the `check_children` argument added to `quark/utils.py::should_ignore_layer` (the amd/GLM-5.2-MXFP4 fix), which is why a commenter on the issue could not reproduce it on main. No change needed. * **Still-broken GPTQ report** (issue comments: `AttributeError: 'RoutedExperts' object has no attribute 'w2_weight'. Did you mean: 'w2_qweight'?`, same drafter-loading path). `AutoGPTQConfig` tracks which modules the checkpoint actually quantized in `modules_in_block_to_quantize` — auto-derived in `maybe_update_config()` from the safetensors dtypes, so it contains only the quantized layers' per-expert module names. `get_linear_quant_method()` consults it via `is_layer_gptq_quantized()`, but the `RoutedExperts` branch of `AutoGPTQConfig.get_quant_method()` **never consulted it at all** — neither on the Marlin path nor on the `MoeWNA16Config` fallback path taken here. The MTP layer therefore got a WNA16 MoE method allocating `w13_qweight`/`w2_qweight`, while the checkpoint supplies `...down_proj.weight`, and `RoutedExperts.load_weights()`'s `getattr(self, \"w2_weight\")` raised. ## 2. The fix and why Consult `modules_in_block_to_quantize` for routed-expert layers, the same way it is already consulted for linear layers. * New `is_moe_layer_gptq_quantized(prefix, quantized_layers)` in `gptq_utils.py`. `prefix` is the fused layer (`...mlp.experts`) while the config lists individual expert projections (`...mlp.experts.0.down_proj`), so a plain substring test in either direction is wrong. The helper normalizes each entry to its `...experts` parent and matches that as a suffix of `prefix`, which works for both absolute names (the auto-derived case) and the block-relative names optimum writes. * Called from `AutoGPTQConfig.get_quant_method()` *before* the Marlin-support check, so both the Marlin path and the `MoeWNA16Config` fallback are covered by the single early `return UnquantizedFusedMoEMethod(layer.moe_config)`. Deliberately conservative: when the config enumerates no expert module at all it carries no information about experts, so the helper returns `True` and behaviour is byte-for-byte unchanged for every config shape that does not list experts. ## 3. Files changed * `vllm/model_executor/layers/quantization/utils/gptq_utils.py` — add `is_moe_layer_gptq_quantized()`. * `vllm/model_executor/layers/quantization/auto_gptq.py` — import it and use it in the `RoutedExperts` branch of `get_quant_method()`. * `tests/quantization/test_auto_gptq.py` — add `test_moe_layer_quantized_only_when_its_experts_are_in_the_checkpoint`. ## 4. Risk / uncertainty * The behaviour change is gated on `modules_in_block_to_quantize` containing at least one `...experts...` entry. Fully-quantized MoE GPTQ checkpoints list every layer's experts, so they keep the quantized method. The residual risk is a checkpoint that lists experts for *some* layers under a naming scheme where the `...experts` suffix does not line up with the vLLM module prefix; that layer would silently fall back to unquantized (loud failure on the following weight load rather than silent wrong numerics, since the parameter names would then mismatch). * Block-relative configs (`mlp.experts.0.down_proj`) still cannot distinguish the MTP layer from the main layers, so the fix does not help there. That is the pre-existing behaviour, asserted in the test so the limitation is explicit. * I could not obtain the exact GPTQ checkpoint the reporter used, so the shape of its `modules_in_block_to_quantize` is inferred from `maybe_update_config()`'s derivation logic plus the `w2_weight`-vs-`w2_qweight` error, not observed. * Not touched: the Quark/MXFP4 path (already fixed on main) and the AWQ path, which uses a different exclusion mechanism (`modules_to_not_convert`). ## 5. How I verified it * **Root cause**, traced from the traceback through `routed_experts.py` (`_load_w2`, `weight_loader`, `build_expert_params_mapping`), `quark/quark_moe.py::create_weights` (`get_packed_dim`), `quark/utils.py`, `auto_gptq.py`, `moe_wna16.py` and `gptq_utils.py`, cross-checked against `Qwen/Qwen3.5-35B-A3B/config.json` (`hidden_size=2048`, `moe_intermediate_size=512` — exactly the 2048×256 vs 2048×512 mismatch) and `amd/Qwen3.5-35B-A3B-MXFP4/config.json` (the `mtp.layers.0.mlp.experts.*` exclude entries). * **Confirmed the MXFP4 half is already fixed** by diffing `quark/utils.py::should_ignore_layer` at the reporter's commit `97c162676` (no `check_children`) against `main` (has it), matching the issue comment reporting it no longer reproduces. * **Helper logic**: executed `is_moe_layer_gptq_quantized` in isolation over 8 cases (main-model layer, MTP layer, block-relative names, no-experts config, fused-expert entry, nested `list[list[str]]` form, empty config) — all pass. * **Lint**: `ruff check` and `ruff format --check` clean on all three files; `py_compile` clean. * **Not run**: the pytest suite. This environment has no `.venv` and no torch installed, so `tests/quantization/test_auto_gptq.py` could not be executed, and no end-to-end load of the affected checkpoints was performed. ## Duplicate check GitHub search for open PRs referencing `49141` in the body returned `total_count: 0`. No open PR touches `is_layer_gptq_quantized` / `get_moe_quant_method`.", "url": "https://github.com/vllm-project/vllm/pull/49539", "createdAt": "2026-07-23T06:59:04Z", "updatedAt": "2026-08-13T03:54:49Z", "timestamp": "2026-08-13T03:54:49Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "rocm", "qwen", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:546b32a229853f400a31", "signalId": "github:vllm-project/vllm:pull_request:48672", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48672", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix: reject required/named tool_choice for Harmony models without a tool parser", "text": "### Summary In the OpenAI-compatible chat path, when no tool-call parser is configured, `tool_choice=\"required\"` and named tool choices were accepted for Harmony (gpt-oss) models and then produced empty or incorrect tool-call results. ### Fix Only exempt Harmony for `tool_choice=\"auto\"` (best effort). Named and `required` now fail closed with a 400 when no tool parser is available. Adds a regression test. Fixes #48207", "url": "https://github.com/vllm-project/vllm/pull/48672", "createdAt": "2026-07-14T23:55:39Z", "updatedAt": "2026-08-13T03:54:48Z", "timestamp": "2026-08-13T03:54:48Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "gpt-oss" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2fa6be26193456d68b28", "signalId": "github:vllm-project/vllm:pull_request:48673", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48673", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "test(quantization): cover tied lm_head/embed_tokens when lm_head excluded from ModelOpt", "text": "### Summary Adds regression coverage for tying an excluded `lm_head` to `embed_tokens` through the quantization method path (`QuantizeMethodBase.tie_weights`), the behavior that fixed the original NVFP4 crash on main. Test only. Related to #48238", "url": "https://github.com/vllm-project/vllm/pull/48673", "createdAt": "2026-07-14T23:59:09Z", "updatedAt": "2026-08-13T03:54:47Z", "timestamp": "2026-08-13T03:54:47Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:022712a075e2a1c1e78c", "signalId": "github:vllm-project/vllm:pull_request:49541", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49541", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] Cover detokenize=False in offline beam search", "text": "Offline counterpart of #46422. Sets `detokenize=False` on the per-step sampling params in the offline beam search path, and threads it through the structured-output beam path as well, so intermediate beam candidates are not detokenized on every step. This does not fix the Whisper slowdown reported in #49197. That one is the per-beam encoder recompute called out in the FIXME in `vllm/entrypoints/generate/beam_search/utils.py`, which is a separate change. Not benchmarked on GPU. ## Test `tests/samplers/test_beam_search_offline.py` covers both the plain and the structured-output beam paths and runs without a GPU.", "url": "https://github.com/vllm-project/vllm/pull/49541", "createdAt": "2026-07-23T06:59:22Z", "updatedAt": "2026-08-13T03:54:37Z", "timestamp": "2026-08-13T03:54:37Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "frontend", "ready" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e4de1e2a4213ddb5c1aa", "signalId": "github:vllm-project/vllm:pull_request:51082", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][TurboQuant] Resolve the cache preset when get_kv_cache_shape gets no dtype hint", "text": "# Issue #50709 — TurboQuant hybrid model crashes with `Unknown cache dtype: auto` Upstream issue: https://github.com/vllm-project/vllm/issues/50709 Reported symptom: serving a hybrid model (Qwen3.5-35B-A3B: full attention + GDN linear attention + MoE) with `--kv-cache-dtype turboquant_4bit_nc` aborts during startup (`determine_available_memory()`) with ``` Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc ``` ## 1. Root cause The only site in the tree that can emit that message is `TurboQuantConfig.from_cache_dtype()` (`vllm/model_executor/layers/quantization/turboquant/config.py:220`), and the only caller that can reach it with `\"auto\"` is `TurboQuantAttentionBackend.get_kv_cache_shape()`. `AttentionBackend.get_kv_cache_shape(num_blocks, block_size, num_kv_heads, head_size, cache_dtype_str=\"auto\")` treats `cache_dtype_str` as an **optional hint** — every other backend derives its shape from the four positional arguments alone. TurboQuant is the only backend that makes the hint *load-bearing*: it re-parses it into a `TurboQuantConfig` to obtain `slot_size_aligned` (the packed `[key|value]` slot width), and hard-fails on anything that is not a preset name. Several in-tree callers legitimately do not supply a per-layer preset: - `AttentionBackend.get_kv_cache_block_dim()` (`vllm/v1/attention/backend.py:100`) defaults the hint to `\"auto\"` and forwards it — so *any* call to `TurboQuantAttentionBackend.get_kv_cache_block_dim(...)` raises. - The KV-connector layout probes call `get_kv_cache_shape()` with no hint at all (`vllm/distributed/kv_transfer/kv_connector/utils.py:430`, `.../offloading/worker.py:250`). - Both model runners downgrade the hint to `\"auto\"` for every KV-cache group whose spec reports `KVQuantMode.NONE`: - v2: `vllm/v1/worker/gpu/attn_utils.py:323` (`_reshape_kv_cache`) - v1: `vllm/v1/worker/gpu_model_runner.py:7415` (`_reshape_kv_cache_tensors`) That downgrade exists for `--kv-cache-dtype-skip-layers`, where individual layers keep the native dtype while the engine-wide `cache_dtype` stays quantized. It reasons from the *spec's* quant mode but hands the result to the *group's backend*, and nothing ties those two together. The v1 runner's `_reshape_kv_cache_tensors` is reached from `determine_available_memory()` → `profile_cudagraph_memory()` → `_init_minimal_kv_cache_for_profiling()` → `initialize_kv_cache()`, which is exactly the stage named in the report. On hybrid models the extra spec types (`MambaSpec` for the GDN layers, plus the skip/boundary machinery that TurboQuant enables around them) are what make a group reach the reshape with a `KVQuantMode.NONE` spec while still being served by the TurboQuant backend. Reproduced directly (see §5): `get_kv_cache_shape(..., cache_dtype_str=\"auto\")` and `get_kv_cache_block_dim(16, 8, 128)` both raise the reported error verbatim. ## 2. The fix and why `vllm/v1/attention/backends/turboquant_attn.py`: added `_resolve_tq_preset()` and routed `get_kv_cache_shape()` through it. When the caller-supplied hint is not a TurboQuant preset, the preset is taken from the engine's configured `cache_config.cache_dtype`; if that is not a preset either, the original `ValueError` is raised unchanged. Why this is safe and correct rather than a papering-over: - A layer can only ever be served by this backend if its KV cache dtype is a `turboquant_*` preset. `TurboQuantAttentionBackend.supports_kv_cache_dtype()` requires `kv_cache_dtype.startswith(\"turboquant_\")`, and `AttentionBackend.validate_configuration()` enforces it for *both* auto-selected and explicitly selected (`--attention-config backend=...`) backends (`vllm/platforms/cuda.py:407-423`, `vllm/v1/attention/backend.py:344`). Since there is a single engine-wide `--kv-cache-dtype`, `cache_config.cache_dtype` is by construction the preset of every layer this backend serves. - The fix restores the documented contract of `get_kv_cache_shape`: the shape is a property of the backend and its configuration, not of a hint the caller may not have. It therefore covers all four caller classes above at once, instead of patching one runner and leaving `get_kv_cache_block_dim()` and the KV-connector probes still broken. - It is not a silent fallback: outside a TurboQuant run the clear diagnostic is preserved (covered by a test). Explicitly *not* changed, to keep the diff scoped to this issue: - The `KVQuantMode.NONE → \"auto\"` heuristic in the two runners. It is the proximate trigger, but it is shared with MLA/`fp8_ds_mla` and is already the subject of open upstream PRs (see §4); changing it here would collide. - `get_kv_cache_shape`'s hardcoded `cache_dtype_str=\"turboquant_4bit_nc\"` default. Worth revisiting separately: callers that omit the argument (the KV-connector probes) currently get a `turboquant_4bit_nc` slot width even on a `turboquant_k3v4_nc` run. Harmless today — those probes only inspect the shape's rank and the position of `num_blocks` — but latent. ## 3. Files changed | File | Change | | --- | --- | | `vllm/v1/attention/backends/turboquant_attn.py` | New `_resolve_tq_preset()`; `get_kv_cache_shape()` resolves the preset through it (+25/-1). | | `tests/quantization/test_turboquant.py` | New `TestTurboQuantKVCacheShape` regression class (5 tests) + one import (+50). | Test placement follows AGENTS.md (\"reuse before create\"): the existing TurboQuant suite already covers config/preset behaviour and runs on CPU. ## 4. Risk / uncertainty - **Duplicate check.** Done through the GitHub web UI (the task forbids running `gh`). No open PR references #50709. Adjacent open PRs touch the same `KVQuantMode.NONE → \"auto\"` heuristic — #47618 (\"Unify per-layer KV cache dtype selection across v1/v2\"), #48907 (\"Prefer per-spec `cache_dtype_str` when reshaping KV cache\"), #49798 (\"add TQFullAttentionSpec guard in v1 `_reshape_kv_cache_tensors`\") — but all three fix the *runner* side for MLA specs that carry a `cache_dtype_str` field. `TQFullAttentionSpec` has no such field, and none of them makes the TurboQuant backend tolerate a missing hint, so `get_kv_cache_block_dim()` and the KV-connector probes stay broken under all three. This change is complementary, not a duplicate — but if any of them lands first the interaction should be re-checked. - **Main uncertainty: I could not reproduce the exact hybrid configuration** that makes a TurboQuant-backed group's spec report `KVQuantMode.NONE`. There is no GPU and no model checkpoint in this environment, and the reporter attached no traceback. I verified the raise site, the callers, and the backend-selection invariant statically, and reproduced the error itself directly; I did not observe the upstream spec that loses the quant mode. The fix is therefore placed at the point where the invariant is *provable* (backend ⇒ TurboQuant dtype) rather than at whichever spec path produces the `NONE`. If the real trigger is a spec whose page size was also computed with the unquantized formula, this change turns the hard failure into a tensor shape/size mismatch at reshape time — a different, still-loud error, not silent corruption. - **Behaviour change:** `get_kv_cache_shape(\"auto\")` now succeeds inside a TurboQuant run instead of raising. Nothing in-tree depends on that raise. - `_resolve_tq_preset()` calls `get_current_vllm_config()`. This is already the established pattern in this file (`TurboQuantAttentionImpl.__init__`), and when no config is set the helper falls through to the original error rather than guessing. - **Not verified:** GPU kernels, end-to-end serving, and model quality. Per AGENTS.md a model eval would be required before submitting this upstream; it cannot be run here. This change does not alter any quantization math or kernel — only which preset name the shape helper resolves — so no accuracy impact is expected. ## 5. How I verified it Environment: `uv venv --python 3.12`, CPU-only torch 2.13.0 + `requirements/common.txt` (no GPU, no compiled `vllm._C`; the TurboQuant config/shape path is pure Python). **Reproduction of the reported error on unmodified `main`:** ``` $ .venv/bin/python -c \"...TurboQuantAttentionBackend.get_kv_cache_shape(4,16,8,128,cache_dtype_str='auto')\" RAISED: Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, ... $ .venv/bin/python -c \"...TurboQuantAttentionBackend.get_kv_cache_block_dim(16, 8, 128)\" RAISED block_dim: Unknown TurboQuant cache dtype: 'auto'. Valid presets: turboquant_k8v4, ... ``` **Test fails before the fix / passes after.** With the source change stashed and the new tests in place, all 5 fail with the reported `ValueError`. With the fix applied: ``` $ .venv/bin/python -m pytest tests/quantization/test_turboquant.py -q -k KVCacheShape 5 passed, 127 deselected ``` **No regressions in the touched and adjacent suites:** ``` $ .venv/bin/python -m pytest tests/quantization/test_turboquant.py -q 115 passed, 17 skipped $ .venv/bin/python -m pytest tests/v1/worker/test_attn_utils.py tests/v1/test_kv_cache_spec_registry.py -q 42 passed ``` Note: every test in this environment also reports a teardown `ERROR` from `tests/conftest.py`'s autouse `cleanup_dist_env_and_memory()` fixture (`RuntimeError: Cannot access accelerator device when none is available`). This is the missing GPU, not the change — it reproduces identically on files I did not touch (`tests/v1/test_kv_cache_spec_registry.py`: 42 passed, 42 errors). **Behavioural spot-check** (`turboquant_k3v4_nc` engine, slot 118 vs `turboquant_4bit_nc` slot 134 — the resolver picks the configured preset, an explicit hint still wins, and a non-TurboQuant run still errors): ``` auto -> (4, 8, 16, 118) block_dim -> 0 explicit -> (4, 8, 16, 134) cache_dtype=\"auto\" engine -> ValueError: Unknown TurboQuant cache dtype: 'auto' ``` **Lint:** `ruff check` and `ruff format --diff` clean on both changed files (88-char limit respected).", "url": "https://github.com/vllm-project/vllm/pull/51082", "createdAt": "2026-08-04T21:34:13Z", "updatedAt": "2026-08-13T03:54:18Z", "timestamp": "2026-08-13T03:54:18Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:882c97d7a07c49c8afd0", "signalId": "github:vllm-project/vllm:pull_request:49104", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49104", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Bump `openai` to `>=2.25.0` to support namespace tools types", "text": "## Purpose Fix the dependency lower bound issue #49103. We have recently added OpenAI responses API namespace tools support, but the corresponding openai types are only supported for `openai>=2.25.0`. Hence the lower bound bump. Note that the dependency bump is necessary in the sense that any client with `openai<2.25.0` got an inevitable `ImportError` from the very beginning of vllm online serving process (see the error traceback in the linked issue). ## Test Plan No additional tests required - it just resolves the missing dependency lower bound bump. ## Test Result N/A ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49104", "createdAt": "2026-07-19T15:41:00Z", "updatedAt": "2026-08-13T03:52:31Z", "timestamp": "2026-08-13T03:52:31Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready", "ci/build" ], "author": "cjackal", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:005c13e592ba56062966", "signalId": "github:vllm-project/vllm:issue:51842", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51842", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails (because vLLM expects k_proj in a fused-qkv ViT attention layer)", "text": "### Your current environment vLLM: 0.27.0 Python: 3.10.19 PyTorch: 2.13.0+cu130 GPU: NVIDIA GeForce RTX 4070 Ti SUPER (16 GB) NVIDIA driver: 610.62 OS: Ubuntu 22.04.5 LTS under WSL2 Model: MiniCPM-V-4.6 Model path: /mnt/e/BaiduNetdiskDownload/MiniCPM-V-4.6 --trust-remote-code enabled ### 🐛 Describe the bug ### Description Starting `OpenBMB/MiniCPM-V-4.6` with vLLM `0.27.0` fails while loading model weights. The failure occurs in vLLM's built-in `minicpmv4_6.py` implementation, before the server starts. The model's vision attention layer exposes fused `qkv_proj` parameters, but the vLLM weight-loading path attempts to load a non-existent `k_proj` parameter. ### Reproduction ```bash VLLM_WSL2_ENABLE_PIN_MEMORY=1 vllm serve \\ --model /mnt/e/BaiduNetdiskDownload/MiniCPM-V-4.6 \\ --trust-remote-code \\ --gpu-memory-utilization 0.9 \\ --port 8000 \\ --served-model-name OpenBMB/MiniCPM-V-4.6 \\ --max-model-len 10240 \\ --max-num-seqs 4 ``` Startup fails while loading weights: ```bash ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters belonging to (MiniCPMV4_6ViTWindowAttentionSelfAttn) are: {'qkv_proj.bias', 'qkv_proj.weight', 'out_proj.bias', 'out_proj.weight'} Relevant traceback: File \".../vllm/model_executor/models/minicpmv4_6.py\", line 1283, in load_weights return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) File \".../vllm/model_executor/models/minicpmv4_6.py\", line 709, in load_weights return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) File \".../vllm/model_executor/models/utils.py\", line 395, in _load_module raise ValueError(msg) ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters belonging to (MiniCPMV4_6ViTWindowAttentionSelfAttn) are: {'qkv_proj.bias', 'qkv_proj.weight', 'out_proj.bias', 'out_proj.weight'} ``` ### Suspected cause MiniCPMV4_6ViTWindowAttentionSelfAttn uses fused qkv_proj.{weight,bias} parameters, while the current vLLM checkpoint mapping or loader appears to target a separate k_proj parameter. Could the MiniCPM-V-4.6 vision-attention weight mapping be updated to handle the fused QKV projection layout? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51842", "createdAt": "2026-08-11T16:08:19Z", "updatedAt": "2026-08-13T03:50:58Z", "timestamp": "2026-08-13T03:50:58Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "llery2021", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a1dabd0c3951cd93ae09", "signalId": "github:vllm-project/vllm:pull_request:51256", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51256", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Reserve the bonus query slot in DFlash scheduling budget", "text": "## Purpose The existing generic parallel-drafting calculation only reserves K - 1 slots for DFlash, leaving the scheduling budget short by one slot per request. For example, with: max_num_batched_tokens = 2048 max_num_seqs = 256 num_speculative_tokens = 8 the previous calculation allowed: max_num_scheduled_tokens = 2048 - 7 * 256 = 256 However, a full DFlash batch may require: 256 * (8 + 1) = 2304 query tokens, exceeding max_num_batched_tokens. DFlash has K + 1 query tokens per request, so its net drafting expansion is K slots rather than the generic parallel-drafting value of K - 1.", "url": "https://github.com/vllm-project/vllm/pull/51256", "createdAt": "2026-08-06T11:25:35Z", "updatedAt": "2026-08-13T03:50:24Z", "timestamp": "2026-08-13T03:50:24Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "bug", "ready" ], "author": "HF-001", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:075c8dec5abc245e95e4", "signalId": "github:vllm-project/vllm:issue:50282", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50282", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Expose startup status and health endpoint before model engine is ready", "text": "### 🚀 The feature, motivation and pitch vLLM currently exposes its HTTP API only after the model engine has completed initialization. For models with expensive startup phases, this can leave the service unreachable for a long time even though the process is alive and making progress. For example, startup may include: Loading model checkpoints Compiling execution paths Allocating KV cache FlashInfer or backend autotuning CUDA graph capture Model-specific kernel warmup With large or hybrid models, these phases can take many minutes. During that period, an external orchestrator or monitoring system cannot distinguish between: Normal initialization A stalled startup An out-of-memory condition A crashed worker A restart loop A process that is alive but not yet ready to serve inference I would like vLLM to expose a lightweight management or health endpoint before the model engine is ready. A possible design would separate process liveness, startup state, and inference readiness: ``` GET /livez GET /readyz GET /startupz ``` Example startup response: ``` { \"state\": \"starting\", \"phase\": \"model_warmup\", \"detail\": \"mamba2_ssd_kernel_warmup\", \"elapsed_seconds\": 842 } ``` A minimal implementation does not need an estimated completion time or exact percentage. Even a stable state and phase enum would be useful: ``` PROCESS_STARTED LOADING_MODEL COMPILING ALLOCATING_KV_CACHE AUTOTUNING_KERNELS CAPTURING_GRAPHS WARMING_UP READY FAILED ``` The management endpoint should be served by a lightweight component that starts before engine initialization and remains available if model initialization fails. This would make vLLM much easier to operate behind Kubernetes, systemd, appliance supervisors, load balancers, and enterprise monitoring systems. The goal is not necessarily to reduce startup time. The primary goal is to make long startup sequences observable and diagnosable. **Do not optimize startup first. Make startup observable first.** ### Alternatives The current alternatives are all external workarounds and provide incomplete information. Monitoring the process An orchestrator can verify that the vLLM process or container is running, but process liveness does not indicate whether initialization is progressing or stalled. `Process alive != Model ready` **Monitoring the inference port** The OpenAI-compatible endpoint can be polled until it becomes reachable. However, this only exposes two states: ``` Unavailable Available ``` It does not expose the difference between normal startup, initialization failure, deadlock, OOM, or repeated restarts. Parsing logs A supervisor can parse vLLM logs and infer startup progress from messages such as checkpoint loading, compilation, cache allocation, or warmup. This is fragile because: - Log messages are not a stable API - Formats may change between versions - Different model architectures use different startup stages - Logs may come from multiple worker processes - Structured failure state is difficult to derive reliably **Adding an external wrapper or sidecar** A wrapper can expose its own health endpoint and inspect the child process, logs, or inference port. However, it still cannot reliably access internal engine initialization state without parsing implementation-specific output. An external supervisor is useful for lifecycle management, but vLLM itself is the component that knows the current startup phase and failure reason. **Increasing startup probe timeouts** Kubernetes `startupProbe` or equivalent timeouts can be increased to accommodate slow model initialization. This prevents premature restarts but still provides no visibility into whether startup is progressing. ### Additional context This becomes increasingly important for large models and non-trivial architectures where startup is not dominated only by checkpoint I/O. A real startup sequence may look like: ``` Process started ↓ Checkpoint loading ↓ torch.compile ↓ KV cache allocation ↓ Backend kernel autotuning ↓ Model-specific kernel warmup ↓ API server ready ``` On large models, the interval between process creation and API readiness may be tens of minutes. Long initialization itself is acceptable in many production environments when it is expected and observable. The operational problem is that, during this interval, monitoring systems currently cannot tell whether vLLM is: ``` 🟡 Starting normally 🔴 Failed 🔴 Stalled 🔴 Out of memory 🔴 Restarting ``` This is also relevant for multi-model systems. A management plane may need to report states such as: ``` Model A: READY Model B: WARMING_UP Model C: FAILED ``` without treating the entire appliance or node as unavailable. It may also be useful to expose the same information as Prometheus metrics, for example: ``` vllm_engine_startup_state{state=\"warming_up\"} 1 vllm_engine_startup_phase{phase=\"kernel_warmup\"} 1 vllm_engine_startup_elapsed_seconds 842 ``` The exact endpoint names and phase granularity are open to discussion. The key requirements are: The status interface becomes reachable before model engine initialization completes. It distinguishes liveness from inference readiness. It exposes a stable, machine-readable startup state. It reports initialization failure when possible. It does not depend on parsing human-readable logs. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/50282", "createdAt": "2026-07-29T14:42:40Z", "updatedAt": "2026-08-13T03:47:02Z", "timestamp": "2026-08-13T03:47:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "xexxo2002-arch", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f97e08be06cfa1aaad4b", "signalId": "github:vllm-project/vllm:pull_request:52076", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52076", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Clearer comments in `BlockPool.free_blocks()`", "text": "The comments explaining eviction precedence in `BlockPool.free_blocks()` were ambiguous/confusing. Make them clearer / more explicit.", "url": "https://github.com/vllm-project/vllm/pull/52076", "createdAt": "2026-08-13T00:42:30Z", "updatedAt": "2026-08-13T03:46:23Z", "timestamp": "2026-08-13T03:46:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "njhill", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1895d5704d5eacd0ad33", "signalId": "github:vllm-project/vllm:pull_request:51459", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51459", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Fix and extend PR/issue auto-labeling", "text": "## Purpose Auditing the labeling setup against the actual repo turned up rules that had stopped matching the code they were written for, plus issue-side labeling that covered only 6 of 62 labels. ### Rules that were missing real work - **`deepseek` missed the `vllm/models/` tree.** Newer models live in `vllm/models//` rather than `vllm/model_executor/models/`. The rule only knew the old location, so only the title match applied — **64 of 99 PRs** touching `vllm/models/deepseek_v4/` went unlabeled. Same gap for `deepseek_v32`. - **`speculative-decoding` missed the Model Runner V2 subtree.** `vllm/v1/worker/gpu/spec_decode/` has seen 94 PRs in 6 months, more than the original `vllm/v1/spec_decode/` at 81. This also folds `dflash` and `dspark` under the umbrella label, per review feedback that both are spec decoding techniques. ### Rules that could never fire Four conditions used `files=` (exact match) with regex syntax, or carried a doubled backslash that YAML passes through literally: ```yaml - files=^examples/features/speculative_decoding/ # never matches - files~=^\\\\.buildkite/intel_jobs/ # never matches ``` A further 13 conditions pointed at files that have since moved or been deleted — `harmony` moved to the Rust frontend, `pallas` is gone. Repointed where the code moved, removed where it is gone. ### Retired: `new-model` The rule fired on any PR touching a model file **and** `registry.py`. `registry.py` is edited for renames, aliases, refactors and removals, not just additions. Of the 25 most recent PRs carrying the label, 10 added a model. The rest included: - `[Model] Remove Ouro`, `[Model] Remove Plamo2`, `Remove PersimmonForCausalLM and FuyuForCausalLM` — PRs that **delete** models - `[model registry] some simple typos` - `[Perf] Optimize clamp to clamp_`, `[Kernel] ReplaySSM: cache SSM inputs`, `Revert \"Blackwell decode optimizations\"` Whether a new architecture was registered depends on what changed *inside* `registry.py`. Mergify matches file paths, not diff contents, so this rule cannot be made accurate — tightening the paths does not help. 111 open PRs currently carry the label. The label remains on issues, applied by the New Model template, where the requester states the intent explicitly. ### Rule that fired too often `nvidia` matched `requirements/cuda.txt`, so dependency bumps were tagged as NVIDIA backend work. Measured across 200 merged PRs, 17 carried the label and this was the one clear miss. ### New rules `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`, `DSv4`, `dflash`, `torch.compile`, `ray`, `vllm-ir`. ### Issue labeling Extended from 6 labels to 31. Terms match **titles only** by design: issue bodies carry pasted `collect_env` output that names unrelated hardware and libraries, which tags the reporter's environment rather than the topic. ### Guard against recurrence `tools/pre_commit/check_label_rules.py` fails when a file condition matches nothing in the tree, and runs via pre-commit when `mergify.yml` changes. It caught two bugs in this change while it was being written. ### Also removed: the New Model issue template `.github/ISSUE_TEMPLATE/600-new-model.yml` is deleted. - Usage fell from 155 issues in 2025 to 30 in 2026, only 3 since July - It applied a label named `new model`, but the label is `new-model`. GitHub silently skips a label that does not exist, so requests went untagged from the March rename onward — five months, unnoticed - 292 of the 304 ever filed are closed - Model support arrives as planned work rather than as community requests `blank_issues_enabled` is `false`, so model requests route to Feature Request. **Nothing applies `new-model` after this change.** The label is left in place so it stays on the 917 PRs and 304 issues carrying it; deleting the label itself is a separate call. ## Before merging **The five new labels must be created first** — `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`. Mergify errors on a label that does not exist. Two related items a PR cannot do: - `mrv1-only`'s description currently reads \"apply only to Model Runner V2 (not applicable to Model Runner V2)\" — says V2 twice. - `DSv4` has a rule here mirroring the `kimi`/`k3` split, but whether to keep it as a distinct label is a maintainer call. ## Test Plan Rules were validated against the real file tree and against real issues and PRs, rather than by inspection. ```bash # every file condition in mergify.yml matches at least one tracked file python tools/pre_commit/check_label_rules.py pre-commit run check-label-rules --files .github/mergify.yml pre-commit run ruff-check --files tools/pre_commit/check_label_rules.py pre-commit run ruff-format --files tools/pre_commit/check_label_rules.py ``` ## Test Result ``` tracked files: 6430 rules: 44 file conditions matching >=1 file: 154 file conditions matching NOTHING: 0 Check auto-label rules still match real files............................Passed ruff check...............................................................Passed ruff format..............................................................Passed ``` Negative test: injecting a condition pointing at a nonexistent path makes the check exit 1 and name the offending rule. New rules were spot-checked against the tree — `glm` matches 21 files and correctly ignores `midashenglm.py`, which contains the substring \"glm\" but is an unrelated model. Issue rules were replayed over 600 real issues: 43% now receive at least one label, and the existing rules agreed with hand-applied labels (`kimi` 35 fires / 31 already labeled, `rocm` 34 / 33, `k3` 27 / 26). No model evaluation is included because this change touches only CI labeling configuration. It does not affect inference, model output, accuracy, or serving. ## Note on AI assistance AI assistance (Claude Code) was used to audit the existing rules, measure PR and issue volumes, and draft these changes. Every measurement quoted above is reproducible with the commands shown. I have reviewed each changed line.", "url": "https://github.com/vllm-project/vllm/pull/51459", "createdAt": "2026-08-07T22:57:38Z", "updatedAt": "2026-08-13T03:43:40Z", "timestamp": "2026-08-13T03:43:40Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "ready", "ci/build" ], "author": "jcotant-inferact", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:157362ebd93a8ff510c3", "signalId": "github:vllm-project/vllm:issue:52089", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52089", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Continuous Host Memory Growth / Possible Memory Leak with V2 Runner on Qwen3-14B and Qwen3-Rerank-4B", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 | packaged by Anaconda, Inc. | (main, Jul 9 2026, 14:38:16) [GCC 14.3.0] (64-bit runtime) Python platform : Linux-6.8.0-31-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.2.51 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4090 GPU 1: NVIDIA GeForce RTX 4090 Nvidia driver version : 595.80 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD EPYC 7543 32-Core Processor CPU family: 25 Model: 1 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 1 BogoMIPS: 5589.49 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core intel_ppin ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd virt_ssbd arat umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache: 8 MiB (16 instances) L3 cache: 64 MiB (4 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.14 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.26.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.11.0 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0 [pip3] torchcodec==0.15.0 [pip3] torchvision==0.26.0 [pip3] transformers==5.14.1 [pip3] triton==3.6.0 [conda] flashinfer-python 0.6.14 pypi_0 pypi [conda] numpy 2.3.5 pypi_0 pypi [conda] nvidia-cublas 13.1.0.3 pypi_0 pypi [conda] nvidia-cuda-cccl 13.3.3.4.1 pypi_0 pypi [conda] nvidia-cuda-crt 13.3.73 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvcc 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvdisasm 13.3.73 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.26.0 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-cufile 1.15.1.6 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-base 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-core 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu12 4.6.0 pypi_0 pypi [conda] nvidia-cutlass-dsl-libs-cu13 4.6.0 pypi_0 pypi [conda] nvidia-ml-py 13.610.43 pypi_0 pypi [conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvshmem-cu13 3.4.5 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] nvidia-nvvm 13.3.73 pypi_0 pypi [conda] pyzmq 27.1.0 pypi_0 pypi [conda] tokenspeed-triton 3.8.10.post20260721 pypi_0 pypi [conda] torch 2.11.0 pypi_0 pypi [conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi [conda] torchaudio 2.11.0 pypi_0 pypi [conda] torchcodec 0.15.0 pypi_0 pypi [conda] torchvision 0.26.0 pypi_0 pypi [conda] transformers 5.14.1 pypi_0 pypi [conda] triton 3.6.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-31 0 N/A GPU1 PHB X 0-31 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/cuda/lib64: CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_ubuntu ```
## Description I observed continuous and approximately linear **host memory growth** when running long-running data processing workloads with the **V2 Runner** and the following models: * Qwen3-14B-FP8 * Qwen3-Rerank-4B The memory usage keeps increasing over time and eventually consumes almost all available system memory. I reproduced the issue with both: * vLLM `v0.25.1` * vLLM `v0.27.0` In contrast, the same Qwen3-14B and Qwen3-Rerank-4B workloads remain stable when running with the **V1 Engine** on `vLLM==0.11.2`. I also tested `vLLM==0.25.1` with TranslateGemma. Its host memory usage remains stable during long-running inference, which suggests that this may be related to the V2 Runner and/or its interaction with these Qwen3 models rather than a general memory behavior of vLLM. ## Observed Behavior With V2 Runner: ```text Qwen3-14B-FP8 ↓ long-running data cleaning / inference workload ↓ host memory continuously increases ↓ memory usage grows approximately linearly ↓ eventually reaches almost 100% ↓ process/system becomes unstable or runs out of memory ``` The memory growth is not a short-term allocation spike. It continues accumulating throughout the long-running workload. The attached screenshots show the memory usage trend. In the long-term monitoring view, host memory usage increases from roughly 10% to nearly 100% over the observation period, with only a small drop/recovery when the process is restarted. ## Comparison ### V2 Runner | Model | vLLM Version | Engine | Prefix Caching | Long-running Memory | | --------------- | --------------: | ------ | -------------- | ------------------------ | | Qwen3-14B-FP8 | 0.25.1 | V2 | Enabled | ❌ Continuously increases | | Qwen3-14B-FP8 | 0.27.0 | V2 | Enabled | ❌ Continuously increases | | Qwen3-Rerank-4B | 0.25.1 / 0.27.0 | V2 | Enabled | ❌ Continuously increases | ### V1 Engine | Model | vLLM Version | Engine | Prefix Caching | Long-running Memory | | --------------- | -----------: | ------ | -------------- | ------------------- | | Qwen3-14B-FP8 | 0.11.2 | V1 | Enabled | ✅ Stable | | Qwen3-Rerank-4B | 0.11.2 | V1 | Enabled | ✅ Stable | ### Another V2 Model | Model | vLLM Version | Engine | Long-running Memory | | -------------- | -----------: | ------ | ------------------- | | TranslateGemma | 0.25.1 | V2 | Stable | This makes the regression particularly interesting because: 1. The issue occurs with multiple Qwen3 models. 2. It occurs across vLLM `0.25.1` and `0.27.0`. 3. The same Qwen3 models are stable under the V1 Engine. 4. Another model (TranslateGemma) is stable under V2. 5. Prefix caching is enabled in the tested deployments. ## Reproduction The following is the deployment command used for Qwen3-14B-FP8: ```bash gpu_id=${1:-0} export PYTORCH_ALLOC_CONF=\"expandable_segments:True\" CUDA_VISIBLE_DEVICES=${gpu_id} vllm serve /data/checkpoint/Qwen3-14B-FP8 \\ --gpu-memory-utilization 0.92 \\ --tensor-parallel-size 1 \\ --served-model-name Qwen3-14B-FP8 \\ --dtype auto \\ --max_model_len 2048 \\ --max-num-seqs 128 \\ --host 0.0.0.0 \\ --port $((8002 + gpu_id)) \\ --enable-prefix-caching \\ --enable-chunked-prefill ``` The workload continuously sends inference requests for a long-running data-cleaning pipeline. ## Environment * Model: `Qwen3-14B-FP8` * Additional affected model: `Qwen3-Rerank-4B` * vLLM: `0.25.1`, `0.27.0` * Stable comparison: `vLLM==0.11.2` (V1 Engine) * GPU: NVIDIA RTX 4090 * Engine: V2 for affected versions * `--enable-prefix-caching` * `--enable-chunked-prefill` * `--max_model_len 2048` * `--max-num-seqs 128` * `--gpu-memory-utilization 0.92` * `PYTORCH_ALLOC_CONF=expandable_segments:True` ## Expected Behavior Host memory usage should eventually reach a relatively stable level after the engine and model have warmed up. For a long-running inference service, memory usage should not continuously grow with the number of processed requests if the workload characteristics remain approximately stable. This is what I observe with: * Qwen3 models under V1 Engine (`v0.11.2`) * TranslateGemma under V2 Runner (`v0.25.1`) ## Actual Behavior With V2 Runner and Qwen3-14B / Qwen3-Rerank-4B, host memory continuously increases during the workload. The growth appears approximately linear with runtime/request volume and eventually exhausts available system memory. The behavior persists across vLLM `0.25.1` and `0.27.0`. ## Additional Observation The fact that the same Qwen3 models are stable with the V1 Engine is a strong indication that the issue may be related to a V2-specific component or code path. Possible areas that may be worth investigating include: * V2 request lifecycle / request state cleanup * prefix caching metadata or cache management * scheduler state * sequence/request objects retained after completion * async engine / output processing * KV cache bookkeeping * chunked prefill related state * Python-side objects associated with completed requests I have not yet determined which component is responsible for the retained memory. ## Evidence I have attached four memory-monitoring screenshots showing the observed behavior and comparison runs. The most obvious long-term behavior is the approximately linear growth of host memory usage under the affected V2 configuration, eventually reaching almost 100%. ## Request Could you please help investigate whether this is a known memory leak or memory retention issue in the V2 Runner? In particular, it would be useful to know whether there were any changes between V1 and V2 around: * request lifecycle management * prefix caching * scheduler state * sequence/request object cleanup * async output processing that could cause completed requests or related metadata to remain referenced over time. \"Image\"", "url": "https://github.com/vllm-project/vllm/issues/52089", "createdAt": "2026-08-13T03:33:00Z", "updatedAt": "2026-08-13T03:42:47Z", "timestamp": "2026-08-13T03:42:47Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "77wheat", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:24dd9eb275e7d3506391", "signalId": "github:vllm-project/vllm:pull_request:51896", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51896", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Reject oversized media before fully downloading it", "text": "# Reject oversized media before fully downloading it ## What this fixes Several media paths checked size only after a remote response or inline audio value had already been turned into one complete `bytes` object. Batch speech requests could also download several oversized rows concurrently before applying their per-row limit. For example, with a 1 MiB limit, a remote server can return a 2 MiB body. Before this change, `async_get_bytes()` called the response's full-body read and allocated all 2 MiB before any limit ran. After this change, vLLM checks `Content-Length` when present and otherwise reads bounded chunks, stopping as soon as the cumulative total exceeds 1 MiB. Oversized inline base64 audio is rejected from its encoded length before decoding. ## Why it matters A client able to submit media URLs or batch audio could force the API or batch process to consume memory and download bandwidth chosen by the remote response, even though the request would later be rejected. Concurrent rows multiplied that cost and could make the process run out of memory. ## The change The three related sites are combined in commit `4cb3fe0`: | Site | What now happens | |---|---| | PTP-VLLM-037 — chat audio | URL downloads stream under the audio byte limit; oversized base64 is rejected before decoding. | | PTP-VLLM-038 — batch speech | The audio byte limit is passed into each URL download before concurrent row processing. | | PTP-VLLM-094 — shared remote media | `VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB` sets a shared default ceiling; sync and async readers check headers and streamed bytes before caching or decoding. | PTP-VLLM-041 is intentionally not included. Current `main` already has stronger per-modality item validation in the Rust frontend, and applying the older patch would weaken that behavior. ## How it was tested PTP-VLLM-037 changed from 4 failures without the fix to all 4 passing. PTP-VLLM-038 changed from 3 failures and 1 pass to all 4 passing. The PTP-VLLM-094 module could not collect on the old tree because the new shared limit did not exist; with the change, all 4 tests passed. The cases cover a declared oversized `Content-Length`, chunked responses without that header, an exact-limit control, early stop after the first over-limit chunk, base64 pre-checking, and batch handler code not being reached for an oversized row. ## Reference Advisory: GHSA-p6g9-7v3x-m8mv Track B: this is a public PR against `vllm-project/vllm@main`. ## Credit - @KernelClint (Clinton Thomas) and @dhalf (Lucas Bourtoule) - Patch the Planet (Trail of Bits + OpenAI collaboration); discovered using GPT-5.5-Cyber All commits include DCO `Signed-off-by` trailers.", "url": "https://github.com/vllm-project/vllm/pull/51896", "createdAt": "2026-08-11T23:55:04Z", "updatedAt": "2026-08-13T03:40:50Z", "timestamp": "2026-08-13T03:40:50Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "frontend", "multi-modality" ], "author": "KernelClint", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:017232c74789e547a893", "signalId": "github:vllm-project/vllm:pull_request:52090", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52090", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix: Deprecated pylance warnings on using AsyncIterator with asynccontextmanager", "text": "## Purpose Fix the deprecated pylance warnings ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52090", "createdAt": "2026-08-13T03:33:09Z", "updatedAt": "2026-08-13T03:39:10Z", "timestamp": "2026-08-13T03:39:10Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "frontend", "verified" ], "author": "elizabetht", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8a3cde9ff139264751e0", "signalId": "github:vllm-project/vllm:pull_request:50180", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50180", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Skip generic required/named tool grammar for parsers that set supports_required_and_named", "text": "# Issue #49981 — `tool_choice: \"required\"` xgrammar FSM crash / infinite hang (GLM-5.2) ## 1. Root cause `ToolParser.supports_required_and_named` is the flag a tool parser sets to say *\"my model does not emit the standard tool-call JSON; do not use the generic JSON-based `required`/named handling for me.\"* GLM parsers (`Glm47MoeModelToolParser`, used for `--tool-call-parser glm45/glm47`, i.e. the GLM-4.5/4.7/5.x family) set `supports_required_and_named = False` because those models emit XML-ish tool calls plus thinking special tokens. The **parsing** side honoured that flag — `DelegatingParser.parse` treats `required`/named as `auto` and routes through the parser's `extract_tool_calls` (`vllm/parser/abstract_parser.py:433-447`). The **constraint** side did not. `ToolParser.adjust_request` unconditionally called `get_json_schema_from_tools()` and installed the resulting `{\"type\": \"array\", \"minItems\": 1, \"items\": {\"anyOf\": [...]}}` schema as `request.structured_outputs.json`, i.e. as a hard xgrammar decoding constraint. So for GLM with `tool_choice: \"required\"` the two halves disagreed: decoding was forced into a JSON grammar the model never produces, while the output was then handed to the XML parser that could not have read that JSON anyway. The failure modes reported in the issue follow directly: - **500 / `Failed to advance FSM ... grammar rejected tokens [..., 154842, ...]`** — GLM's first tokens are its native thinking/tool special tokens (154842 is one of them). The JSON grammar has no transition for them, so the FSM cannot advance and the request is terminated. - **Infinite hang with no `max_tokens`** — the array schema only reaches an accepting state once a valid JSON array is closed. With several tools the model keeps being pushed into a grammar it cannot satisfy and never emits a legal stop, so generation runs until the length cap. With no `max_tokens` set there is no cap. - **`auto` works** — no JSON schema is installed for `auto`. - **A specific named function \"works\"** in the reporter's environment because the single-function schema is small enough that the model can sometimes be coerced into it; it is broken for the same reason. Note this is only reachable when the xgrammar structural-tag path is *not* taken. With the default `VLLM_ENFORCE_STRICT_TOOL_CALLING=1`, `_apply_structural_tag` installs the `glm_4_7` builtin structural tag first and `adjust_request` returns early. Setting `VLLM_ENFORCE_STRICT_TOOL_CALLING=0` (or using any `supports_required_and_named = False` parser with no structural tag wired up, e.g. `InklingEngineToolParser`) drops into the broken JSON-schema path. This is the same defect class already fixed one-parser-at-a-time for Gemma4 (`Gemma4EngineToolParser.adjust_request`, regression after #45588), Poolside (`PoolsideV1ToolParser.adjust_request`) and the Rust bridge (`RustToolParser.adjust_request`) — each carries its own copy of the skip. GLM and Inkling never got one. ## 2. The fix `vllm/tool_parsers/abstract_tool_parser.py` — in `ToolParser.adjust_request`, return the request unchanged when `supports_required_and_named` is `False` and `tool_choice` is `\"required\"` or a named function, before the JSON schema is built and installed. Why in the base class rather than adding a fourth per-parser override: the flag already *means* \"the generic JSON required/named path does not apply to me\", and the parsing side reads it that way. Enforcing it in one place makes the two halves consistent, fixes GLM and Inkling together, and stops the next `supports_required_and_named = False` parser from re-introducing the bug. The existing per-parser overrides are left alone — they also flip `skip_special_tokens`, and they have their own regression tests. Behaviour after the fix: `required`/named on these parsers is unconstrained and parsed as `auto`, which is exactly what the serving layer already assumed. GLM still gets `skip_special_tokens = False` from `ParserEngineToolAdapter.adjust_request` → `ParserEngine.adjust_request`, so its native delimiters still reach the extractor. ## 3. Files changed | File | Change | |---|---| | `vllm/tool_parsers/abstract_tool_parser.py` | Skip the forced tool JSON schema for `required`/named when `supports_required_and_named` is `False`; import `ChatCompletionNamedToolChoiceParam` and `ToolChoiceFunction`. | | `tests/tool_parsers/test_structural_tag_registry.py` | Added `test_parser_without_required_and_named_support_skips_schema_constraints` (parametrized `required` / named), next to the existing `test_non_structural_tag_parser_uses_schema_constraints` that pins the opposite case. | | `NOTES.md` | This file. | ## 4. Risk / uncertainty - **Behaviour change:** `required` on these parsers is no longer grammar-forced. It was never actually enforcing anything useful there — it was producing output the model's own parser rejects — but a caller who relied on the constraint existing will now see an unconstrained generation. This matches what Gemma4/Poolside/Rust already do, and the default strict-tool-calling path (structural tag) is untouched and still enforces `required` properly. - **Mistral:** `MistralParser` flips `supports_required_and_named` to `False` on the *instance*, but only inside `parse`/`parse_delta`, which run after `adjust_request`; the parser is constructed per request in the renderer, so the new branch never fires for Mistral. Verified by `tests/tool_parsers/test_mistral_tool_parser.py` passing unchanged. - **Unverified:** I could not run GLM-5.2-NVFP4 end-to-end — no GPU in this environment. The 500/hang reproduction and its disappearance are argued from the code path and the reporter's token IDs, not observed on hardware. Since the change only removes a decoding constraint (never adds one) and does not touch model code, no model-quality eval is implicated; the on-GPU check worth doing before merge is a GLM-5.2 request with `tool_choice: \"required\"`, 4+ tools, `enable_thinking: false` and no `max_tokens`, under `VLLM_ENFORCE_STRICT_TOOL_CALLING=0`. ## 5. How I verified it Environment: CPU-only venv (`uv venv --python 3.12`, CPU torch 2.13 + test deps); no GPU, so only the non-GPU unit suites are runnable. ``` .venv/bin/python -m pytest tests/tool_parsers/test_structural_tag_registry.py -q → 54 passed ``` Confirmed the new test actually catches the bug — with the `abstract_tool_parser.py` change stashed and the test kept: ``` FAILED ...::test_parser_without_required_and_named_support_skips_schema_constraints[required] FAILED ...::test_parser_without_required_and_named_support_skips_schema_constraints[named] 2 failed, 52 passed ``` Regression sweep over every suite that touches tool parsing / request adjustment: ``` .venv/bin/python -m pytest tests/tool_parsers/ tests/parser/ \\ tests/tool_use/test_gemma4_responses_adjust_request.py -q → 4710 passed, 3 skipped, 34 xfailed, 15 errors .venv/bin/python -m pytest \\ tests/entrypoints/openai/responses/test_parsable_context_unit.py \\ tests/tool_use/test_chat_completion_request_validations.py \\ tests/tool_use/test_responses_request_validations.py -q → 40 passed ``` The 15 errors are all in `tests/tool_parsers/test_llama3_json_tool_parser.py` and are environment-only: fixture setup gets HTTP 401 fetching the gated `meta-llama/Llama-3.2-1B-Instruct` config from Hugging Face (no HF token here). They fail identically before and after the change. Lint: ``` ruff check → All checks passed! ruff format --diff → 2 files already formatted ``` AI assistance was used for this investigation and patch.", "url": "https://github.com/vllm-project/vllm/pull/50180", "createdAt": "2026-07-28T20:37:49Z", "updatedAt": "2026-08-13T03:36:30Z", "timestamp": "2026-08-13T03:36:30Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "tool-calling" ], "author": "pjdurden", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:459b035bcc50b21e85db", "signalId": "github:vllm-project/vllm:pull_request:52003", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52003", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Mypy Fix] Mypy fix for \"vllm/model_executor/models/[cC][dD]\"", "text": "## Purpose Mypy fix for \"vllm/model_executor/models/[cC][dD]\" ## Test ```bash pre-commit run --hook-stage manual mypy-3.13 -a Run mypy for Python 3.13.................................................Passed ```", "url": "https://github.com/vllm-project/vllm/pull/52003", "createdAt": "2026-08-12T14:55:21Z", "updatedAt": "2026-08-13T03:33:57Z", "timestamp": "2026-08-13T03:33:57Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "speculative-decoding", "ready", "multi-modality", "deepseek" ], "author": "yewentao256", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:a201e53d62807a7d2c90", "signalId": "github:vllm-project/vllm:pull_request:51316", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51316", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend][gRPC] Add RL lifecycle control", "text": "## Purpose Add reinforcement-learning lifecycle operations to the Rust frontend's existing gRPC `Control` service. - Add pause and resume RPCs for scheduler control. - Add sleep and wake RPCs for GPU memory management. - Add weight-transfer initialization, start, update, finish, and version RPCs. - Advertise RL capabilities through the EngineCore ready handshake and reject unsupported operations before dispatch. - Serialize state mutations and require generation to be paused before weight updates. - Reuse the configured NCCL, IPC, or sparse-NCCL weight-transfer backend. - Document the administrative trust boundary for the unauthenticated control surface. This lets trusted out-of-process RL sidecars use the native Rust gRPC frontend instead of the development HTTP server. It adds no listener, port, or separate weight-transfer implementation. ## Test Plan - Run Rust formatting, compilation, and Clippy checks for the affected crates. - Run the focused gRPC control tests. - Run the Python-to-Rust MessagePack compatibility test. - Compile the changed Python engine modules. - Validate the complete diff for whitespace errors. ## Test Result - `cargo fmt --all -- --check`: passed. - `cargo check -p vllm-server -p vllm-engine-core-client`: passed. - `cargo +1.96.1 clippy -p vllm-server -p vllm-engine-core-client --all-targets -- -D warnings`: passed. - `cargo test -p vllm-server grpc::tests::control_ -- --nocapture`: 4 passed. - `cargo test -p vllm-engine-core-client python_msgpack_fixtures_match_rust_encoding -- --nocapture`: 1 passed. - `uv run --no-sync python -m compileall -q vllm/v1/engine rust/src/engine-core-client/src/tests/python_compat.py`: passed. - `git diff --check upstream/main...HEAD`: passed. **AI assistance disclosure:** This PR was authored with AI assistance and reviewed by the submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described. - [x] The test plan is included. - [x] The test results are included. - [x] The necessary security and weight-transfer documentation is updated.
", "url": "https://github.com/vllm-project/vllm/pull/51316", "createdAt": "2026-08-06T21:45:34Z", "updatedAt": "2026-08-13T03:33:46Z", "timestamp": "2026-08-13T03:33:46Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "documentation", "rust" ], "author": "connorcarpenter15", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:eaf0b1c2b322f34e43df", "signalId": "github:vllm-project/vllm:pull_request:48761", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48761", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[WIP][XPU][Test]add xpu yaml", "text": "## Purpose only for xpu UT, please do not merge ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48761", "createdAt": "2026-07-15T17:34:17Z", "updatedAt": "2026-08-13T03:29:48Z", "timestamp": "2026-08-13T03:29:48Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "intel-gpu", "ci/build", "nvidia", "rust" ], "author": "wincent8", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:702024ab113063061d16", "signalId": "github:vllm-project/vllm:pull_request:51614", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51614", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KV Offload] Emit self-describing CPU events at KV-group block granularity", "text": "## Purpose Fix self-describing CPU KV events for hybrid KV-cache layouts where request hashes are computed more frequently than the full-attention group's block size. For example, DeepSeek V4 may use: - tokens_per_hash = 4, derived from the GCD of its KV-group block sizes - tokens_per_block = 256 for the full MLA group The offloading event tracker currently publishes every raw 4-token hash and sets BlockStored.block_size = 4. For one 256-token CPU chunk, this produces 64 hashes and describes them as independently stored 4-token blocks, even though the full-attention group is stored and indexed at 256-token block granularity. Consumers configured for the 256-token vLLM block size may reject or misinterpret these CPU events. ## Fix Resolve each chunk's raw request hashes from tokens_per_hash granularity to the KV group's tokens_per_block granularity using the same resolve_block_hashes() conversion used by GPU KV events. The event now: - emits the tail hash of each constituent KV-group block - sets block_size to GroupOffloadConfig.tokens_per_block - retains whole-chunk token_ids - preserves multiple constituent block hashes when blocks_per_chunk > 1 Examples with 4-token raw hashes and 256-token KV-group blocks: ``` CPU chunk size Emitted raw-hash indices Event block size ━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ 256 [63] 256 ──────────────── ────────────────────────── ────────────────── 512 [63, 127] 256 ``` This does not change GPU prefix-cache events, Mamba partial-hash hits, CPU offload keys, or CPU chunk lookup granularity. Sliding-window and SSM groups continue using their existing placeholder event path. ## Test coverage Added a parameterized regression test covering: - one 256-token block per CPU chunk - two 256-token blocks in a 512-token CPU chunk - a 4-token request-hash granularity in both cases", "url": "https://github.com/vllm-project/vllm/pull/51614", "createdAt": "2026-08-10T04:34:00Z", "updatedAt": "2026-08-13T03:28:16Z", "timestamp": "2026-08-13T03:28:16Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "bug", "ready", "kv-connector" ], "author": "ziqifan617", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:fb399b096e97ff8fa076", "signalId": "github:vllm-project/vllm:pull_request:51879", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51879", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Expose data-parallel topology to offloading backends", "text": "## Purpose Native KV-offloading backends currently receive the engine's `data_parallel_index`, but not the total number of data-parallel replicas or the process-local DP rank. Consequently, `OffloadingParallelConfig` does not contain enough information to describe the DP topology. Add `data_parallel_size` and `data_parallel_rank_local` to `OffloadingParallelConfig` and populate them from `ParallelConfig`. `data_parallel_rank_local` remains `None` when it is not set, such as outside SPMD mode. ## Changes - Add `OffloadingParallelConfig.data_parallel_size`. - Add `OffloadingParallelConfig.data_parallel_rank_local`. - Propagate both values from `build_offloading_config()`. - Update test configurations that construct `OffloadingParallelConfig` directly. - Extend the DP propagation test to verify: - `data_parallel_index = 2` - `data_parallel_size = 4` - `data_parallel_rank_local = 1` This PR only exposes existing configuration values to offloading backends. It does not change DP initialization, offloading policy, or data movement. ## Duplicate-work check Open-PR searches for `data_parallel_size offloading`, `data_parallel_rank_local offloading`, and `KV offload data parallel` found no separate PR implementing this change. The only exact match was this PR. ## Test plan ```bash .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_config.py \\ -q \\ --confcutdir=tests/v1/kv_connector/unit/offloading_connector .venv/bin/python -m pytest \\ tests/v1/kv_offload/test_factory.py \\ tests/v1/kv_offload/test_file_mapper.py \\ -q \\ --confcutdir=tests/v1/kv_offload .venv/bin/python -m pytest \\ tests/v1/kv_connector/unit/offloading_connector/test_worker.py \\ -q \\ --confcutdir=tests/v1/kv_connector/unit/offloading_connector \\ -k 'prepare_store_kv or handle_preemptions_non_writer or start_kv_transfers_non_writer or accepts_plugin_spec_default_layout' .venv/bin/python -m pytest \\ tests/v1/kv_offload/tiering/test_fs_tier.py::test_factory_forwards_locality_to_fs_tier \\ tests/v1/kv_offload/tiering/test_obj_tier.py::test_invalid_locality_raises_at_construction \\ tests/v1/kv_offload/tiering/test_obj_tier.py::test_obj_tier_replicated_layout_collapses_mapper_identity \\ -q \\ --confcutdir=tests/v1/kv_offload .venv/bin/pre-commit run ruff-format --files .venv/bin/pre-commit run ruff-check --files git diff --check ``` Results: - Offloading configuration tests: 42 passed - Factory and file-mapper tests: 55 passed - Worker tests: 6 passed, 2 deselected - Tiering tests: 4 passed - Ruff format and check: passed - `git diff --check`: passed", "url": "https://github.com/vllm-project/vllm/pull/51879", "createdAt": "2026-08-11T21:06:22Z", "updatedAt": "2026-08-13T03:27:27Z", "timestamp": "2026-08-13T03:27:27Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "kv-connector" ], "author": "ziqifan617", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1dbebc8da99878d07b19", "signalId": "github:vllm-project/vllm:pull_request:51813", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51813", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix and test EPLB balancedness calculation", "text": "## Purpose Fix EPLB balancedness logging to aggregate rank load within each MoE layer. The current reduction uses the layer axis, so it can report perfect balance when one EP rank receives all tokens in every layer. ## Test Plan ```bash pytest -q tests/distributed/test_eplb_algo.py ``` ## Test Result The new regression case has equal token totals per layer with all tokens routed to one rank. Before the fix, it produced `avg_tokens=100` and `max_tokens=100`; after the fix, it produces the expected `avg_tokens=50` and `max_tokens=200`. ```text 15 passed, 1 skipped ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] Documentation updates are not required for this metrics-only fix.
", "url": "https://github.com/vllm-project/vllm/pull/51813", "createdAt": "2026-08-11T12:03:34Z", "updatedAt": "2026-08-13T03:27:22Z", "timestamp": "2026-08-13T03:27:22Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "jdebache", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:972a2085ec7f45176243", "signalId": "github:vllm-project/vllm:pull_request:48666", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48666", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Gemma-4 FA4 FP8 Kernel", "text": " ## Purpose Gemma-4 uses 256-wide heads in `sliding_attention` and 512-wide heads in `full_attention`. On SM90, the full-attention layers upgrade from FA3 to the FA4 CuTeDSL kernel. This PR wires the FA4 FP8-KV-dequant path from [vllm-project/flash-attention#164](https://github.com/vllm-project/flash-attention/pull/164) into vLLM, allowing Gemma-4 to use FP8 KV cache across both FA3 sliding attention and FA4 full attention. It also preserves FP8 KV-cache scales for Gemma-4 MTP. The BF16 draft model uses Q-only attention over target-model KV caches; aliasing only the cache tensor left the draft attention modules with default K/V scales of `1.0`. The fix copies the target attention module's device and host K/V scales when wiring each shared cache. After this PR, when FLASH_ATTN is used, BF16 would need explicitly set version=4. FP8 should **not** explicitly set version=4, instead leave it unset or version=3. The reason is SWA FP8 layer should run with version=3, only FP8 full attention head dim 512 layer will automatically use FA4. ## Test Plan - **Accuracy:** `gemma-4-31B-it`, 4x H200, TP=1/DP=4, max model length 262144. Compare BF16 TRITON, FP8 TRITON, and FP8 FA4 with AIME 2025, GPQA Diamond, MMLU-Pro, and GSM8k. - **Serving performance:** 4x H200, TP=4, 25k shared prefix + 3k unique input tokens, OSL=200, concurrency 1-64. - **MTP:** 2x H200, TP=2, four-layer BF16 Gemma-4 assistant, three draft tokens, concurrency 32, and all 512 entries from the same SPEED-Bench dataset. Compare BF16 weights/BF16 KV against FP8 weights/FP8 KV with the scale-sharing fix. - `pre-commit run --files vllm/v1/worker/gpu/spec_decode/gemma4/speculator.py` ### Raw MTP serve commands Set the checkpoint and dataset locations for your environment: ```bash export BF16_MODEL= export FP8_MODEL= export DRAFT_MODEL= export DATASET= ``` BF16 weights and BF16 KV: ```bash vllm serve \"$BF16_MODEL\" \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 2 \\ --max-num-seqs 64 --max-num-batched-tokens 8192 --max-model-len 8192 \\ --gpu-memory-utilization 0.9 --trust-remote-code \\ --async-scheduling --enable-prefix-caching \\ --attention-backend FLASH_ATTN --attention-config.flash_attn_version=4 \\ --max-cudagraph-capture-size 256 \\ --speculative-config \"{\\\"model\\\":\\\"$DRAFT_MODEL\\\",\\\"num_speculative_tokens\\\":3}\" \\ --compilation-config '{\"use_inductor_graph_partition\":1,\"pass_config\":{\"eliminate_noops\":true,\"fuse_allreduce_rms\":true}}' \\ --hf-overrides '{\"text_config\":{\"use_bidirectional_attention\":null}}' ``` FP8 weights and FP8 KV: ```bash vllm serve \"$FP8_MODEL\" \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 2 \\ --max-num-seqs 64 --max-num-batched-tokens 8192 --max-model-len 8192 \\ --gpu-memory-utilization 0.9 --trust-remote-code \\ --async-scheduling --enable-prefix-caching \\ --attention-backend FLASH_ATTN --kv-cache-dtype fp8 \\ --max-cudagraph-capture-size 256 \\ --speculative-config \"{\\\"model\\\":\\\"$DRAFT_MODEL\\\",\\\"num_speculative_tokens\\\":3}\" \\ --compilation-config '{\"use_inductor_graph_partition\":1,\"pass_config\":{\"eliminate_noops\":true,\"fuse_allreduce_rms\":true}}' \\ --hf-overrides '{\"text_config\":{\"use_bidirectional_attention\":null}}' ``` Both MTP runs used: ```bash aiperf profile \\ --model \"$MODEL\" --tokenizer \"$MODEL\" --tokenizer-trust-remote-code \\ --endpoint-type chat --endpoint /v1/chat/completions --streaming \\ --use-legacy-max-tokens --url http://localhost:8000 \\ --input-file \"$DATASET\" \\ --extra-inputs ignore_eos:true --extra-inputs skip_special_tokens:false \\ --use-server-token-count --concurrency 32 --request-count 512 \\ --num-dataset-entries 512 --random-seed 1234 ``` ## Test Result ### Accuracy | Benchmark | BF16 TRITON | FP8 TRITON | FP8 FA4 | |---|---:|---:|---:| | AIME 2025 (avg@64) | 87.29 | 87.14 | 86.88 | | GPQA Diamond (avg@16) | 85.42 | 85.39 | 86.55 | | MMLU-Pro | 86.44 | 86.25 | 86.64 | | GSM8k (flexible-extract) | 94.62 | 94.62 | 94.47 | All configurations are within 1.2 percentage points on every benchmark. ### Serving performance Decode throughput in output tokens/s/GPU: | Concurrency | FP8 FA4 | BF16 FA4 | FP8 TRITON | BF16 TRITON | |---:|---:|---:|---:|---:| | 1 | 27.3 | 24.3 | 17.9 | 15.0 | | 16 | 186.5 | 168.4 | 105.0 | 102.5 | | 64 | 261.6 | 241.1 | 118.5 | 127.5 | FP8 FA4 is 8-12% faster than BF16 FA4 and up to 2.21x faster than FP8 TRITON across this sweep. ### MTP speculative decoding Both runs completed 512/512 requests without errors. | Configuration | Mean acceptance length | Draft-token acceptance | Output tok/s | Avg ITL (ms) | |---|---:|---:|---:|---:| | BF16 weights + BF16 KV | 3.6543 | 88.48% | 4,370.34 | 7.13 | | FP8 weights + FP8 KV | 3.6370 | 87.90% | 4,693.33 | 6.64 | The fixed FP8 run is within 0.0173 accepted tokens and 0.58 percentage points of the BF16 acceptance baseline while delivering 7.4% higher output throughput. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/48666", "createdAt": "2026-07-14T23:28:39Z", "updatedAt": "2026-08-13T03:26:49Z", "timestamp": "2026-08-13T03:26:49Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "documentation", "speculative-decoding", "ci/build", "v1", "mrv2" ], "author": "jhaotingc", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1f7cb565356868be165d", "signalId": "github:vllm-project/vllm:pull_request:51836", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51836", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[qwen3] correct linear_attn norm and a_log parameter dtype to F32 for qwen 3.5 series", "text": "## Purpose For Qwen 3.5 series (Gated Delta Net) models, the parameter `language_model.model.layers.{i}.linear_attn.norm.weight` is stored in **fp32** in the [HF checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B/tree/main?show_file_info=model.safetensors.index.json), but vLLM currently allocates it in **bf16** (for bf16 models) and silently casts the loaded values into it. If this is used with offline sharded state, the pre-sharded chedckpoints norm layer will use bf16, so the sharded checkpoint deviates from the original fp32 weight. However, Qwen 3.6 series, the linear_attn.norm weight is stored in `bf16`. We need to differentiates the two series and use the correct dtype. The same for linear attn a log parameter, for Qwen 3.5 series, the a log parameter dtype is `F32`, as well for Qwen 3.6 series, the a log parameter dtype is `BF16`. This PR adds a new text config attribute `real_model_type` demonstrating the real model type is qwen3_5 or qwen3_6. For now, the official qwen 3.5 and qwen 3.6 repo config.json does not contains `real_model_type` in `text_config`, the behavior after this pr is the same with before. But if `text_config` has the `real_model_type` field specified, and its value is, for now, `qwen3_5` or `qwen3_6`, the linear attention a_log and norm parameter will be correctly set with the weight. ## Test Plan Tested manually with offline pre shard command: ``` python examples/features/sharded_state/save_sharded_state_offline.py --model ~/.cache/huggingface/hub/models--Qwen--Qwen3.x-27B/snapshots/6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 --output /home/sharded_weigts --max-num-seqs 512 --tensor-parallel-size 2 --max-file-size 1073741824 ``` for Qwen 3.5 series: 1. the sharded output norm layer parameter is F32 2. the sharded output a log layer parameter is BF16 For Qwen 3.6 series: 1. the sharded output norm layer parameter is BF16 2. the sharded output a log layer parameter is BF16 ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51836", "createdAt": "2026-08-11T15:14:24Z", "updatedAt": "2026-08-13T03:12:25Z", "timestamp": "2026-08-13T03:12:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "qwen" ], "author": "andyxning", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9538eed7fc80fba8bf65", "signalId": "github:vllm-project/vllm:pull_request:52084", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52084", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][DSV4] Optimize sparse top-k metadata kernels for higher prefill throughput", "text": "## Purpose Optimize sparse top-k metadata kernels for higher prefill throughput ## Test ``` vllm bench serve \\ --backend vllm \\ --base-url http://localhost:8000 \\ --model deepseek-ai/DeepSeek-V4-Flash-0731 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 64 \\ --num-prompts 128 \\ --num-warmups 8 \\ --request-rate inf \\ --ignore-eos \\ --temperature 0 \\ --seed \"$seed\" \\ --save-result ``` ``` vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --enable-expert-parallel \\ --tensor-parallel-size 8 \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 \\ --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --no-enable-prefix-caching \\ --max-num-batched-tokens 16384 ``` ## Test Plan | Tokens | 128 Workers | 256 Workers | Relative Change | | ---: | ---: | ---: | ---: | | 1,024 | 9.75 us | 8.27 us | **15.2% faster** | | 4,096 | 22.40 us | 15.52 us | **30.7% faster** | | 16,384 | 74.76 us | 47.14 us | **36.9% faster** | ### Paired A/B/A Serving Results | Run | Workers | Mean Output Throughput | Mean TPOT | | --- | ---: | ---: | ---: | | A1 | 256 | 638.67 tokens/s | 102.26 ms | | B | 128 | 629.61 tokens/s | 104.80 ms | | A2 | 256 | 639.44 tokens/s | 101.88 ms | The second optimized run improved output throughput by **1.56%** and reduced mean TPOT by **2.79%** relative to the paired 128-worker baseline. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52084", "createdAt": "2026-08-13T02:57:54Z", "updatedAt": "2026-08-13T03:08:55Z", "timestamp": "2026-08-13T03:08:55Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "chaunceyjiang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c7dd376cdcca084e4b48", "signalId": "github:vllm-project/vllm:issue:41963", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:41963", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[ROCm/MI325X] DeepSeek-V4-Flash: Triton fp8_mqa_logits kernel requires 96KB shared memory, MI325X limit is 64KB", "text": "## Environment - **Hardware**: AMD MI325X × 8 (255.98 GiB HBM each) - **vLLM version**: 0.20.2rc1.dev90+g7a576e2c7 (rocm/vllm-dev:nightly, 2026-05-06) - **Model**: deepseek-ai/DeepSeek-V4-Flash - **Config**: TP=8, EP=8, `--kv-cache-dtype fp8`, `--enforce-eager` ## Bug At inference time, the first request fails with a Triton shared memory OOM: ``` triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 98304, Hardware limit: 65536. Reducing block sizes or `num_stages` may help. ``` Call stack: ``` rocm_aiter_mla_sparse.py:538 → rocm_fp8_mqa_logits aiter/ops/triton/attention/fp8_mqa_logits.py:53 → _fp8_mqa_logits_kernel ``` ## Root Cause The Triton kernel `_fp8_mqa_logits_kernel` in `aiter/ops/triton/attention/fp8_mqa_logits.py` is configured with block sizes that require **96 KB** of LDS (Local Data Share / shared memory). The AMD MI325X GPU supports a maximum of **64 KB** per workgroup. The PR #40871 was developed and tested exclusively on **MI355X**, which appears to support the larger shared memory configuration required by this kernel. ## Impact DeepSeek-V4-Flash is completely non-functional for inference on MI325X. The model loads correctly but crashes on the first forward pass. ## Suggested Fix Add a hardware-capability check before launching the kernel, and reduce `BLOCK_SIZE` or `num_stages` for GPUs with ≤64 KB shared memory. The fix should be in `aiter` (the AITER library used by vLLM ROCm). ## Related - PR #40871 \"Add AMD support for DeepSeek V4\" (merged 2026-05-05) — tested on MI355X only - Issue #41961: `mul_cuda` not implemented for `Float8_e8m0fnu` (MI325X) - Issue #41962: OOM in `rocm_dequantize_blocked_k_cache` + CUDAGraph incompatibility", "url": "https://github.com/vllm-project/vllm/issues/41963", "createdAt": "2026-05-07T14:01:05Z", "updatedAt": "2026-08-13T03:06:53Z", "timestamp": "2026-08-13T03:06:53Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "rocm", "stale" ], "author": "angelopaolosg", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:92d060a39b76ca75a1dd", "signalId": "github:vllm-project/vllm:pull_request:51323", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51323", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[1/N] HiSparse: host-resident sparse-MLA decode hot-buffering", "text": "Initial implementation based on #46326, with substantial improvements; notably, a heirarchical design which includes a standard device resident KV cache in addition to the hot buffer and host cache. Includes support for DeepSeek v4. Supersedes #46326 Preliminary results on B300: \"hisparse_pd_staged_pareto\" \"hisparse_blog_shapes\"", "url": "https://github.com/vllm-project/vllm/pull/51323", "createdAt": "2026-08-07T00:21:12Z", "updatedAt": "2026-08-13T02:58:41Z", "timestamp": "2026-08-13T02:58:41Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "documentation", "needs-rebase", "ci/build", "v1", "deepseek", "kv-connector", "nvidia", "mrv2" ], "author": "MatthewBonanni", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d024281880d8a402d614", "signalId": "github:vllm-project/vllm:pull_request:51624", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51624", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][Power] Unqualized MoE Backend for Power (VSX)", "text": "This PR adds PowerPC specific unquantized backend support for fused MoE using Power10 VSX MMA instructions. ## Purpose Currently, grouped GEMM is not supported for Power architecture in vLLM. This PR introduces a Power/VSX specific unquantized CPU backend for Fused MoE. Key features include: - Addition of `csrc/cpu/micro_gemm/cpu_micro_gemm_vsx.hpp` to support grouped GEMM on Power architectures. - Use of Power10 MMA instructions for optimized BF16 matrix multiplication. ## Test Plan - Verified mathematical correctness across all unaligned and aligned MoE kernel test cases using: ```bash pytest tests/kernels/moe/test_cpu_fused_moe.py -v ``` - Performance benchmarking using `vllm bench` with `google/gemma-4-26B-A4B-it` to compare against the `main` branch. ## Test Result The unquantized Power10 MMA implementation yields massive speedups in the prefill phase (TTFT), and provides strong throughput gains at higher batch sizes compared to the `main` branch. | Concurrency | TTFT Mean (Prefill) vs Main | Total Tok/s (Throughput) vs Main | TPOT Mean (Decode) vs Main | |-------------|-----------------------------|----------------------------------|----------------------------| | **C=1** | 35.5% faster | 1.1% faster | same as main | | **C=4** | 35.6% faster | same as main | same as main | | **C=8** | 37.0% faster | same as main | same as main | | **C=64** | 35.4% faster | 15.7% faster | 9.9% faster | | **C=128** | 35.1% faster | 13.9% faster | 10.2% faster | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51624", "createdAt": "2026-08-10T06:48:10Z", "updatedAt": "2026-08-13T02:56:10Z", "timestamp": "2026-08-13T02:56:10Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "cpu" ], "author": "Akashcodes732", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:38ca246c0c0d95016dc2", "signalId": "github:vllm-project/vllm:pull_request:44987", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44987", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] eplb", "text": "Enable XPU eplb and add TorchDistXCCLStagedEplbCommunicator for this. Torch_xccl and torch_gloo can both be used on XPU. ``` model=Qwen/Qwen3-30B-A3B VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model $model --enforce-eager --port 30088 --trust-remote-code --gpu-memory-util=0.9 --no-enable-prefix-caching --max-num-batched-tokens=8192 --no-enable-log-requests --max-model-len=8192 --block-size 64 -tp=4 \\ --enable_expert_parallel --enable_eplb --eplb-config '{\"num_redundant_experts\":32, \"step_interval\":200, \"log_balancedness\":true, \"communicator\":\"torch_xccl\", \"use_async\":false}' lm_eval --model local-completions --tasks gsm8k --model_args \"model=$model,base_url=http://localhost:30088/v1/completions,max_gen_toks=1024,num_concurrent=64,timeout=9999999\" --limit 100 ``` log: \"image\" lm_eval limits 100 results: \"image\"", "url": "https://github.com/vllm-project/vllm/pull/44987", "createdAt": "2026-06-09T09:03:45Z", "updatedAt": "2026-08-13T02:44:02Z", "timestamp": "2026-08-13T02:44:02Z", "metrics": { "reactions": 4, "comments": 5 }, "labels": [ "intel-gpu" ], "author": "mayuyuace", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2ca20322dee4a6950597", "signalId": "github:vllm-project/vllm:pull_request:52081", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52081", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"Avoid long-blocking H2D copies in ViT\" (#51841)", "text": "Reverts the changes from #51841 (\"Avoid long-blocking H2D copies in ViT\"). ## Why `Arm CPU Test` has failed on every build since #51841 merged (nightly [#83608](https://buildkite.com/vllm/ci/builds/83608), plus per-commit postmerge builds 83603 and 83607 on the same commit). It passed on build 83539, the last run before the merge. The failure is at engine startup, during `profile_run`: ``` File \"vllm/model_executor/models/qwen3_vl.py\", line 709, in rot_pos_emb pinned = torch.empty( RuntimeError: pin_memory=True requires a CUDA or other accelerator backend; no pinned memory allocator is available on this system. ``` `Qwen3_VisionTransformer.rot_pos_emb` now allocates a staging buffer with `pin_memory=True` unconditionally. On CPU-only platforms there is no pinned-memory allocator, so `torch.empty(..., pin_memory=True)` raises and `EngineCore` init dies: ``` RuntimeError: Worker failed with error 'pin_memory=True requires a CUDA or other accelerator backend; no pinned memory allocator is available on this system.' ``` ## Alternative to reverting The H2D optimization is worth keeping. A narrower fix would be to gate the pinned allocation on accelerator availability and fall back to the plain `torch.cat` path otherwise, e.g. `pin_memory=current_platform.is_pin_memory_available()`. The same guard should be considered for the per-row `mrope_positions` copy in `gpu_model_runner.py`, which relies on the source being pinned to be genuinely async. Reverting here because it restores CPU platforms immediately with no correctness or accuracy loss; a re-land with the platform guard is preferable if the author can turn it around quickly. Auto-generated by CI failure analyzer.", "url": "https://github.com/vllm-project/vllm/pull/52081", "createdAt": "2026-08-13T02:06:56Z", "updatedAt": "2026-08-13T02:36:13Z", "timestamp": "2026-08-13T02:36:13Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "qwen" ], "author": "vllm-agent", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:627e6d09dbd0230ece75", "signalId": "github:vllm-project/vllm:pull_request:51767", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51767", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[FS-Offloading] : Simplify fs manager", "text": "## Purpose The tracking of failed keys on `main` is handled in the fs/manager. This makes the manager more complicated than necessary. This PR makes failed_keys a first class citizen in the DualQueueThreadPool. The DualQueueThreadPool already maintains a JobState to track a Job's progress and reasons about failure/success based on results directly from the threadpool. With this change, the manager simply relies on the finished job results from the DualQueueThreadPool. This PR also adds batching infrasturcture to DualQueueThreadPool, but exercises it naively (same policy as main), i.e. all keys from a single request are processed by a single thread. ## Test Plan `pytest -s tests/v1/kv_offload/tiering/test_fs_tier.py` vllm serve command: ``` ++ vllm serve openai/gpt-oss-120b --tensor-parallel-size=2 --kv-transfer-config '{ \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"blocks_per_chunk\" : 1, \"spec_name\": \"TieringOffloadingSpec\", \"cpu_bytes_to_use\": 150323855360, \"eviction_policy\": \"lru\", \"secondary_tiers\": [{ \"type\": \"fs\", \"root_dir\": \"/mnt/nvme-storage/\", \"n_read_threads\": 16, \"n_write_threads\": 16 }] ,\"enable_cross_layers_blocks\" : \"True\" } }' --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --port 8000 ``` lm eval command: ``` TARGET_URL=\"http://127.0.0.1:8000\" MODEL=\"openai/gpt-oss-120b\" LM_EVAL_NUM_CONCURRENT=1000 LM_EVAL_TASKS=\"gsm8k\" lm_eval \\ --model local-completions \\ --model_args \"base_url=${TARGET_URL}/v1/completions,model=${MODEL},tokenized_requests=False,num_concurrent=${LM_EVAL_NUM_CONCURRENT},trust_remote_code=True\" \\ --tasks ${LM_EVAL_TASKS} \\ --seed 42 \\ --num_fewshot 25 \\ --gen_kwargs temperature=0.0 ``` ## Test Result Unit tests pass lm eval outputs : `Run 1 : cold cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6247|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4223|± |0.0136| ``` `Run 2 : warm cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6270|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4223|± |0.0136| ``` `Run 3 : warm cache` ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6156|± |0.0134| | | |strict-match | 25|exact_match|↑ |0.4155|± |0.0136| ``` ## Performance `guidellm command` ``` BENCH_RATE=\"64\" BENCH_RATE_TYPE=\"concurrent\" BENCH_MAX_SECONDS=\"700\" BENCH_RANDOM_SEED=\"889\" BENCH_TURNS=5 BENCH_PROMPT_TOKENS=\"4096\" BENCH_OUTPUT_TOKENS=\"512\" BENCH_PREFIX_TOKENS=\"10000\" PREFIX_COUNT=$((4 * BENCH_RATE)) #PREFIX_COUNT=$BENCH_RATE DATA=\"{\\\"kind\\\":\\\"synthetic_text\\\",\\\"prompt_tokens\\\":${BENCH_PROMPT_TOKENS},\\\"output_tokens\\\":${BENCH_OUTPUT_TOKENS},\\\"turns\\\":${BENCH_TURNS},\\\"prefix_buckets\\\": [{\\\"bucket_weight\\\": 100, \\\"prefix_count\\\": ${PREFIX_COUNT}, \\\"prefix_tokens\\\": ${BENCH_PREFIX_TOKENS}}]}\" guidellm run \\ --backend \"kind=openai_http,target=http://127.0.0.1:8000,request_format=/v1/completions\" \\ --profile \"kind=concurrent,streams=${BENCH_RATE}\" \\ --constraint \"kind=max_duration,seconds=${BENCH_MAX_SECONDS}\" \\ --seed \"kind=static,value=${BENCH_RANDOM_SEED}\" \\ --data \"$DATA\" ``` `main commit 419b51b38517bd446c6e` ``` ℹ Server Throughput Statistics (All Requests) |============|=======|======|=========|==============|===============|==============| | Benchmark | Requests ||| Input Tokens | Output Tokens | Total Tokens | | Strategy | Concurrency || Per Sec | Per Sec | Per Sec | Per Sec | | | Mdn | Mean | Mean |||| |------------|-------|------|---------|--------------|---------------|--------------| | concurrent | 64.0 | 63.7 | 3.2 | 75537.7 | 1661.7 | 77083.7 | |============|=======|======|=========|==============|===============|==============| ``` `PR` ``` ℹ Server Throughput Statistics (All Requests) |============|=======|======|=========|==============|===============|==============| | Benchmark | Requests ||| Input Tokens | Output Tokens | Total Tokens | | Strategy | Concurrency || Per Sec | Per Sec | Per Sec | Per Sec | | | Mdn | Mean | Mean |||| |------------|-------|------|---------|--------------|---------------|--------------| | concurrent | 64.0 | 63.7 | 3.2 | 75408.5 | 1647.8 | 77056.2 | |============|=======|======|=========|==============|===============|==============| ``` No regression with PR.", "url": "https://github.com/vllm-project/vllm/pull/51767", "createdAt": "2026-08-11T06:14:33Z", "updatedAt": "2026-08-13T02:34:39Z", "timestamp": "2026-08-13T02:34:39Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [], "author": "varun-sundar-rabindranath", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:adc5773ccb71051d0ce3", "signalId": "github:vllm-project/vllm:pull_request:51598", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51598", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][RFC] Enable scoped AITER W8A8 support on gfx1100", "text": "## Summary Enable scoped gfx1100-compatible AITER W8A8, GDN decode/causal-conv1d single-token, and sampler paths without widening the CDNA/RDNA4 master gates to unsupported CK/ASM operations. Addresses vllm-project/vllm#51136. ## Changes - Add independent fail-closed gates for gfx1100 W8A8, GDN decode, and lazy sampling. - Route W8A8 through public `aiter.gemm_a8w8`, accepting only dynamic per-token activation and channelwise weight scales. - Probe only the two AITER Triton symbols required for GDN decode and keep per-request generators on the native sampler fallback. - Register only the required W8A8 custom op; preserve CDNA/RDNA4 behavior and keep unrelated AITER features disabled. - Require `is_fused_moe_enabled()` before an explicit AITER MoE backend can be selected. This is a fallback fix, not gfx1100 MoE support. ## Validation The tests below were run locally on a W7900; test-file changes are intentionally excluded from this production-only PR. - gfx1100 gate, registration, layout, and MoE fallback tests: 21 passed. - Existing unquantized MoE backend-selection tests: 7 passed. - Official lazy sampler routing test: 2 passed. - AITER GDN recurrent tests: 12 passed; causal-conv1d single-token/fused-reshape tests: 22 passed. - Real top-k, top-p, and joint sampler integration: passed. - Ruff, format, and `git diff --check`: passed. ### W8A8 versus baseline - Baseline: existing vLLM Triton W8A8 kernel. - Public AITER matched the baseline numerically on all 18/18 model-observed shapes and measured `1.119x–3.786x`. - The vLLM custom op to public `aiter.gemm_a8w8` passed on a real gfx1100 GPU. ## Final-source status - Qwen3.6 W8A8 startup selected `AiterInt8ScaledMMLinearKernel`, lazy AITER sampling, and Triton/FLA GDN prefill. - An unrelated base-image vLLM ROCm skinny-GEMM assertion blocked request completion before all decode/sampling paths ran. - Final-source E2E correctness and TPS are not claimed. ## Dependencies and limitations - Depends on ROCm/aiter#4512 and the gfx1100 A8W8 config patch. - Requires an AITER-compatible Triton release and a matching gfx1100-capable vLLM ROCm extension build. - Released-package validation, final model serving, and upstream CI remain pending. ## Publication Draft PR: [vllm-project/vllm#51598](https://github.com/vllm-project/vllm/pull/51598).", "url": "https://github.com/vllm-project/vllm/pull/51598", "createdAt": "2026-08-10T02:24:11Z", "updatedAt": "2026-08-13T02:26:52Z", "timestamp": "2026-08-13T02:26:52Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "rocm" ], "author": "01xjw", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3036e78b631a1603d5d1", "signalId": "github:vllm-project/vllm:pull_request:50620", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50620", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][NIXL] Include transfer mode (push/pull) in the compatibility hash", "text": "#### Overview: Include the NIXL transfer mode (push vs pull) in the connector so a push (WRITE) connector and a pull (READ) connector can never be paired, and so an external router can distinguish them. Follow-up to #49230 (now merged), addressing review feedback from @iyastreb ([#49230 thread](https://github.com/vllm-project/vllm/pull/49230#discussion_r3686841923), [this PR's thread](https://github.com/vllm-project/vllm/pull/50620#discussion_r3702697050)). #### Details: The push (`NixlPushConnector`, WRITE) and pull (`NixlConnector`, READ) connectors use incompatible transfer protocols, but nothing prevented them from being paired across prefill/decode, and the transfer mode wasn't visible to an external router. **Compatibility hash (worker side):** - `NixlBaseConnectorWorker._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorWorker`, threaded into `compute_nixl_compatibility_hash`. - Add `transfer_mode` to the compatibility-hash factors so mismatched connectors are rejected early at handshake with a clear message. - Bump the NIXL connector version **6 → 7** and document the new factor. **Router support (scheduler side):** - `NixlBaseConnectorScheduler._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorScheduler`. - Emit `transfer_mode` in the `kv_transfer_params` returned by both the pull and push `request_finished` paths, so an external router can route pull vs push producers. Consumed by [vllm-project/router#187](https://github.com/vllm-project/router/pull/187). **Not a duplicate:** this transfer-mode work was split out of #49230 (which fixed the speculative-config factors) to keep that PR scoped; no other open PR addresses mixing push/pull connectors. **Tests run:** - `.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py -k \"transfer_mode_changes_compatibility_hash or scheduler_advertises_transfer_mode or speculative_config_compatibility_hash\" -q` — **passed**. - `pre-commit run --files ` — all applicable hooks **passed**. - GPU-dependent handshake tests were not run locally (macOS, no CUDA); relying on CI. AI assistance was used to investigate, implement, and test this change. Every changed line was reviewed by the human submitter before submission. #### Where should the reviewer start? - `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py` — `compute_nixl_compatibility_hash` (new `transfer_mode` factor + version bump). - `base_worker.py` / `push_worker.py` — worker `_TRANSFER_MODE`. - `base_scheduler.py` / `pull_scheduler.py` / `push_scheduler.py` — scheduler `_TRANSFER_MODE` emitted into `kv_transfer_params`. - `tests/v1/kv_connector/unit/test_nixl_connector.py` — hash test, scheduler-mode test, and the pull `kv_transfer_params` assertion.", "url": "https://github.com/vllm-project/vllm/pull/50620", "createdAt": "2026-07-31T19:41:03Z", "updatedAt": "2026-08-13T02:25:53Z", "timestamp": "2026-08-13T02:25:53Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "documentation", "ready", "kv-connector" ], "author": "tzulingk", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:304ff358e28bcd02d6b7", "signalId": "github:vllm-project/vllm:issue:51971", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51971", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Qwen3 MoE GPTQ `qzeros` shape mismatch on ROCm gfx1201", "text": "### Your current environment
(Run from within remote container that I started via `docker --context` from host) ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.3 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) CMake version : version 3.31.10 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.12.0+git6bbd260 Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-7.0.0-28-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : (gfx1201) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: AuthenticAMD Model name: AMD Ryzen Threadripper PRO 9995WX 96-Cores CPU family: 26 Model: 8 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 1 Stepping: 1 Frequency boost: enabled CPU max MHz: 5460.5269 CPU min MHz: 1217.1060 BogoMIPS: 4992.72 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx_vnni avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect movdiri movdir64b overflow_recov succor smca fsrm avx512_vp2intersect flush_l1d debug_swap amd_lbr_pmc_freeze Virtualization: AMD-V L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 96 MiB (96 instances) L3 cache: 384 MiB (12 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-191 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Reduced Speculation Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.3.5 [pip3] onnx==1.22.0 [pip3] onnx-ir==0.2.1 [pip3] onnxscript==0.7.1 [pip3] onnxslim==0.1.95 [pip3] pyzmq==27.1.0 [pip3] torch==2.12.0+git6bbd260 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+34c52a6 [pip3] torchvision==0.27.1+df56172 [pip3] transformers==5.14.1 [pip3] triton==3.7.1+git0263a6a6 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : 7.2.53211-c2d9476115 vLLM Version : 0.26.1rc1.dev542+gb22afe45a (git sha: b22afe45a) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU0 0 ================================= Hops between two GPUs ================================== GPU0 GPU0 0 =============================== Link Type between two GPUs =============================== GPU0 GPU0 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: -1 ================================== End of ROCm SMI Log =================================== ============================== Environment Variables ============================== LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib: PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151 MAX_JOBS=16 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
### 🐛 Describe the bug `Qwen/Qwen3-30B-A3B-GPTQ-Int4` cannot start on ROCm. vLLM reaches model warmup, then the RDNA W4A16 kernel rejects the GPTQ zero-point layout: ```text AssertionError: zp shape mismatch: torch.Size([16, 640]) vs (5120, 16) ``` Is this checkpoint layout unsupported, or should the ROCm kernel transpose/normalize `qzeros` before the assertion? ## Environment - vLLM image: `vllm/vllm-openai-rocm@sha256:8f71438ee82c2022cdca92ffc8e7d93573e65f3bf59c2d5d915a5ed7d13db1e2` - vLLM: `0.26.1rc1.dev542+gb22afe45` - Model: `Qwen/Qwen3-30B-A3B-GPTQ-Int4` (`revision=main`) - Quantization detected: `auto_gptq` - GPU: AMD Radeon AI PRO R9700, `gfx1201` (reported by host) - Platform: ROCm Docker image, single GPU, tensor parallel size `1` - Context: `32768` - GPU memory utilization: `0.90` ## Reproduction ```bash docker --context compose \\ -f compose.yaml up vllm ``` compose.yaml defines the elevant server arguments: ```text Qwen/Qwen3-30B-A3B-GPTQ-Int4 --max-model-len 32768 --gpu-memory-utilization 0.90 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes --default-chat-template-kwargs '{\"enable_thinking\":true}' ``` Relevant startup output: ```text Using RDNAHybridW4A16LinearKernel for AutoGPTQLinearMethod Layer 'model.layers.0.mlp.experts' is not supported by GPTQMoeMarlin. Falling back to Moe WNA16 kernels. Using 'TRITON' WNA16 MoE backend. Loading weights took 102.46 seconds Model loading took 15.65 GiB memory ``` Failure: ```text File .../vllm/model_executor/kernels/linear/mixed_precision/rdna_hybrid_w4a16.py, line 228 assert zp.shape == (N, num_groups), ( AssertionError: zp shape mismatch: torch.Size([16, 640]) vs (5120, 16) ``` The failure occurs during dummy warmup/KV-cache profiling, before the API becomes healthy. ## Backend tests All tests used the same pinned nightly image, model, and runtime configuration. | `--moe-backend` | Result | |---|---| | omitted / `triton` | Reaches warmup, then fails with `zp shape mismatch: torch.Size([16, 640]) vs (5120, 16)` | | `emulation` | `WNA16 MoE backend 'EMULATION' does not support the deployment configuration since the MoeWNA16 checkpoint layout is not supported.` | | `triton_unfused` | `moe_backend='triton_unfused' is not supported for WNA16 MoE. Expected one of ['triton', 'marlin', 'humming', 'flashinfer_trtllm', 'emulation'].` | | `marlin` | `WNA16 MoE backend 'MARLIN' does not support the deployment configuration since the MoeWNA16 checkpoint layout is not supported.` | | `humming` | `WNA16 MoE backend 'HUMMING' does not support the deployment configuration since kernel does not support current device rocm.` | | `flashinfer_trtllm` | `WNA16 MoE backend 'FLASHINFER_TRTLLM' does not support the deployment configuration since kernel does not support current device rocm.` | ## Questions 1. Is `Qwen3-30B-A3B-GPTQ-Int4` expected to be supported by ROCm AutoGPTQ/WNA16 kernels on gfx1201? 2. Is `qzeros` shape `[16, 640]` a known checkpoint-layout variant requiring a transpose to `[5120, 16]`? 3. Is there a patch, conversion command, or supported quantization format we should use? 4. If this is fixed upstream, which commit/tag contains the fix? The same checkpoint starts loading and consumes approximately 15.65 GiB before the kernel assertion; this is not an OOM or model-download failure. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51971", "createdAt": "2026-08-12T09:16:10Z", "updatedAt": "2026-08-13T02:25:36Z", "timestamp": "2026-08-13T02:25:36Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "rocm", "quantization" ], "author": "MartyMcFlyInTheSky", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1c0de9bc9ca33c98fd05", "signalId": "github:vllm-project/vllm:pull_request:51982", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51982", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add generic CI trace collectors", "text": "## What changed - export CMake/source → target → GNU build-ID artifact → mangled CUDA-kernel provenance from the CUDA image build - capture `ninja -t deps` and emit repo-local header → target edges - scan final nvcc object files and emit exact compiling translation-unit → mangled-kernel edges; ambiguous/generated/external objects are counted and left unmapped - persist immutable `build-graph.jsonl` and `kernel-map.jsonl` in the test image, then publish those bytes once from `image-build` with repository SHA, exact image digest, byte counts, and SHA-256 hashes - make runtime trace jobs publish only Python/GPU evidence plus hashes that reference the image-build provenance - wrap unchanged production pytest command lists with Python node/outcome and per-test execution collectors - capture fork-tree CUDA/NVTX/PyTorch evidence under pinned Nsight Systems and conservatively attribute launches as exact test, serial temporal test, or job union - add opt-in exact-node deep mode with ordered Python call/return events, CUDA launch callchains, active PyTorch/NVTX scopes, kernel identities, and native provenance - install the ordered recorder from the auto-loaded pytest trace plugin so ordinary in-process kernel tests and spawn-per-test engine tests both emit test-scoped call paths - export Nsight JSON payloads and recover launch-time Python stacks with `--cudabacktrace=kernel:0`, removing the prior 80 µs CUDA-API threshold - fail closed on missing correlated launches, callchains, Python launch stacks, test attribution, conflicting ABI aliases, or malformed frame counts ## Why The selector needs evidence from the exact production commands, image, and hardware without duplicating test definitions. Static build ownership now belongs to the immutable image build, while runtime jobs publish only facts they observe. Ambiguous runtime/native evidence remains explicit rather than being guessed. An open-PR search for CI trace collectors/test selection found no competing implementation beyond this draft. Companion ci-infra draft: https://github.com/vllm-project/ci-infra/pull/469 ## Validation - `45 passed` across `tests/tools/ci_test_selection` - all applicable repository pre-commit hooks passed for the 11 changed files - synthetic Nsight fixtures cover exact/temporal/job-union attribution, process-scoped correlation, ABI-alias deduplication, launch-time Python stacks, malformed-stack rejection, and exact object → translation-unit provenance - an independent clean replay from Buildkite #83631's raw SQLite trace produced byte-identical deep GPU, native-provenance, and summary outputs to the in-job parser - manifest tests verify exact static bytes, image digest, repository SHA, and runtime reference hashes - no model-output evaluation is applicable ## Live evidence [Buildkite #83555](https://buildkite.com/vllm/ci/builds/83555) is the accepted FlashMLA/H100 pilot: 157 passed/2 skipped, all 600,146 kernel launches exact-test attributed, 12,337 structural native edges, 9,460 artifact-to-kernel edges, and an 8/8 GNU build-ID join. [Buildkite #83563](https://buildkite.com/vllm/ci/builds/83563) is the accepted combined genericity pilot: Engine Python-only capture passed 142 nodes with zero GPU claims; Responses GPU capture passed 298 nodes with 7,198,722/7,198,723 temporal-test-attributed launches and zero job-union fallback. The integrity-clean combined graph contains 13,417 nodes/39,951 edges. [Buildkite #83631](https://buildkite.com/vllm/ci/builds/83631) is the accepted exact-node deep trace on an NVIDIA L4 at this PR's exact head `06b7bad16543ba2334cbe8cb7f26673fcb0e1b87` and ci-infra `0977a235b25fb033f8affb8a8048d4a3fad78df3`. All 3 jobs passed. The trace attributed 10,367/10,367 launches to `tests/v1/engine/test_engine_core_client.py::test_engine_core_client[False]`, recovered 9,975 exact launch-time Python stacks (96.22%, 358,491 frames), and joined all 4,333 vLLM-artifact-matched launches to exact object-derived translation units. The remaining 6,034 external/JIT launches and 392 launches without a sampleable Python context remain explicit rather than inferred. All 20 artifacts independently matched the manifest (449,377,710 bytes total), and a clean parser replay reproduced the three derived outputs byte-for-byte. The exact pytest wall time was 168s versus 1,638s in diagnostic #83614 (9.75× faster). Three short, manually auditable exact-node traces passed at this PR's current exact head `a0bd151aa56811d3897456025ae76373df16cc9a`, ci-infra `e404b7e040504f8973670a91821207dce94d96eb`, and one shared immutable image digest: - [#83643 SiluAndMul](https://buildkite.com/vllm/ci/builds/83643): 464 exact-test call/return events, 634 executed repository lines across 28 files, and 50/50 launches with exact Python stacks; all 3 vLLM-built launches joined exact translation units. - [#83646 RMSNorm](https://buildkite.com/vllm/ci/builds/83646): 500 exact-test call/return events, 670 lines across 30 files, and 98/98 launches with exact Python stacks; all 3 vLLM-built launches joined exact translation units. - [#83648 FlashMLA](https://buildkite.com/vllm/ci/builds/83648): 1,526 exact-test call/return events, 101 lines across 6 files, 4,197/4,197 exact-test launches, and 3,590 exact Python stacks (85.5%, 125,491 frames); all 605 vLLM-built launches joined exact translation units. Across these three builds, all 57 manifest files (125,298,874 bytes) independently hash-verified, all SQLite integrity checks passed, and clean parser replay reproduced every deep GPU/native/summary output byte-for-byte. The remaining 3,734 external/JIT launches are explicit and unmapped. ## Contribution notes AI assistance was used in implementation and validation. The human submitter must review every changed line and understand the design before moving this PR out of draft. This PR must remain draft. It does not merge, skip jobs, or enable production enforcement.", "url": "https://github.com/vllm-project/vllm/pull/51982", "createdAt": "2026-08-12T11:18:34Z", "updatedAt": "2026-08-13T02:25:01Z", "timestamp": "2026-08-13T02:25:01Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "ci/build" ], "author": "khluu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6a2bfc40ed5a766a3d70", "signalId": "github:vllm-project/vllm:pull_request:51700", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51700", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO)", "text": " DBO for Model Runner V2 (RFC: #50738 ) is two PRs: #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) -> #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) DBO for Model Runner V2 ([RFC #50738](https://github.com/vllm-project/vllm/issues/50738)) is two PRs: Stacked on top of #50945; the diff below contains that PR's commits too, so review only the last three (`[Feat][WIP] FULL cudagraph capture…` onward). @RyanJHamby wrote the first of them ([draft](https://github.com/RyanJHamby/vllm/commit/327520123d7a0903b28685a4282080a01dafcc2a)); the other two are the fixes needed to make it run on a GPU. ## Summary #50945 runs microbatched steps eager. This PR captures and replays them as FULL CUDA graphs: at replay there are no microbatch threads, no `threading.Event` handoffs and no per-microbatch Python, so the CPU launch path that dominates eager decode disappears and the interleaving the handoff protocol produces during capture is frozen into the graph. Scope is FULL only. `deepep_high_throughput` + DP>1 already forces `cudagraph_mode=NONE` in `compilation.py`, so capturing microbatched steps only ever applies to the decode-side backends (`deepep_low_latency` / `nixl_ep` / `allgather_reducescatter`). PIECEWISE + DBO stays unsupported, as in the V1 runner. ## Design **Candidate generation.** `_init_candidates` emits a `num_ubatches=2` twin for every FULL capture candidate, in both the uniform-decode and the mixed branch (`_maybe_ubatch_twin`). The decode branch matters: backends that only support uniform batches (MLA) get downgraded to `FULL_AND_PIECEWISE`, so their FULL graphs *are* the uniform-decode ones — generating twins only in the mixed branch captures nothing at all on those models, which is what the first GPU run of the draft showed. The gate is `UBatchRunner.wants_ubatch`, the same predicate the DP handshake votes with, so captured shapes cannot drift from dispatchable ones. It has to stay a pure function of config: ranks dispatch independently after the all-reduce, and a rank that replays while another runs eager would hang the expert all-to-all. **Dispatch.** The microbatching branch of `sync_cudagraph_and_dp_padding` stops hardcoding `cg_mode=NONE` and goes through `cudagraph_manager.dispatch(..., num_ubatches=...)`, asked with the synced uniform token count so uniform-decode graphs can match. Its existing fallback covers \"no graph for this shape\", so microbatched steps run either way. Every rank then pads to `desc.num_tokens` rather than the raw cross-rank maximum, because dispatch rounds up to the captured size. **Capture.** `UBatchRunner.run()` splits into `begin_capturable_run()` (spawn threads, wait the ready barrier — pure CPU/event work, outside the graph) and a `finish()` callback (handoff and join, inside the graph), mirroring V1's `gpu_ubatch_wrapper.py::_capture_ubatches`. Two things V1 does that a naive port misses, both of which stopped the draft from capturing anything: - the graph must be captured **on the stream the microbatch threads launch onto**. `torch.cuda.graph()` opens its own side stream unless told otherwise, so the threads' work would land outside the graph. `UBatchRunner` now owns a `capture_stream` that `CudaGraphManager` passes as `torch.cuda.graph(..., stream=...)`. - each microbatch thread initializes its CUDA and cuBLAS state **before** the capture opens — cuBLAS allocates its workspace on first use per stream and allocating mid-capture is illegal — and sets its device index, which a fresh thread does not inherit. **Attention metadata.** `UBatchRunner.prepare()` takes the descriptor's `cg_mode` instead of hardcoding `NONE`, so FULL builds per-microbatch metadata against the padded sizes; that is what keeps shapes and buffer addresses identical between capture and replay. `for_capture` is threaded through for the worst-case `max_seq_len`, as the non-ubatched path does. **Replay.** `execute_model` replays through `run_fullgraph(batch_desc)` with no threads and no `ubatch_runner.forward_context`, matching the non-ubatched FULL branch. `prepare()` still runs: it is what writes this step's `query_start_loc` / `seq_lens` into the persistent per-microbatch buffers the graph reads from. Everything else from #50945 is unchanged: `ubatching.py` and the `dbo_yield*` hooks, the slicing, the persistent buffers, the per-microbatch metadata builders, the DP handshake. That was the P0 bet — the buffers have been capture-safe from the start, so this is capture plumbing rather than a re-architecture. ## Testing DeepSeek-V2-Lite and Qwen3-30B-A3B-Thinking-2507-FP8, `DP2 / TP1 / EP2`, `nixl_ep`, 2× L20X, `VLLM_USE_V2_MODEL_RUNNER=1`. ```bash pre-commit run --files # ruff, ruff-format, mypy-3.10: pass .venv/bin/python -m pytest tests/v1/worker/test_gpu_ubatch_slicing.py -q # 19 passed (GPU), 15 passed / 4 skipped (CPU-only) ``` Capture happens: with DBO enabled, on Qwen3-30B-A3B, 0.46 → 0.96 GiB. Both DP ranks identical. Replay happens: an instrumented build logging the branch taken in `execute_model` shows microbatched graphs being replayed under real traffic on both models, and mixed batches correctly falling back to eager microbatching on MLA, which has no mixed FULL graph: ``` replay cg_mode=FULL, num_tokens=32..64, uniform_token_count=1, num_ubatches=2 plain-full cg_mode=FULL, num_tokens=1..32, uniform_token_count=1, num_ubatches=1 (below the DBO threshold) eager-ubatch cg_mode=NONE, num_tokens=609, uniform_token_count=None, num_ubatches=2 (mixed batch, MLA) ``` ## Accuracy GSM8K, 1319 questions, 5-shot, greedy. | build | DeepSeek-V2-Lite | Qwen3-30B-A3B | |---|---|---| | main (`11ba93f3`), no DBO | 0.3457 | — | | #50945, no DBO | 0.3495 | — | | #50945, `--enable-dbo` (eager) | 0.3563 | — | | this PR, no DBO | — | 0.5451 | | this PR, `--enable-dbo` + FULL CG | 0.3503 | 0.5383 | Run-to-run spread is ~1pp on this harness, so microbatching with captured graphs is accuracy-neutral on both models. For reference, an earlier broken revision scored 0.135–0.149 with a 5–6% invalid rate on DeepSeek-V2-Lite, so a regression here is unmistakable. ## Known limits - Perf validation is deferred to P4 and needs cross-node EP; the single-node numbers I can produce here are not representative of what DBO is for. - PIECEWISE + DBO remains unsupported, matching the V1 runner. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51700", "createdAt": "2026-08-10T15:44:41Z", "updatedAt": "2026-08-13T02:23:55Z", "timestamp": "2026-08-13T02:23:55Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "nvidia", "mrv2" ], "author": "specture724", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d9559c9aa47eebf84566", "signalId": "github:vllm-project/vllm:pull_request:50857", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50857", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Release] Add XPU wheel release build", "text": "## Summary This PR adds XPU wheel support to the release Buildkite pipeline. Changes included: - Add a new Buildkite release job to build and upload the x86_64 XPU wheel. - Add an `export_vllm_wheel_release` target in `docker/Dockerfile.xpu` for producing the XPU wheel artifact. - Update nightly wheel index parsing so `+xpu` local-version suffixes are recognized as an XPU wheel variant. ## Details The new release job builds the XPU image target from `docker/Dockerfile.xpu`, copies `dist/` into `artifacts/`, then reuses the existing nightly wheel upload flow: - `.buildkite/scripts/upload-nightly-wheels.sh` - `.buildkite/scripts/annotate-build-artifact.sh` The XPU wheel is built with: ```bash python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38", "url": "https://github.com/vllm-project/vllm/pull/50857", "createdAt": "2026-08-03T11:18:29Z", "updatedAt": "2026-08-13T02:23:46Z", "timestamp": "2026-08-13T02:23:46Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "intel-gpu", "ci/build" ], "author": "wendyliu235", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5f14c188bf1f34264f94", "signalId": "github:vllm-project/vllm:pull_request:51599", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51599", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(v1): decouple async Mamba align D2H counts from InputBatch row shifts (#51571)", "text": "### Target Issue Closes #51571 ### Description When running speculative decoding / MTP in `align` mode with `use_async_scheduling=True`, `GPUModelRunner._update_states_after_model_execute()` passes `input_batch.num_accepted_tokens_cpu_tensor` as the D2H target for `postprocess_mamba_align_gpu()`. However, while the GPU D2H copy is in flight on the CUDA stream, the CPU prepares the next step and calls `InputBatch.condense()`. This compacts finished requests and shifts row indices in `input_batch` *before* `num_accepted_tokens_event` is synchronized in `_prepare_inputs()`. When `_prepare_inputs()` later gathers from `input_batch.num_accepted_tokens_cpu`, it reads corrupted/shifted counts, causing Mamba hidden state copy offsets to misalign for shifted requests. This PR fixes the race condition by: 1. Directing D2H writes to `self.num_accepted_tokens.cpu` (the runner's unmutated CPU buffer snapshot) when `use_async_scheduling=True`. 2. Gathering historical accepted counts from `self.num_accepted_tokens.np` in `_prepare_inputs()`. 3. Adding a unit test (`test_async_mamba_align_accepted_counts_race`) in `tests/v1/worker/test_gpu_model_runner.py` verifying that shifted row positions preserve true historical counts. ### Test Plan - Ran unit test `pytest tests/v1/worker/test_gpu_model_runner.py -k test_async_mamba_align_accepted_counts_race` (PASSED). - Verified `ruff check` passes cleanly.", "url": "https://github.com/vllm-project/vllm/pull/51599", "createdAt": "2026-08-10T02:31:18Z", "updatedAt": "2026-08-13T02:19:32Z", "timestamp": "2026-08-13T02:19:32Z", "metrics": { "reactions": 2, "comments": 19 }, "labels": [ "ready", "mrv1-only" ], "author": "bandham-manikanta", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:877d72b387e28e9cdc5b", "signalId": "github:vllm-project/vllm:pull_request:48995", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48995", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Helion] Route fusion-only kernels to Helion during CUDA-graph capture", "text": "vLLM's post-grad fusion passes emit native fused ops (RMSNorm+quant, SiLU+mul+quant, QK-norm+RoPE) that have no eager model call site, so the capture-time call-site router used for per_token_group_fp8_quant (#47799) cannot reach them. Instead, add a single post-fusion FX pass that runs after fix_functionalization and retargets schema-compatible native fused ops to CUDA-graph-aware routed Helion ops. - vllm/kernels/helion/routing.py: build native->routed op map for the four fusion-only ops, gated on matching mutation signatures. Each routed op dispatches to Helion only while a CUDA graph is capturing and falls back to the native op otherwise. - vllm/compilation/passes/fusion/helion_routing.py: HelionFusionRoutingPass retargets both direct calls and remaining auto_functionalized nodes. - pass_manager.py: run the pass last (after defunctionalization) so vLLM's copy-elimination rules are preserved; gated on VLLM_USE_HELION_KERNELS. - Skip fused_qk_norm_rope on B200 pending the known numerical mismatch. - silu_mul_fp8 is intentionally excluded: its functional Tensor->Tensor schema is incompatible with native silu_and_mul_quant(out, ...). Tested on H100: focused routing tests (native fallback + cold CUDA-graph capture for all 4 ops) plus test_fusion, test_silu_mul_quant_fusion, and test_qk_norm_rope_fusion pass with routing enabled. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48995", "createdAt": "2026-07-17T23:31:43Z", "updatedAt": "2026-08-13T02:17:53Z", "timestamp": "2026-08-13T02:17:53Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "rocm", "needs-rebase", "v1", "nvidia", "quantization" ], "author": "yushangdi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:59424611c84d18af7ba9", "signalId": "github:vllm-project/vllm:pull_request:50945", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50945", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[1/2][Model Runner V2] DBO support, eager mode", "text": "DBO for Model Runner V2 (RFC #50738) is two PRs: -> #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) ## Purpose Relate to #50738. This PR finishes stage P0, P1, P2 ## Test Plan Benchmark results are shown in the RFC smoke.sh ```bash #!/usr/bin/env bash # Baseline (no DBO) V2 eager server. Must stay config-identical to smoke_dbo.sh # apart from the DBO flags, otherwise the A/B is meaningless. MODEL_PATH=${MODEL_PATH:-/workspace/models/hub/models--deepseek-ai--Deepseek-V2-Lite/snapshots/604d5664dddd88a0433dbae533b7fe9472482de0/} export VLLM_USE_V2_MODEL_RUNNER=1 export NVSHMEM_REMOTE_TRANSPORT=none export NVSHMEM_DISABLE_CUDA_VMM=1 export TRITON_CACHE_DIR=/tmp/triton_cache_$USER export TRITON_HOME=/tmp/triton_home_$USER export TORCHINDUCTOR_CACHE_DIR=/tmp/inductor_cache_$USER export VLLM_CACHE_ROOT=/tmp/vllm_cache_$USER DP=${DP:-2} TP=${TP:-1} PORT=${PORT:-18305} GPUS=${GPUS:-0,1} MAX_NUM_SEQS=${MAX_NUM_SEQS:-256} MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-8192} GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.85} ALL2ALL=${ALL2ALL:-deepep_high_throughput} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p /tmp/{triton_cache,triton_home,inductor_cache,vllm_cache}_$USER # shellcheck disable=SC2086 CUDA_VISIBLE_DEVICES=$GPUS vllm serve \"$MODEL_PATH\" \\ --data-parallel-size \"$DP\" \\ --tensor-parallel-size \"$TP\" \\ --enable-expert-parallel \\ --enforce-eager \\ --max-num-seqs \"$MAX_NUM_SEQS\" \\ --max-num-batched-tokens \"$MAX_NUM_BATCHED_TOKENS\" \\ --gpu-memory-utilization \"$GPU_MEM_UTIL\" \\ --host 127.0.0.1 \\ --port \"$PORT\" \\ --all2all-backend=\"$ALL2ALL\" \\ --trust-remote-code $EXTRA_ARGS ```` smoke_dbo.sh ```bash #!/usr/bin/env bash # DBO-enabled V2 eager server. Config-identical to smoke.sh apart from the DBO # flags. MODEL_PATH=${MODEL_PATH:-/workspace/models/hub/models--deepseek-ai--Deepseek-V2-Lite/snapshots/604d5664dddd88a0433dbae533b7fe9472482de0/} export VLLM_USE_V2_MODEL_RUNNER=1 export NVSHMEM_REMOTE_TRANSPORT=none export NVSHMEM_DISABLE_CUDA_VMM=1 export TRITON_CACHE_DIR=/tmp/triton_cache_$USER export TRITON_HOME=/tmp/triton_home_$USER export TORCHINDUCTOR_CACHE_DIR=/tmp/inductor_cache_$USER export VLLM_CACHE_ROOT=/tmp/vllm_cache_$USER DP=${DP:-2} TP=${TP:-1} PORT=${PORT:-18306} GPUS=${GPUS:-2,3} MAX_NUM_SEQS=${MAX_NUM_SEQS:-256} MAX_NUM_BATCHED_TOKENS=${MAX_NUM_BATCHED_TOKENS:-8192} GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.85} ALL2ALL=${ALL2ALL:-deepep_high_throughput} DBO_DECODE_THRESHOLD=${DBO_DECODE_THRESHOLD:-32} DBO_PREFILL_THRESHOLD=${DBO_PREFILL_THRESHOLD:-512} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p /tmp/{triton_cache,triton_home,inductor_cache,vllm_cache}_$USER # shellcheck disable=SC2086 CUDA_VISIBLE_DEVICES=$GPUS vllm serve \"$MODEL_PATH\" \\ --data-parallel-size \"$DP\" \\ --tensor-parallel-size \"$TP\" \\ --enable-expert-parallel \\ --enable-dbo \\ --dbo-decode-token-threshold \"$DBO_DECODE_THRESHOLD\" \\ --dbo-prefill-token-threshold \"$DBO_PREFILL_THRESHOLD\" \\ --enforce-eager \\ --max-num-seqs \"$MAX_NUM_SEQS\" \\ --max-num-batched-tokens \"$MAX_NUM_BATCHED_TOKENS\" \\ --gpu-memory-utilization \"$GPU_MEM_UTIL\" \\ --host 127.0.0.1 \\ --port \"$PORT\" \\ --all2all-backend=\"$ALL2ALL\" \\ --trust-remote-code $EXTRA_ARGS ``` bench.sh ``` #!/usr/bin/env bash # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project # # Drive load against a running AFD instance (e.g. the attention endpoint started # by decode_bench_server.sh). Thin wrapper around `vllm bench serve`; every knob # is overridable via an environment variable. # # Usage: # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh # HOST=127.0.0.1 PORT=18305 INPUT_LEN=8192 OUTPUT_LEN=256 MAX_CONCURRENCY=64 \\ # MODEL_PATH=/path/to/weights tools/benchmarks/request_generator.sh # # Variables (default): # MODEL_PATH model weights dir/name (required) # HOST server host (127.0.0.1) # PORT server port (18305) # DATASET_NAME vllm bench dataset (random) # INPUT_LEN random input length (ISL) (1024) # OUTPUT_LEN random output length (OSL) (128) # NUM_PROMPTS total prompts to send (1024) # REQUEST_RATE requests/sec (inf = as fast as can) (5) # MAX_CONCURRENCY max in-flight requests (32) # RESULT_DIR directory for saved results (/tmp/results) # RESULT_FILENAME result json filename (decode_bench.json) # EXTRA_ARGS extra args appended to the command (empty) set -euo pipefail MODEL_PATH=${MODEL_PATH:?set MODEL_PATH to the model weights dir/name} HOST=${HOST:-127.0.0.1} PORT=${PORT:-18305} DATASET_NAME=${DATASET_NAME:-random} INPUT_LEN=${INPUT_LEN:-1024} OUTPUT_LEN=${OUTPUT_LEN:-128} NUM_PROMPTS=${NUM_PROMPTS:-1024} REQUEST_RATE=${REQUEST_RATE:-inf} MAX_CONCURRENCY=${MAX_CONCURRENCY:-32} RESULT_DIR=${RESULT_DIR:-/tmp/results} RESULT_FILENAME=${RESULT_FILENAME:-decode_bench.json} SEED=${SEED:-0} EXTRA_ARGS=${EXTRA_ARGS:-} mkdir -p \"$RESULT_DIR\" # shellcheck disable=SC2086 vllm bench serve \\ --host \"$HOST\" --port \"$PORT\" \\ --model \"$MODEL_PATH\" \\ --dataset-name \"$DATASET_NAME\" \\ --random-input-len \"$INPUT_LEN\" \\ --random-output-len \"$OUTPUT_LEN\" \\ --num-prompts \"$NUM_PROMPTS\" \\ --request-rate \"$REQUEST_RATE\" \\ --max-concurrency \"$MAX_CONCURRENCY\" \\ --result-dir \"$RESULT_DIR\" \\ --result-filename \"$RESULT_FILENAME\" \\ --seed \"$SEED\" \\ --ignore-eos \\ --percentile-metrics ttft,tpot,itl,e2el \\ --save-result \\ $EXTRA_ARGS ``` ab.sh ```bash #!/usr/bin/env bash # Rigorous baseline-vs-DBO A/B. This is what produced the headline numbers. # # Both servers must already be running (see README: \"Manual reproduction\"). # Does a warmup run on each, then alternates base/DBO across several seeds so # any drift hits both arms equally, then prints per-seed and mean throughput. # # Why the defaults matter: # - prefix caching MUST be off on the servers, or repeat runs reuse identical # prompts and DBO looks up to +91% faster than it is # - >=30 s runs; sub-10 s runs are dominated by ramp-up and are not stable # - multiple seeds; a single seed is not enough to call a few-percent delta # # Usage: ab.sh [seeds...] # e.g.: ab.sh 16384 16 32 192 1 2 3 set -uo pipefail source \"$(dirname \"${BASH_SOURCE[0]}\")/env.sh\" cd \"$VLLM_DIR\" ISL=$1; OSL=$2; CONC=$3; NP=$4; shift 4 SEEDS=(\"$@\"); [ ${#SEEDS[@]} -eq 0 ] && SEEDS=(1 2 3) mkdir -p \"$RESULT_DIR\" one() { # port num_prompts seed name -> prints total token throughput MODEL_PATH=\"$MODEL_PATH\" PORT=$1 INPUT_LEN=$ISL OUTPUT_LEN=$OSL \\ MAX_CONCURRENCY=$CONC NUM_PROMPTS=$2 SEED=$3 REQUEST_RATE=inf \\ RESULT_DIR=\"$RESULT_DIR\" RESULT_FILENAME=\"$4.json\" \\ bash bench.sh > \"$RESULT_DIR/$4.log\" 2>&1 .venv/bin/python -c \" import json,sys try: d=json.load(open('$RESULT_DIR/$4.json')); print(f\\\"{d['total_token_throughput']:.1f}\\\") except Exception: print('nan')\" } echo \"workload: ISL=$ISL OSL=$OSL conc=$CONC prompts=$NP seeds=${SEEDS[*]}\" echo \"warmup...\" one \"$BASE_PORT\" $((NP / 4)) 99 warm_base >/dev/null one \"$DBO_PORT\" $((NP / 4)) 99 warm_dbo >/dev/null printf \"\\n%-6s %12s %12s %9s\\n\" seed baseline DBO gain BSUM=0; DSUM=0; WINS=0; N=0 for s in \"${SEEDS[@]}\"; do b=$(one \"$BASE_PORT\" \"$NP\" \"$s\" \"ab_base_i${ISL}_s${s}\") d=$(one \"$DBO_PORT\" \"$NP\" \"$s\" \"ab_dbo_i${ISL}_s${s}\") g=$(.venv/bin/python -c \"print(f'{($d/$b-1)*100:+.2f}%')\" 2>/dev/null || echo n/a) printf \"%-6s %12s %12s %9s\\n\" \"$s\" \"$b\" \"$d\" \"$g\" BSUM=$(.venv/bin/python -c \"print($BSUM+$b)\"); DSUM=$(.venv/bin/python -c \"print($DSUM+$d)\") WINS=$(.venv/bin/python -c \"print($WINS + (1 if $d>$b else 0))\"); N=$((N+1)) done .venv/bin/python -c \" b=$BSUM/$N; d=$DSUM/$N print(f'\\nmean {b:12.1f} {d:12.1f} {(d/b-1)*100:+8.2f}% DBO wins $WINS/$N seeds')\" ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50945", "createdAt": "2026-08-04T02:08:31Z", "updatedAt": "2026-08-13T02:17:23Z", "timestamp": "2026-08-13T02:17:23Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "nvidia", "mrv2" ], "author": "specture724", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:882381da120e644209fd", "signalId": "github:vllm-project/vllm:issue:23567", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:23567", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: openai_harmony.HarmonyError: unexpected tokens remaining in message header", "text": "### Your current environment
I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header” In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: https://github.com/openai/harmony/issues/38 But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.
### 🐛 Describe the bug
I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header” In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: https://github.com/openai/harmony/issues/38 But I haven’t found the problem at users not using vllm, so might very well be a vllm problem.
### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/23567", "createdAt": "2025-08-25T15:46:01Z", "updatedAt": "2026-08-13T02:16:11Z", "timestamp": "2026-08-13T02:16:11Z", "metrics": { "reactions": 24, "comments": 47 }, "labels": [ "bug", "stale", "gpt-oss" ], "author": "MoellerAI", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a53fbb89400d3d864918", "signalId": "github:vllm-project/vllm:pull_request:24235", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:24235", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Change the default value of truncate_prompt_tokens in the embedding/rerank/pooling model to -1", "text": "Change the default value of truncate_prompt_tokens in the embedding model to -1，By default, the model is truncated according to its maximum length. ## Purpose The client no longer needs to worry about the maximum length supported by the model and will not report an error if the input text is too long ## Test Plan work wll on bge-m3 ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/24235", "createdAt": "2025-09-04T08:56:46Z", "updatedAt": "2026-08-13T02:16:09Z", "timestamp": "2026-08-13T02:16:09Z", "metrics": { "reactions": 1, "comments": 18 }, "labels": [ "frontend", "needs-rebase", "stale" ], "author": "zhaotyer", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:48fbfe1f6583bae53a22", "signalId": "github:vllm-project/vllm:issue:26431", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:26431", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: KV Cache Quantization not working on v1 (rtx3090) \"type fp8e4nv not supported in this architecture\"", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version : Could not collect CMake version : version 4.1.0 Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.8.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.11 (main, Jun 4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.8.93 CUDA_MODULE_LOADING set to : LAZY GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 Nvidia driver version : 575.57.08 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: AuthenticAMD Model name: AMD Ryzen 5 7600 6-Core Processor CPU family: 25 Model: 97 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 2 CPU max MHz: 5170.0000 CPU min MHz: 545.0000 BogoMIPS: 7585.79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d Virtualization: AMD-V L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 6 MiB (6 instances) L3 cache: 32 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.14.1 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-ml-py==12.575.51 [pip3] nvidia-nccl-cu12==2.27.3 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pynvml==12.0.0 [pip3] pyzmq==27.1.0 [pip3] torch==2.8.0+cu128 [pip3] torchaudio==2.8.0+cu128 [pip3] torchvision==0.23.0+cu128 [pip3] transformers==4.57.0 [pip3] triton==3.4.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-11 0 N/A GPU1 PHB X 0-11 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 NCCL_VERSION=2.25.1-1 NVIDIA_DRIVER_CAPABILITIES=compute,utility NVIDIA_PRODUCT_NAME=CUDA VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=12.8.1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True LD_LIBRARY_PATH=/usr/local/cuda/lib64 CUDA_HOME=/usr/local/cuda CUDA_HOME=/usr/local/cuda PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 CUDA_MODULE_LOADING=LAZY ```
### 🐛 Describe the bug im using the vllm docker image vllm/vllm-openai:v0.11.0 (same configuration worked in v0.10.2 but it had fallback to V0 ) With this command: command: > --model jart25/Qwen3-Coder-30B-A3B-Instruct-Int4-gptq --uvicorn-log-level \"info\" --gpu-memory-utilization 0.70 --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser \"hermes\" --max-model-len 128000 --dtype \"auto\" --kv_cache_dtype fp8_e5m2 It says ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") But i never specified fp8e4nv anywhere. it works when i disable kv cache quantization This is the Error including the stacktrace (Had to remove a bit from the end, as im hitting a character limit ):
Log output ``` Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 97%|█████████▋| 65/67 [00:06<00:00, 13.95it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:06<00:00, 13.72it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:06<00:00, 10.40it/s] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m Capturing CUDA graphs (decode, FULL): 0%| | 0/35 [00:00.98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_ops.py\", line 1243, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._op(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py\", line 614, in unified_attention_with_output vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self.impl.forward(self, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/triton_attn.py\", line 303, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton_reshape_and_cache_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/ops/triton_reshape_and_cache_flash.py\", line 153, in triton_reshape_and_cache_flash vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] reshape_and_cache_kernel_flash[grid]( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 390, in vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 594, in run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] kernel = self.compile(src, target=target, options=options.__dict__) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 339, in compile vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] module = src.make_ir(options, codegen_fns, module_map, context) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 83, in make_ir vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton.compiler.errors.CompilationError: at 1:0: vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def reshape_and_cache_kernel_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] Traceback (most recent call last): vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 666, in worker_busy_loop vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] output = func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 344, in compile_or_warm_up_model vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] cuda_graph_memory_bytes = self.model_runner.capture_model() vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3463, in capture_model vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._capture_cudagraphs( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3523, in _capture_cudagraphs vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._dummy_run(num_tokens, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 120, in decorate_context vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3152, in _dummy_run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] outputs = self.model( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py\", line 121, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self.runnable(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 675, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] hidden_states = self.model(input_ids, positions, intermediate_tensors, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py\", line 317, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] model_output = self.forward(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 403, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def forward( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 375, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super().__call__(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 929, in _fn vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return fn(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_ops.py\", line 1243, in __call__ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._op(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py\", line 614, in unified_attention_with_output vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self.impl.forward(self, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/triton_attn.py\", line 303, in forward vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton_reshape_and_cache_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/attention/ops/triton_reshape_and_cache_flash.py\", line 153, in triton_reshape_and_cache_flash vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] reshape_and_cache_kernel_flash[grid]( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 390, in vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py\", line 594, in run vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] kernel = self.compile(src, target=target, options=options.__dict__) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 339, in compile vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] module = src.make_ir(options, codegen_fns, module_map, context) vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py\", line 83, in make_ir vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] triton.compiler.errors.CompilationError: at 1:0: vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def reshape_and_cache_kernel_flash( vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^ vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ValueError(\"type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')\") vllm | \u001b[1;36m(Worker_TP1 pid=77)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] WorkerProc hit an exception. vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] Traceback (most recent call last): vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py\", line 666, in worker_busy_loop vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] output = func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py\", line 344, in compile_or_warm_up_model vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] cuda_graph_memory_bytes = self.model_runner.capture_model() vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3463, in capture_model vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._capture_cudagraphs( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3523, in _capture_cudagraphs vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] self._dummy_run(num_tokens, vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py\", line 120, in decorate_context vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return func(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py\", line 3152, in _dummy_run vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] outputs = self.model( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py\", line 121, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self.runnable(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 675, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] hidden_states = self.model(input_ids, positions, intermediate_tensors, vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py\", line 317, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] model_output = self.forward(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_moe.py\", line 403, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] def forward( vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 375, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super().__call__(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py\", line 929, in _fn vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return fn(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".98\", line 1072, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 848, in call_wrapped vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._wrapped_call(self, *args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 424, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] raise e vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/fx/graph_module.py\", line 411, in __call__ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return self._call_impl(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \"/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py\", line 1784, in _call_impl vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] return forward_call(*args, **kwargs) vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] File \".2\", line 5, in forward vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query_2, key_2, value, output_3, 'model.layers.0.self_attn.attn'); query_2 = key_2 = value = output_3 = unified_attention_with_output = None vllm | \u001b[1;36m(Worker_TP0 pid=73)\u001b[0;0m ERROR 10-08 09:19:30 [v1/executor/multiproc_executor.py:671] ```
### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/26431", "createdAt": "2025-10-08T16:25:55Z", "updatedAt": "2026-08-13T02:16:05Z", "timestamp": "2026-08-13T02:16:05Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "bug", "stale" ], "author": "meganoob1337", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:9a1bd72dfdbf1a0b2287", "signalId": "github:vllm-project/vllm:issue:29134", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:29134", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Fully Async Spec-Decoding | Make `seq_lens_cpu` in CommonAttentionMetadata optional", "text": "### Proposal to improve performance Currently fully overlapping input-prep with model forward pass is blocked in the spec-decode case by the following Host<>GPU syncs: 1) `_get_valid_sampled_token_count` (ultimately needed to compute `seq_lens_cpu`): https://github.com/vllm-project/vllm/blob/fe25772aa97beb8bcb07ea49e06a2892b521a7ed/vllm/v1/worker/gpu_model_runner.py#L3109 2) in the `num_speculated_tokens > 1` case by needing to update `seq_lens_cpu` for attention metadata building (specifically for all the speculated tokens after the first one hence only impacting `num_speculated_tokens > 1` context: https://github.com/vllm-project/vllm/pull/26498) Ultimately in-order to realize fully async spec decoding we need to build attention metadata without knowing `seq_lens_cpu` (using device `seq_lens` on device is fine since any metadata building GPU kernels will get queued to after this is updated by the CUDA driver). This is currently not entirely possible for all backends (namely FlashInfer do to D2H or H2D, depending on if a host or device tensor is provided, inside the [plan function](https://github.com/flashinfer-ai/flashinfer/blob/cce4952fdd41b353325e11d99e1fc0b0737961ff/flashinfer/decode.py#L940-L955)). However, there are many (important/default) backends like TRTLLM (inside the FlashInfer backend), FlashAttn, FlashAttn-MLA (for pure decode), FlashMLA (for pure decode) etc. that could achieve this full overlap using only the device `seq_lens` tensor. The proposal would be to update common attention metadata from: ``` @dataclass class CommonAttentionMetadata: ... seq_lens: torch.Tensor seq_lens_cpu: torch.Tensor ``` To something like: ``` @dataclass class CommonAttentionMetadata: ... seq_lens: torch.Tensor _seq_lens_cpu: torch.Tensor | None # WARNING: using this property will mean spec-decode with async-scheduling will not achieve # full overlap due to Host<>GPU sync @property def seq_lens_cpu(self): # Potentially log warning here to encourage developers to avoid this property return self. _seq_lens_cpu or seq_lens.to(\"cpu\") ``` Where `_seq_lens_cpu` would be provided if available/known (e.g. async-scheduling but no-spec-decode or no-async-scheduling). This will allow backends that do not need `seq_lens_cpu` to be able to benefit more form async-scheduling while maintaining support for backends that need `seq_lens_cpu` with little to no perf regression (we could potentially add warnings to developers to encourage them to migrate away from using this). NOTE: other changes to `GPUModelRunner._update_states` would be needed to fully realize this (or model runner v2 https://github.com/vllm-project/vllm/pull/25266); but the case of `num_speculated_tokens > 1` could benefit immediately by overlapping the drafter metadata prep with the target model's forward pass. NOTE: this would also mean `max_seq_len` would no-longer represent the true max but an upper-bound that could be off by `num_speculated_tokens`; this shouldn't be a problem given most backends (e.g. FA) use this simply for heuristics. We may want to consider renaming this to `seq_len_upper_bound` in-order to make it clear to backend developers this may not be exact. NOTE: `num_computed_tokens_cpu` would also need to be avoided but a similar approach could be used considering its a derivative of `query_start_loc` and `seq_lens` ``` @dataclass class CommonAttentionMetadata: ... # WARNING: using this property will mean spec-decode with async-scheduling will not achieve # full overlap due to Host<>GPU sync @property def num_computed_tokens_cpu(self): # Potentially log warning here to encourage developers to avoid this property return seq_lens_cpu - (query_start_loc_cpu[1:] - query_start_loc_cpu[0:]) ``` cc @benchislett @WoosukKwon @MatthewBonanni", "url": "https://github.com/vllm-project/vllm/issues/29134", "createdAt": "2025-11-21T00:48:02Z", "updatedAt": "2026-08-13T02:15:59Z", "timestamp": "2026-08-13T02:15:59Z", "metrics": { "reactions": 4, "comments": 6 }, "labels": [ "performance", "stale" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1e2683c5ae93fd697a53", "signalId": "github:vllm-project/vllm:issue:29245", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:29245", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Usage]: 启动 qwen3 vl 超级超级超级慢，sglang 启动很快，可能的原因是什么？", "text": "### Your current environment 连执行 python collect_env.py 都很慢，环境是直接 uv 安装的 ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.2 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 4.1.2 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.9.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-5.10.134-19.100.al8.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.9.86 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA L20Y GPU 1: NVIDIA L20Y GPU 2: NVIDIA L20Y GPU 3: NVIDIA L20Y GPU 4: NVIDIA L20Y GPU 5: NVIDIA L20Y GPU 6: NVIDIA L20Y GPU 7: NVIDIA L20Y Nvidia driver version : 570.148.08 cuDNN version : Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.10.2 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.10.2 HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8468V CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 Stepping: 8 CPU(s) scaling MHz: 70% CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm uintr md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 192 MiB (96 instances) L3 cache: 195 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.5.2 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.8.4.1 [pip3] nvidia-cuda-cupti-cu12==12.8.90 [pip3] nvidia-cuda-nvrtc-cu12==12.8.93 [pip3] nvidia-cuda-runtime-cu12==12.8.90 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.16.0 [pip3] nvidia-cufft-cu12==11.3.3.83 [pip3] nvidia-cufile-cu12==1.13.1.3 [pip3] nvidia-curand-cu12==10.3.9.90 [pip3] nvidia-cusolver-cu12==11.7.3.90 [pip3] nvidia-cusparse-cu12==12.5.8.93 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.3.0 [pip3] nvidia-ml-py==13.580.82 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.8.93 [pip3] nvidia-nvshmem-cu12==3.3.20 [pip3] nvidia-nvtx-cu12==12.8.90 [pip3] pyzmq==27.1.0 [pip3] torch==2.9.0 [pip3] torchaudio==2.9.0 [pip3] torchvision==0.24.0 [pip3] transformers==4.57.1 [pip3] triton==3.5.0 [conda] flashinfer-python 0.5.2 pypi_0 pypi [conda] galore-torch 1.0 pypi_0 pypi [conda] nccl-xray 2.26.2rc2+xray1.1.0 pypi_0 pypi [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi [conda] nvidia-cuda-cccl-cu12 12.9.27 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cuda-nvcc-cu12 12.9.86 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi [conda] nvidia-cudnn-frontend 1.16.0 pypi_0 pypi [conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi [conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi [conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi [conda] nvidia-cutlass-dsl 4.2.1 pypi_0 pypi [conda] nvidia-ml-py 12.575.51 pypi_0 pypi [conda] nvidia-nccl-cu12 2.27.3 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi [conda] pyzmq 27.0.0 pypi_0 pypi [conda] torch 2.8.0 pypi_0 pypi [conda] torch-memory-saver 0.0.9 pypi_0 pypi [conda] torchao 0.9.0 pypi_0 pypi [conda] torchaudio 2.8.0 pypi_0 pypi [conda] torchdata 0.11.0 pypi_0 pypi [conda] torchmetrics 1.8.0 pypi_0 pypi [conda] torchvision 0.23.0 pypi_0 pypi [conda] transformer-engine-torch 2.5.0 pypi_0 pypi [conda] transformers 4.57.1 pypi_0 pypi [conda] triton 3.4.0 pypi_0 pypi ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.11.2 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 PIX NODE NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 NODE PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 NODE NODE PIX NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 NODE NODE NODE PIX SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS PIX NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS NODE PIX NODE NODE 48-95,144-191 1 N/A GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS NODE NODE PIX NODE 48-95,144-191 1 N/A GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=6,7,0,1,2,3,4,5 NCCL_IB_TC=160 CUBLAS_VERSION=12.9.1.4 NVIDIA_REQUIRE_CUDA=cuda>=9.0 NCCL_VERSION=2.27.3 NCCL_NET_GDR_LEVEL=2 NVIDIA_DRIVER_CAPABILITIES=compute,utility,video NCCL_DEBUG=INFO CUDA_ARCH_LIST=7.5 8.0 8.6 9.0 10.0 12.0 NVIDIA_PRODUCT_NAME=CUDA NCCL_IB_GID_INDEX=3 CUDA_VERSION=12.9.1.010 CUBLASMP_VERSION=0.4.0.789 CUDNN_FRONTEND_VERSION=1.12.0 CUDNN_VERSION=9.10.2.21 LD_LIBRARY_PATH=/usr/local/cuda/compat/lib.real:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_DRIVER_VERSION=575.57.08 PYTORCH_BUILD_VERSION=2.7.0 NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS= NVIDIA_PYTORCH_VERSION=25.06 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ``` 可以排除一些 IO 的问题，因为 sglang 启动很正常 ``` (trl) root@qs-44560-984753-ai-304574-default0-0:/env/trl# env CUDA_VISIBLE_DEVICES=6,7 vllm serve /path/qwen3vl-4b-1113/checkpoint-84000 --dtype auto --max-model-len 8192 -tp 2 --trust-remote-code DEBUG 11-23 04:33:31 [plugins/__init__.py:32] No plugins for group vllm.platform_plugins found. DEBUG 11-23 04:33:31 [platforms/__init__.py:36] Checking if TPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu' DEBUG 11-23 04:33:32 [platforms/__init__.py:61] Checking if CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:84] Confirmed CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:112] Checking if ROCm platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi' DEBUG 11-23 04:33:32 [platforms/__init__.py:133] Checking if XPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:153] XPU platform is not available because: No module named 'intel_extension_for_pytorch' DEBUG 11-23 04:33:32 [platforms/__init__.py:160] Checking if CPU platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:61] Checking if CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:84] Confirmed CUDA platform is available. DEBUG 11-23 04:33:32 [platforms/__init__.py:225] Automatically detected platform cuda. DEBUG 11-23 04:34:23 [utils/flashinfer.py:45] flashinfer-cubin package was not found DEBUG 11-23 04:34:50 [entrypoints/utils.py:175] Setting VLLM_WORKER_MULTIPROC_METHOD to 'spawn' DEBUG 11-23 04:34:52 [plugins/__init__.py:40] Available plugins for group vllm.general_plugins: DEBUG 11-23 04:34:52 [plugins/__init__.py:42] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver DEBUG 11-23 04:34:52 [plugins/__init__.py:45] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. INFO 11-23 04:34:52 [config/scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=161589) INFO 11-23 04:34:52 [entrypoints/openai/api_server.py:1977] vLLM API server version 0.11.2 (APIServer pid=161589) INFO 11-23 04:34:52 [entrypoints/utils.py:253] non-default args: {'model_tag': '/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000', 'model': '/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000', 'trust_remote_code': True, 'max_model_len': 8192, 'tensor_parallel_size': 2} (APIServer pid=161589) INFO 11-23 04:34:52 [engine/arg_utils.py:592] HF_HUB_OFFLINE is True, replace model_id [/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000] to model_path [/mnt/tidal-alsh01/usr/huichao/LLaMA-Factory-new/saves/titlerec/qwen3vl-4b-1113/checkpoint-84000] (APIServer pid=161589) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=161589) DEBUG 11-23 04:34:52 [model_executor/models/registry.py:652] Loaded model info for class vllm.model_executor.models.qwen3_vl.Qwen3VLForConditionalGeneration from cache (APIServer pid=161589) DEBUG 11-23 04:34:52 [logging_utils/log_time.py:29] Registry inspect model class: Elapsed time 0.0225311 secs (APIServer pid=161589) INFO 11-23 04:34:52 [config/model.py:631] Resolved architecture: Qwen3VLForConditionalGeneration (APIServer pid=161589) INFO 11-23 04:34:52 [config/model.py:1745] Using max model len 8192 (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:1958] Enabling chunked prefill by default (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:1976] Enabling prefix caching by default (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:2028] Defaulting max_num_batched_tokens to 8192 for OPENAI_API_SERVER usage context. (APIServer pid=161589) DEBUG 11-23 04:34:52 [engine/arg_utils.py:2038] Defaulting max_num_seqs to 1024 for OPENAI_API_SERVER usage context. (APIServer pid=161589) DEBUG 11-23 04:34:57 [config/parallel.py:589] Defaulting to use mp for distributed inference (APIServer pid=161589) INFO 11-23 04:34:57 [config/scheduler.py:216] Chunked prefill is enabled with max_num_batched_tokens=8192. (APIServer pid=161589) DEBUG 11-23 04:34:59 [plugins/__init__.py:32] No plugins for group vllm.stat_logger_plugins found. (APIServer pid=161589) DEBUG 11-23 04:35:02 [plugins/io_processors/__init__.py:33] No IOProcessor plugins requested by the model (APIServer pid=161589) DEBUG 11-23 04:35:12 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:22 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:32 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. (APIServer pid=161589) DEBUG 11-23 04:35:42 [v1/engine/utils.py:949] Waiting for 1 local, 0 remote core engine proc(s) to connect. ``` ### How would you like to use vllm I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29245", "createdAt": "2025-11-22T20:41:27Z", "updatedAt": "2026-08-13T02:15:57Z", "timestamp": "2026-08-13T02:15:57Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [ "usage", "stale" ], "author": "hucorz", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:fbd595f19dd7fefb32d7", "signalId": "github:vllm-project/vllm:issue:29405", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:29405", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Qwen3 Omni Transcriptions", "text": "### 🚀 The feature, motivation and pitch Would love to see this model included in the supported as a Transcription model. According to the docs it looks like only 4 different models are supported as of now. https://docs.vllm.ai/en/latest/models/supported_models/#transcription ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/29405", "createdAt": "2025-11-25T12:35:16Z", "updatedAt": "2026-08-13T02:15:52Z", "timestamp": "2026-08-13T02:15:52Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [ "good first issue", "feature request", "stale" ], "author": "justinlime", "state": "closed", "assignees": [ "mu-hashmi" ], "change": "updated" } }, { "id": "event:538cd46ebffe59b28e1d", "signalId": "github:vllm-project/vllm:issue:29688", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:29688", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "Document Ray Serve LLM incompatibility with vLLM v1 engine for PD disaggregation", "text": "## Summary vLLM v1 engine with `RayDistributedExecutor` is incompatible with Ray Serve LLM's `build_pd_openai_app` due to nested placement group conflicts. Users attempting PD (prefill/decode) disaggregation with Ray Serve LLM encounter silent failures. This should be documented to guide users toward working deployment patterns. ## Environment - **vLLM**: 0.10.0+ (v1 engine) - **Ray**: 2.43.0 - **Hardware**: AWS p5.48xlarge (8x H100 80GB) - **NIXL**: 0.7.1 ## Problem When deploying vLLM with Ray Serve LLM's PD disaggregation API: ```python from ray.serve.llm import build_pd_openai_app, LLMConfig app = build_pd_openai_app({ \"prefill_config\": prefill_llm_config, \"decode_config\": decode_llm_config, }) ``` The deployment fails with: ``` RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} ``` ## Root Cause 1. Ray Serve LLM pre-allocates GPUs in a placement group 2. vLLM v1 detects Ray environment and uses `RayDistributedExecutor` 3. `RayDistributedExecutor` attempts to create its own placement group 4. Inner placement group fails because GPUs are already reserved This creates a fundamental conflict between the two resource allocation strategies. ## Working Deployment Pattern Standalone vLLM deployment works correctly: ```bash # Prefill instance (kv_producer) python -m vllm.entrypoints.openai.api_server \\ --model Qwen/Qwen2-7B-Instruct \\ --port 8100 \\ --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"]}}' # Decode instance (kv_consumer) python -m vllm.entrypoints.openai.api_server \\ --model Qwen/Qwen2-7B-Instruct \\ --port 8200 \\ --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"]}}' ``` Both UCX and Libfabric (EFA) backends work correctly with this approach. ## Documentation Request Please add a note to the NixlConnector documentation (https://docs.vllm.ai/en/stable/features/nixl_connector_usage.html) indicating: 1. Ray Serve LLM's `build_pd_openai_app` is currently incompatible with vLLM v1 engine 2. Standalone vLLM deployment is the recommended approach for PD disaggregation 3. Users need to implement their own routing layer to coordinate prefill→decode flow ## Verified Working Configuration | Component | Version | Status | |-----------|---------|--------| | vLLM standalone | 0.10.0+ | ✅ Working | | NixlConnector + UCX | 0.7.1 | ✅ Working | | NixlConnector + Libfabric | 0.7.1 | ✅ Working | | Ray Serve LLM wrapper | 2.43.0 | ❌ Fails | ## Related Issues - Ray issue filed: ray-project/ray#59064 - #27055 - Libfabric garbage output (fixed in NIXL 0.7.1) ## Additional Context Full test suite and reproduction steps: https://github.com/dmvevents/ray-testing", "url": "https://github.com/vllm-project/vllm/issues/29688", "createdAt": "2025-11-28T16:11:35Z", "updatedAt": "2026-08-13T02:15:50Z", "timestamp": "2026-08-13T02:15:50Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [ "ray", "stale" ], "author": "dmvevents", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:5473d9e9a47890408e19", "signalId": "github:vllm-project/vllm:issue:30016", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:30016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor)", "text": "# TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor) ## Summary When deploying vLLM with `tensor_parallel_size > 1` on Ray Serve, use the **multiprocessing executor** (`distributed_executor_backend=\"mp\"`) instead of the Ray executor. This avoids placement group context issues with vLLM v1's subprocess architecture. ## Problem Description Attempting to use `tensor_parallel_size > 1` with Ray Serve and the Ray executor (`distributed_executor_backend=\"ray\"`) results in worker initialization failures: ```python # This FAILS with TP > 1 on Ray Serve: @serve.deployment(ray_actor_options={\"num_gpus\": 2}) class VLLMDeployment: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, distributed_executor_backend=\"ray\", # ❌ Fails ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) ``` **Error symptoms:** - Workers cannot find or access Ray's placement group - \"Placement group not found\" or similar initialization errors - Timeout waiting for workers to initialize ## Root Cause vLLM v1 architecture: 1. Ray Serve creates an actor with a placement group 2. vLLM spawns `EngineCore` as a **subprocess** 3. The subprocess **loses Ray's placement group context** (stored in thread-local storage) 4. When `EngineCore` tries to spawn Ray worker actors for TP, they cannot access the placement group 5. Worker initialization fails This is an architectural interaction between: - Ray's placement group context (thread-local) - vLLM v1's subprocess-based EngineCore - Ray Serve's placement group creation ## ✅ Solution: Multiprocessing Executor **Use the multiprocessing executor instead:** ```python @serve.deployment( name=\"vllm-tp2\", num_replicas=1, ray_actor_options={\"num_gpus\": 2}, ) @serve.ingress(app) class VLLMDeployment: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, distributed_executor_backend=\"mp\", # ✅ Use multiprocessing! trust_remote_code=True, enforce_eager=True, gpu_memory_utilization=0.4, ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) ``` ### Why This Works 1. **Ray Serve allocates GPUs** - `ray_actor_options={\"num_gpus\": 2}` gives the actor 2 GPUs 2. **vLLM inherits GPU visibility** - Actor has `CUDA_VISIBLE_DEVICES=0,1` 3. **Multiprocessing spawns workers** - vLLM creates worker processes using Python multiprocessing 4. **Workers inherit GPU environment** - Each worker process gets access to the GPUs 5. **Ray assigns different GPUs** - Each worker gets a different GPU via Ray's internal allocation 6. **NCCL enables communication** - Workers coordinate via NCCL for tensor parallelism **No placement groups needed** - Everything stays within the Ray Serve actor's resource allocation. ## Complete Working Example ```python #!/usr/bin/env python3 import ray from ray import serve from vllm.engine.async_llm_engine import AsyncLLMEngine from vllm import AsyncEngineArgs, SamplingParams from fastapi import FastAPI from fastapi.responses import JSONResponse ray.init(address=\"auto\") app = FastAPI() @serve.deployment( name=\"vllm-tp2\", num_replicas=1, ray_actor_options={\"num_gpus\": 2}, max_ongoing_requests=5, ) @serve.ingress(app) class VLLMWithTP2: def __init__(self): engine_args = AsyncEngineArgs( model=\"Qwen/Qwen2-7B-Instruct\", tensor_parallel_size=2, trust_remote_code=True, distributed_executor_backend=\"mp\", # KEY: multiprocessing enforce_eager=True, gpu_memory_utilization=0.4, max_model_len=2048, ) self.engine = AsyncLLMEngine.from_engine_args(engine_args) @app.post(\"/v1/chat/completions\") async def chat(self, request: dict): messages = request.get(\"messages\", []) max_tokens = min(request.get(\"max_tokens\", 50), 100) prompt = \"\\n\".join([ f\"{msg.get('role', 'user')}: {msg.get('content', '')}\" for msg in messages ]) + \"\\nassistant:\" sampling_params = SamplingParams(temperature=0.7, max_tokens=max_tokens) request_id = f\"req_{int(time.time()*1000)}\" results_generator = self.engine.generate(prompt, sampling_params, request_id) final_output = None async for output in results_generator: final_output = output text = final_output.outputs[0].text if final_output else \"\" return JSONResponse({ \"id\": request_id, \"model\": \"Qwen/Qwen2-7B-Instruct\", \"tp_size\": 2, \"executor\": \"multiprocessing\", \"choices\": [{ \"message\": {\"role\": \"assistant\", \"content\": text.strip()}, \"finish_reason\": \"stop\", }], }) @app.get(\"/health\") async def health(self): return {\"status\": \"healthy\", \"tp_size\": 2, \"executor\": \"multiprocessing\"} # Deploy serve.run(VLLMWithTP2.bind(), name=\"vllm-tp2\", route_prefix=\"/\") ``` ## Test Results **Deployment logs:** ``` Worker_TP0 pid=4701 (GPU 0) Worker_TP1 pid=4702 (GPU 1) NCCL version 2.27.3+cuda12.9 comm rank 0 nRanks 2 localRanks 2 Channel 00-23/24 configured Loading checkpoint shards: 100% | 4/4 Application ready at http://127.0.0.1:8000/ ``` **Inference test:** ```bash $ curl -X POST http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{\"messages\":[{\"role\":\"user\",\"content\":\"What is 2+2?\"}]}' { \"id\": \"req_1764813981407\", \"model\": \"Qwen/Qwen2-7B-Instruct\", \"tp_size\": 2, \"executor\": \"multiprocessing\", \"choices\": [{ \"message\": { \"role\": \"assistant\", \"content\": \"2 + 2 equals 4.\" } }] } ``` ✅ **Status:** Working perfectly with TP=2 ## Architecture Comparison ### Ray Executor (Fails with Ray Serve) ``` Ray Serve Actor └─> Placement Group (2 GPUs) └─> EngineCore Subprocess ⚠️ (loses PG context) └─> Ray Worker Actor 0 ❌ (can't find PG) └─> Ray Worker Actor 1 ❌ (can't find PG) ``` ### Multiprocessing Executor (Works) ``` Ray Serve Actor (num_gpus=2) └─> CUDA_VISIBLE_DEVICES=0,1 └─> EngineCore Subprocess (inherits CUDA vars) └─> Worker Process 0 (GPU 0) ✅ └─> Worker Process 1 (GPU 1) ✅ └─> NCCL Communication ✅ ``` ## When to Use Each Executor ### Use Multiprocessing Executor (`\"mp\"`) When: - ✅ Deploying with Ray Serve - ✅ Single-node TP (2-8 GPUs on one node) - ✅ Want simple, reliable deployment - ✅ Don't need multi-node TP ### Use Ray Executor (`\"ray\"`) When: - Multi-node TP is required (TP across nodes) - **NOT deploying with Ray Serve** (use standalone vLLM) - Need Ray's advanced scheduling features ## Limitations - **Single node only** - Multiprocessing executor works within one node - **Multi-node TP** - Would require Ray executor (but not compatible with Ray Serve) - **Recommended TP values** - 2-8 GPUs on single node For multi-node scenarios, use standalone vLLM deployment instead of Ray Serve. ## Environment - **vLLM:** v0.11.0 - **Ray:** 2.40+ - **Ray Serve:** Enabled - **GPUs:** NVIDIA H100 (AWS p5.48xlarge) - **NCCL:** 2.27.3+cuda12.9 - **CUDA:** 12.6 ## Additional Resources - **Full implementation:** https://github.com/dmvevents/ray-testing/tree/feature/tp-greater-than-1-support - **Comprehensive analysis:** https://github.com/dmvevents/ray-testing/blob/feature/tp-greater-than-1-support/TP2_SUCCESS_SUMMARY.md - **Deployment scripts:** https://github.com/dmvevents/ray-testing/blob/feature/tp-greater-than-1-support/deploy_tp2_mp_executor.py ## Related Issues This documents a **working solution** for TP > 1 with Ray Serve. No code changes needed - just use the multiprocessing executor. If multi-node TP with Ray Serve is needed in the future, that would require architectural changes to preserve placement group context across subprocess boundaries. --- **TL;DR:** Use `distributed_executor_backend=\"mp\"` for TP > 1 with Ray Serve. It works perfectly.", "url": "https://github.com/vllm-project/vllm/issues/30016", "createdAt": "2025-12-04T02:36:27Z", "updatedAt": "2026-08-13T02:15:46Z", "timestamp": "2026-08-13T02:15:46Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "ray", "stale" ], "author": "dmvevents", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:4a8aba7a46a7f9bfb37e", "signalId": "github:vllm-project/vllm:pull_request:34143", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:34143", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Fix] Bump lmcache minimum version to 0.3.11", "text": "The `get_kv_events()` API used in `LMCacheConnectorV1` is only available starting from lmcache 0.3.11. Bump the minimum version requirement to prevent runtime AttributeError. ## Purpose - Bumps the minimum `lmcache` version from `>= 0.3.9` to `>= 0.3.11` in `requirements/kv_connectors.txt` - The `get_kv_events()` API used in `LMCacheConnectorV1.get_kv_connector_kv_cache_events()` is only available starting from lmcache 0.3.11, causing a runtime `AttributeError` with older versions Fixes https://github.com/vllm-project/vllm/issues/34136 ## Test Plan N/A ## Test Result N/A ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/34143", "createdAt": "2026-02-09T14:56:35Z", "updatedAt": "2026-08-13T02:15:13Z", "timestamp": "2026-08-13T02:15:13Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "needs-rebase", "ci/build", "stale", "kv-connector" ], "author": "MohanKumar21", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:369a939f8a32be96a483", "signalId": "github:vllm-project/vllm:pull_request:34354", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:34354", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix step3p5 tool parser and unnecessary unstreamed tool args in serving.", "text": "## Purpose The step3.5 tool parser and test cases have been modified, resulting in a more stable parser. In `vllm/entrypoints/openai/chat_completion/serving.py`, some parsers do not require checking unstreamed tool arguments. For example: **qwen3coder_tool_parser**: does not maintain variables for checking unstreamed tool arguments (`streamed_args_for_tool`), may cause out of index error. **qwen3xml_tool_parser**: has some bugs during maintenance, causing duplicate parameter sending. **step3p5_tool_parser**: has the same problem as qwen3xml. The added skip logic only applies to some parsers that already have the problem and will not affect other parsers. ## Test Plan add test `/data/open/vllm/tests/tool_parsers/test_step3p5_tool_parser.py` ## Test Result test_step3p5_tool_parser: All passed. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/34354", "createdAt": "2026-02-11T16:26:45Z", "updatedAt": "2026-08-13T02:15:08Z", "timestamp": "2026-08-13T02:15:08Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "frontend", "needs-rebase", "stale", "tool-calling", "qwen" ], "author": "mariohong128", "state": "closed", "assignees": [ "chaunceyjiang" ], "change": "updated" } }, { "id": "event:d0ed1a32172284559d8d", "signalId": "github:vllm-project/vllm:pull_request:35305", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:35305", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Fix MoE g_idx params causing ValueError with actorder=null AWQ models", "text": "## Summary - Fix `CompressedTensorsWNA16MarlinMoEMethod.create_weights()` to only register `g_idx` parameters when `self.actorder` is set, preventing `ValueError` from `track_weights_loading()` when loading compressed-tensors AWQ MoE models with `actorder=null` - Fix `CompressedTensorsWNA16MoEMethod.create_weights()` to remove unreachable `g_idx` registration (class asserts `actorder != \"group\"`) - Fix `process_weights_after_loading()` to derive `num_experts`/`device` from `w13_weight_packed` instead of `w13_weight_g_idx` Fixes #35303 ## Motivation `CompressedTensorsWNA16MarlinMoEMethod.create_weights()` unconditionally registers `w13_weight_g_idx`, `w2_weight_g_idx`, `w13_g_idx_sort_indices`, and `w2_g_idx_sort_indices` as `nn.Parameter`. When `actorder` is `null` (no activation ordering), the checkpoint does not contain these tensors. The strict weight validation in `DefaultModelLoader.track_weights_loading()` then raises: ``` ValueError: Following weights were not initialized from checkpoint: { 'language_model.model.layers.0.mlp.experts.w13_weight_g_idx', 'language_model.model.layers.0.mlp.experts.w2_weight_g_idx', ... (all MoE layers) } ``` This affects any compressed-tensors AWQ MoE model with `actorder=null`, such as `cpatonn/Qwen3-VL-30B-A3B-Instruct-AWQ-4bit`. These models load successfully on v0.15.1 but crash on the v0.16.0 nightlies. The `process_weights_after_loading()` method already handles `actorder != \"group\"` by replacing these with empty tensors — but validation runs *before* that method, so the error fires first. A similar bug was previously fixed for non-MoE models (Issue #5088 / PR #5108 for GPTQ Marlin `g_idx_sort_indices`), but the MoE code path was not covered. ## Changes ### `CompressedTensorsWNA16MarlinMoEMethod` (Marlin backend) 1. **`create_weights()`**: Wrap the four `g_idx` `register_parameter()` calls in `if self.actorder:` so they are only registered when the checkpoint actually contains them. 2. **`process_weights_after_loading()`**: - Get `num_experts`/`device` from `layer.w13_weight_packed` (always present) instead of `layer.w13_weight_g_idx` (absent when actorder is null). - In the `else` branch (no actorder), use plain `torch.empty()` tensors instead of `nn.Parameter` for the empty g_idx placeholders, since they don't need gradient tracking or checkpoint loading. ### `CompressedTensorsWNA16MoEMethod` (non-Marlin backend) 3. **`create_weights()`**: Remove g_idx parameter registration entirely. This class asserts `weight_quant.actorder != \"group\"` in `__init__`, so g_idx tensors are never present in the checkpoint and should never be registered. ## Test plan - [x] Verified fix loads and runs `cpatonn/Qwen3-VL-30B-A3B-Instruct-AWQ-4bit` successfully on v0.16.0rc2.dev472 (CUDA 13.0, RTX 5090) - [ ] Existing compressed-tensors MoE tests should still pass (actorder=group path unchanged) - [ ] Models with `actorder=\"group\"` should still correctly load g_idx from checkpoint 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/35305", "createdAt": "2026-02-25T15:52:21Z", "updatedAt": "2026-08-13T02:14:55Z", "timestamp": "2026-08-13T02:14:55Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "jhsmith409", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:c524d4d4bc66246a0fac", "signalId": "github:vllm-project/vllm:issue:36222", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:36222", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Usage]: MoE flatten_tp_size should not unconditionally include dp_size — DP loses its original semantics for MoE layers", "text": "## Summary In FusedMoEParallelConfig.flatten_tp_across_dp_and_pcp (vllm/model_executor/layers/fused_moe/config.py:979-988), when EP is not enabled, dp_size is unconditionally folded into the flattened TP size: ``` flatten_tp_size = dp_size * pcp_size * tp_size flatten_tp_rank = dp_rank * pcp_size * tp_size + pcp_rank * tp_size + tp_rank ``` This means that for MoE layers, DP no longer behaves as \"data parallelism\" (independent replicas processing different data). Instead, it becomes an additional weight-sharding dimension — effectively turning DP ranks into extra TP ranks for MoE weights only. I believe this conflates two distinct parallelism concepts and leads to practical deployment issues. Concrete failure case When deploying Qwen3.5-35B-A3B-FP8 (which has intermediate_size=512 for MoE experts): TP=8, DP=1: intermediate_size_per_partition = 512 / 8 = 64, which is smaller than the FP8 block quantization size (128). This causes a quantization alignment error. TP=4, DP=2 (hoping to avoid the above issue): The MoE layer computes flatten_tp_size = 2 * 1 * 4 = 8, so intermediate_size_per_partition = 512 / 8 = 64 — the exact same problem. DP=2 does not help at all because DP is silently absorbed into TP for MoE layers. There is currently no way to deploy this model on 8 GPUs with TP=4 and have MoE layers actually use tp_size=4 (giving intermediate_size_per_partition = 512 / 4 = 128), because the flatten logic always forces MoE to shard across all devices. Semantic concern For non-MoE layers (Attention, Dense MLP), DP ranks hold identical weight replicas and process different batches independently. But for MoE layers (when EP is disabled), DP ranks hold different weight slices and must communicate to produce correct results. This means: - \"DP\" has fundamentally different meanings for MoE vs. non-MoE layers within the same model, which is confusing. - Users cannot independently control the MoE sharding strategy — it is implicitly determined by dp_size, which was chosen for throughput/batch scheduling reasons, not for MoE weight distribution. - The tight coupling prevents valid deployment configurations (as shown above). ## Proposal I suggest introducing a separate, explicit parameter to control MoE-layer parallelism independently of DP, rather than unconditionally folding dp_size into flatten_tp_size. ### How would you like to use vllm I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/36222", "createdAt": "2026-03-06T07:02:21Z", "updatedAt": "2026-08-13T02:14:41Z", "timestamp": "2026-08-13T02:14:41Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "usage", "stale" ], "author": "gerayking", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d571334d5909f5111961", "signalId": "github:vllm-project/vllm:issue:36315", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:36315", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers'", "text": "### Your current environment vllm 0.17.0 lastest. ### 🐛 Describe the bug CUDA_VISIBLE_DEVICES=1,2 vllm serve /home/ub3960x/model_data/Qwen3.5-9B \\ --tensor-parallel-size 2 \\ --gpu-memory-utilization 0.80 \\ --port 30005 \\ --served-model-name qwen3.5-9b \\ --max-model-len 16384 \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser hermes \\ --trust-remote-code (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] █ █ █▄ ▄█ (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.17.0 (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] █▄█▀ █ █ █ █ model /home/ub3960x/model_data/Qwen3.5-9B (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:302] (APIServer pid=31946) INFO 03-07 14:45:02 [utils.py:238] non-default args: {'model_tag': '/home/ub3960x/model_data/Qwen3.5-9B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'port': 30005, 'model': '/home/ub3960x/model_data/Qwen3.5-9B', 'trust_remote_code': True, 'max_model_len': 16384, 'served_model_name': ['qwen3.5-9b'], 'reasoning_parser': 'qwen3', 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.8} (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) INFO 03-07 14:45:02 [model.py:531] Resolved architecture: Qwen3ForCausalLM (APIServer pid=31946) INFO 03-07 14:45:02 [model.py:1554] Using max model len 16384 (APIServer pid=31946) INFO 03-07 14:45:02 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048. (APIServer pid=31946) INFO 03-07 14:45:02 [vllm.py:747] Asynchronous scheduling is enabled. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (APIServer pid=31946) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (EngineCore_DP0 pid=32058) INFO 03-07 14:45:08 [core.py:101] Initializing a V1 LLM engine (v0.17.0) with config: model='/home/ub3960x/model_data/Qwen3.5-9B', speculative_config=None, tokenizer='/home/ub3960x/model_data/Qwen3.5-9B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.5-9b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (EngineCore_DP0 pid=32058) WARNING 03-07 14:45:08 [multiproc_executor.py:945] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore_DP0 pid=32058) INFO 03-07 14:45:08 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.1.64 (local), world_size=2, local_world_size=2 (Worker pid=32161) INFO 03-07 14:45:14 [parallel_state.py:1393] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:42561 backend=nccl (Worker pid=32162) INFO 03-07 14:45:14 [parallel_state.py:1393] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:42561 backend=nccl (Worker pid=32161) :1325: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=32161) :1325: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=32161) INFO 03-07 14:45:14 [pynccl.py:111] vLLM is using nccl==2.27.5 (Worker pid=32162) :1325: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. (Worker pid=32162) :1325: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. (Worker pid=32161) WARNING 03-07 14:45:15 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=32162) WARNING 03-07 14:45:15 [symm_mem.py:67] SymmMemCommunicator: Device capability 12.0 not supported, communicator is not available. (Worker pid=32161) WARNING 03-07 14:45:15 [custom_all_reduce.py:165] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly. (Worker pid=32162) WARNING 03-07 14:45:15 [custom_all_reduce.py:165] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly. (Worker pid=32162) INFO 03-07 14:45:15 [parallel_state.py:1715] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A, EPLB rank N/A (Worker pid=32161) INFO 03-07 14:45:15 [parallel_state.py:1715] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (Worker pid=32161) INFO 03-07 14:45:15 [base.py:106] Offloader set to NoopOffloader (Worker pid=32162) INFO 03-07 14:45:15 [base.py:106] Offloader set to NoopOffloader (Worker pid=32161) (Worker_TP0 pid=32161) INFO 03-07 14:45:15 [gpu_model_runner.py:4255] Starting to load model /home/ub3960x/model_data/Qwen3.5-9B... (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] WorkerProc failed to start. (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] Traceback (most recent call last): (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 771, in worker_main (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] worker = WorkerProc(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 597, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.worker.load_model() (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py\", line 337, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model_runner.load_model(load_dummy_weights=dummy_weights) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4271, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = model_loader.load_model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=self.vllm_config, model_config=self.model_config (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/base_loader.py\", line 54, in load_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = initialize_model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, model_config=model_config, prefix=prefix (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/utils.py\", line 56, in initialize_model (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = model_class(vllm_config=vllm_config, prefix=prefix) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 287, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = Qwen3Model( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=maybe_prefix(prefix, \"model\") (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 256, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] super().__init__( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=prefix, decoder_layer_type=Qwen3DecoderLayer (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen2.py\", line 368, in __init__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] assert config.max_window_layers == config.num_hidden_layers, ( (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/transformers/configuration_utils.py\", line 207, in __getattribute__ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return super().__getattribute__(key) (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ (Worker pid=32162) (Worker_TP1 pid=32162) ERROR 03-07 14:45:16 [multiproc_executor.py:800] AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers' (Worker pid=32162) (Worker_TP1 pid=32162) INFO 03-07 14:45:16 [multiproc_executor.py:749] Parent process exited, terminating worker (Worker pid=32161) (Worker_TP0 pid=32161) INFO 03-07 14:45:16 [multiproc_executor.py:749] Parent process exited, terminating worker (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] WorkerProc failed to start. (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] Traceback (most recent call last): (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 771, in worker_main (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] worker = WorkerProc(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 597, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.worker.load_model() (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py\", line 337, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model_runner.load_model(load_dummy_weights=dummy_weights) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4271, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = model_loader.load_model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=self.vllm_config, model_config=self.model_config (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/base_loader.py\", line 54, in load_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = initialize_model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, model_config=model_config, prefix=prefix (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return func(*args, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/model_loader/utils.py\", line 56, in initialize_model (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] model = model_class(vllm_config=vllm_config, prefix=prefix) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 287, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] self.model = Qwen3Model( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=maybe_prefix(prefix, \"model\") (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen3.py\", line 256, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] super().__init__( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] vllm_config=vllm_config, prefix=prefix, decoder_layer_type=Qwen3DecoderLayer (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/compilation/decorators.py\", line 305, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] old_init(self, **kwargs) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/model_executor/models/qwen2.py\", line 368, in __init__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] assert config.max_window_layers == config.num_hidden_layers, ( (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/transformers/configuration_utils.py\", line 207, in __getattribute__ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] return super().__getattribute__(key) (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^ (Worker pid=32161) (Worker_TP0 pid=32161) ERROR 03-07 14:45:16 [multiproc_executor.py:800] AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers' [rank0]:[W307 14:45:17.261277764 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] EngineCore failed to start. (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] Traceback (most recent call last): (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1090, in run_engine_core (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] return func(*args, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 834, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] super().__init__( (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] vllm_config, (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ...<3 lines>... (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] internal_dp_balancing, (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 110, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self.model_executor = executor_class(vllm_config) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 100, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] super().__init__(vllm_config) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/tracing/otel.py\", line 178, in sync_wrapper (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] return func(*args, **kwargs) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/abstract.py\", line 103, in __init__ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self._init_executor() (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~~~~^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 180, in _init_executor (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] self.workers = WorkerProc.wait_for_ready(unready_workers) (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/executor/multiproc_executor.py\", line 697, in wait_for_ready (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] raise e from None (EngineCore_DP0 pid=32058) ERROR 03-07 14:45:18 [core.py:1100] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. (EngineCore_DP0 pid=32058) Process EngineCore_DP0: (EngineCore_DP0 pid=32058) Traceback (most recent call last): (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/multiprocessing/process.py\", line 313, in _bootstrap (EngineCore_DP0 pid=32058) self.run() (EngineCore_DP0 pid=32058) ~~~~~~~~^^ (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/multiprocessing/process.py\", line 108, in run (EngineCore_DP0 pid=32058) self._target(*self._args, **self._kwargs) (EngineCore_DP0 pid=32058) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1104, in run_engine_core (EngineCore_DP0 pid=32058) raise e (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/python3.13/site-packages/vllm/v1/engine/core.py\", line 1090, in run_engine_core (EngineCore_DP0 pid=32058) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore_DP0 pid=32058) File \"/home/ub3960x/miniconda3/lib/py ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/36315", "createdAt": "2026-03-07T07:37:14Z", "updatedAt": "2026-08-13T02:14:38Z", "timestamp": "2026-08-13T02:14:38Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "skfeng36", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:67f267f9a372001077b6", "signalId": "github:vllm-project/vllm:pull_request:36979", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:36979", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[refactor] Refactor SpeculativeConfig for speculative method extensibility", "text": "## Purpose Refactor `SpeculativeConfig:__post_init__` from a monolithic method into a dispatcher to family specific init methods. The current `__post_init__` interleaves the config logic for all speculative methods into a single nested if/elif chain. This makes it hard to trace a single method's setup without reading the entire function, and adding a new speculative method requires changes in multiple branches. The goal of this refactor is for each speculative 'family' to have its own method, allowing us to more easily add new speculative methods in the future: `_init_ngram_family` -> ngram/ngram_gpu (no model loading) `_init_suffix_family` -> suffix (no model loading) `_init_mtp_family` -> mtp (reuse of target model weights) `_init_eagle_family` -> eagle/eagle3/extract_hidden_states `_init_draft_model_family` -> draft_model/medusa/mlp_speculator The code for normalizing method names, autodetecting based on model name and the shared configuration for model-loading families have also been split into seperate methods. There are some behavioural changes that are all either minor or bug fixes: 1. deepseek_v32 enforce_eager=True Old: enforce_eager=True only ran when model=True New: always gets set in _init_mtp_family() 2. MTP with method='mtp', model=None and num_speculative_tokens=None Old: MTP init skipped on line 363 New: Use n_predict if available in `_init_mtp_family` -> `_init_model_config_tail` 3. Medusa method but mismatched hf_config Old: If user set method='medusa' but hf_config.model_type was not 'medusa', NotImplementedError is raised New: Gets accepted with warning (trusting user input) ## Test Plan `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/test_config.py -v` `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/config/test_model_arch_config.py -v` `FLASHINFER_DISABLE_VERSION_CHECK=1 pytest tests/v1/e2e/test_spec_decode.py -v` ## Test Result test_config.py -> All passed. test_model_arch_config.py -> All passed. test_spec_decode.py -> All passed (multi-GPU, unsupported backends and flaky tests automatically skipped. PR doesn't affect runtime so should not be an issue)", "url": "https://github.com/vllm-project/vllm/pull/36979", "createdAt": "2026-03-13T12:59:39Z", "updatedAt": "2026-08-13T02:14:25Z", "timestamp": "2026-08-13T02:14:25Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "needs-rebase", "stale", "verified" ], "author": "TQCB", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8b488a76a919591754c6", "signalId": "github:vllm-project/vllm:issue:37035", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:37035", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: cudaErrorIllegalAddress in gdn_attn.py:237 when using qwen3_next_mtp with num_speculative_tokens=5 under load", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu129 Is debug build : False CUDA used to build PyTorch : 12.9 ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.9.86 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 Nvidia driver version : 595.79 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: AuthenticAMD Model name: AMD Ryzen Threadripper 3990X 64-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 1 Stepping: 0 BogoMIPS: 5789.20 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip rdpid Virtualization: AMD-V Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 2 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 32 MiB (64 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-127 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.6 [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.9.1.4 [pip3] nvidia-cuda-cupti-cu12==12.9.79 [pip3] nvidia-cuda-nvrtc-cu12==12.9.86 [pip3] nvidia-cuda-runtime-cu12==12.9.79 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft-cu12==11.4.1.4 [pip3] nvidia-cufile-cu12==1.14.1.1 [pip3] nvidia-curand-cu12==10.3.10.19 [pip3] nvidia-cusolver-cu12==11.7.5.82 [pip3] nvidia-cusparse-cu12==12.5.10.65 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-cutlass-dsl==4.4.1 [pip3] nvidia-cutlass-dsl-libs-base==4.4.1 [pip3] nvidia-ml-py==13.590.48 [pip3] nvidia-nccl-cu12==2.27.5 [pip3] nvidia-nvjitlink-cu12==12.9.86 [pip3] nvidia-nvshmem-cu12==3.4.5 [pip3] nvidia-nvtx-cu12==12.9.79 [pip3] pyzmq==27.1.0 [pip3] torch==2.10.0+cu129 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.10.0+cu129 [pip3] torchvision==0.25.0+cu129 [pip3] transformers==4.57.6 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.17.1rc1.dev126+gbc2c0c86e (git sha: bc2c0c86e) vLLM Build Flags: CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ENABLE_CUDAGRAPH_GC=1 VLLM_USAGE_SOURCE=production-docker-image TORCHINDUCTOR_CACHE_DIR=/root/.cache/torch/inductor VLLM_USE_FLASHINFER_SAMPLER=1 CUDA_VERSION=12.9.1 VLLM_ENABLE_CUDA_COMPATIBILITY=0 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 ```
- **vLLM version**: `v0.17.1rc1.dev126+gbc2c0c86e` (nightly, `vllm/vllm-openai:nightly`) - **Model**: `cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4` - **Quantization**: `compressed-tensors` (AWQ BF16 INT4) - **Hardware**: 2× NVIDIA GPU (tensor-parallel-size=2), WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2) - **Attention backend**: FlashInfer (`--attention-backend FLASHINFER`) - **Python**: 3.12 - **PyTorch**: inferred from nightly image ### 🐛 Describe the bug ## Summary vLLM crashes with `CUDA error: an illegal memory access was encountered` in `gdn_attn.py:237` (`spec_state_indices_tensor = block_table_tensor[...]`) when running `qwen3_next_mtp` speculative decoding with `num_speculative_tokens=5` and FlashInfer attention backend under concurrent load. Both TP workers (TP0 and TP1) crash with the same error. The crash occurs consistently once the request queue fills (~500 waiting requests) and KV cache usage exceeds ~63%. The crash reproduces reliably across multiple runs (observed twice within a single session, ~7 minutes apart). ## Suspected Root Cause FlashInfer PR [#2679](https://github.com/flashinfer-ai/flashinfer/pull/2679) (\"feat(gdn): add BF16 state kernel with MTP support beyond T>4\") is open and unmerged as of this report. It specifically adds GDN MTP kernel support for **T>4 speculative tokens**. With `num_speculative_tokens=5` (T=5), the existing GDN kernel likely has an OOB access when building `spec_state_indices_tensor` from the block table, which manifests as `cudaErrorIllegalAddress`. Related open issue: [vllm#36613](https://github.com/vllm-project/vllm/issues/36613) (same error class, different model/config). ## Reproduction ### Docker Compose (vLLM service definition) ```yaml vllm: image: vllm/vllm-openai:nightly container_name: vllm ipc: host ports: - \"8000:8000\" volumes: - vllm-models:/root/.cache/huggingface - vllm-compile:/root/.cache/vllm - vllm-flashinfer:/root/.cache/flashinfer - vllm-torch:/root/.cache/torch - vllm-triton:/root/.triton environment: - VLLM_WORKER_MULTIPROC_METHOD=spawn - VLLM_SLEEP_WHEN_IDLE=1 - VLLM_ENABLE_CUDAGRAPH_GC=1 - VLLM_USE_FLASHINFER_SAMPLER=1 - RAY_memory_monitor_refresh_ms=0 - TORCHINDUCTOR_CACHE_DIR=/root/.cache/torch/inductor deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] command: > -O3 --model cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4 --served-model-name local --quantization compressed-tensors --max-model-len=100000 --max-num-seqs=8 --block-size=32 --max-num-batched-tokens=2048 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --attention-backend FLASHINFER --tensor-parallel-size=2 --gpu-memory-utilization=0.90 --trust-remote-code --disable-custom-all-reduce --speculative-config '{\"method\":\"qwen3_next_mtp\",\"num_speculative_tokens\":5}' ``` ### Benchmark command (triggers the crash under load) ```bash docker run --rm -it \\ --ipc=host --network=host \\ -v \"./guidellm:/results:rw\" \\ ghcr.io/vllm-project/guidellm:latest -- benchmark \\ --target \"http://127.0.0.1:8000\" \\ --profile sweep \\ --max-seconds 30 \\ --warmup 0.1 \\ --cooldown 0.1 \\ --max-errors 5 \\ --detect-saturation \\ --data \"abisee/cnn_dailymail\" \\ --data-args '{\"name\": \"3.0.0\"}' \\ --data-column-mapper '{\"text_column\":\"article\"}' ``` The crash occurs once the queue saturates (~504 waiting requests, KV cache ~63–66%). ## Stack Trace (Crash 1 — Worker_TP0, full Python traceback available) ``` (Worker_TP0 pid=409) ERROR 03-14 03:28:43 [multiproc_executor.py:932] Traceback (most recent call last): File \".../vllm/v1/worker/gpu_model_runner.py\", line 3254, in synchronize_input_prep yield File \".../vllm/v1/worker/gpu_model_runner.py\", line 3723, in execute_model self._build_attention_metadata( File \".../vllm/v1/worker/gpu_model_runner.py\", line 2102, in _build_attention_metadata _build_attn_group_metadata(kv_cache_gid, attn_gid, cm) File \".../vllm/v1/worker/gpu_model_runner.py\", line 2053, in _build_attn_group_metadata attn_metadata_i = builder.build( File \".../vllm/v1/attention/backends/gdn_attn.py\", line 237, in build spec_state_indices_tensor = block_table_tensor[ ^^^^^^^^^^^^^^^^^^^ torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` Both TP0 and TP1 workers raise the same error at `gdn_attn.py:237`. The secondary traceback (after exception propagation) surfaces at: ``` File \".../vllm/v1/worker/gpu_model_runner.py\", line 3256, in synchronize_input_prep self.prepare_inputs_event.record() torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` Followed by: ``` (EngineCore pid=210) ERROR [multiproc_executor.py:273] Worker proc VllmWorker-0 died unexpectedly, shutting down executor. RuntimeError: cancelled vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. ``` ## Crash 2 (7 minutes later, same session) Identical error, but this time `VllmWorker-1` dies first and the Python traceback is replaced by a C++ `terminate called after throwing an instance of 'c10::AcceleratorError'`. Conditions: 8 running, 504 waiting, KV cache 65.4%. ## Scheduler State at Crash Both crashes share the same scheduler snapshot: - `num_running_reqs=8`, `num_waiting_reqs=504` - `kv_cache_usage≈0.63–0.65` - All 8 requests had `scheduled_spec_decode_tokens: [-1, -1, -1, -1, -1]` (5 spec tokens, all rejected/pending) - `step_counter=0` — crash on the very first step after restart ## Confirmed Bisection | `num_speculative_tokens` | Result | |---|---| | 5 | **Crashes** — `cudaErrorIllegalAddress` in `gdn_attn.py:237` under load | | 4 | **Stable** — no crash observed under identical load | This confirms the T>4 boundary as the root cause. The GDN FlashInfer kernel does not correctly support 5+ speculative tokens, consistent with flashinfer PR [#2679](https://github.com/flashinfer-ai/flashinfer/pull/2679) which adds that support but is not yet merged. ## Workaround Set `num_speculative_tokens` to 4 or fewer. Alternatively, disable MTP entirely by removing `--speculative-config`. ## Attached Logs - [bug_report_crash1.log](https://pastebin.com/gq5kkWiD) — full Worker_TP0/TP1 output for crash 1 (Python traceback, C++ frames) - [bug_report_crash2.log](https://pastebin.com/iA5DVgAG) — crash 2 output (C++ terminate, no Python traceback) ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37035", "createdAt": "2026-03-14T04:10:52Z", "updatedAt": "2026-08-13T02:14:24Z", "timestamp": "2026-08-13T02:14:24Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "unstale" ], "author": "Quentin-M", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:94105bf314fccd1d1020", "signalId": "github:vllm-project/vllm:pull_request:37227", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:37227", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Use list.extend() over append loops in FlatLogprobs + minor hot-path cleanups", "text": "Minor Python-level performance optimizations in hot paths: 1. **FlatLogprobs.append_fast**: Replace per-item append() loop with list.extend() — ~1.37x speedup on the per-token logprobs path. 2. **common_prefix**: Compare elements by index instead of creating slice copies — ~9.4x speedup on long shared prefixes. 3. **FlatLogprobs.__iter__**: Use self[i] instead of self.__getitem__(i), remove redundant range(0, ...). 4. **OutputProcessor.propagate_error**: Use dict.values() instead of dict.items() when key is unused — ~1.3x speedup. All changes are semantically identical refactors. Existing tests pass.", "url": "https://github.com/vllm-project/vllm/pull/37227", "createdAt": "2026-03-16T20:49:32Z", "updatedAt": "2026-08-13T02:14:16Z", "timestamp": "2026-08-13T02:14:16Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "stale", "v1" ], "author": "vaibhavhariram", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f017c2fa99b1058b4ff0", "signalId": "github:vllm-project/vllm:issue:37271", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:37271", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: In_proj_ba of GDN in Qwen3Next use MergeColumnParallelLinear may cause accuracy decrease?", "text": "### Your current environment
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug I found that in_proj_ba is a MergeColumnParallelLinear now, and it was a ColumnParallelLinear before. It leads to a different way of weight loading. And it will cause accuracy decrease on gsm8k dataset. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/37271", "createdAt": "2026-03-17T08:39:37Z", "updatedAt": "2026-08-13T02:14:12Z", "timestamp": "2026-08-13T02:14:12Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "bug", "stale" ], "author": "SunnyLee151064", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f9ee4c03972af75d3d40", "signalId": "github:vllm-project/vllm:pull_request:37390", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:37390", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Quark OCP-MX W4A6 support: dequant dtype + apply_weights", "text": " Fixes two issues preventing Quark W4A6 models (MXFP4 weights / MXFP6 activations, e.g. `ziliangpeng/DeepSeek-V3-Quark-MXFP4-v4-w4a6`) from running on ROCm: 1. **MoE QuantType crash** (`rocm_aiter_moe.py`): W4A6 models set `weight_quant_dtype=\"mxfp4\"` but not the `use_mxfp4_w4a4/w4a16` flags, causing AITER MoE to receive `QuantType.NO` and crash with \"Unsupported kernel config for moe heuristic dispatch\". Fixed by adding a fallback check on `weight_quant_dtype`. This is the caller-side fix suggested by @valarLip in ROCm/aiter#2457. 2. **Emulate path byte tensor crash** (`quark_ocp_mx.py`): In the emulation path (used when native MXFP6 kernels are not available), input `x` can arrive as `uint8` from upstream quantized tensor flow. This causes `F.linear(uint8, uint8)` to crash and the Quark dequant kernel to reject the dtype. Fixed with defensive dtype handling: non-floating-point inputs are cast to `bfloat16` before dequantization. The activation QDQ path (`quant_dequant_func`) is preserved for accuracy. ### Test Plan **Hardware:** AMD Instinct MI350X / MI355X (gfx950), 8xGPU **Model:** `ziliangpeng/DeepSeek-V3-Quark-MXFP4-v4-w4a6` **1. Build and install vLLM with this PR:** ```bash # Start ROCm container docker run -it --network=host --device=/dev/kfd --device=/dev/dri \\ --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \\ -v /path/to/models:/workspace/models \\ vllm/vllm-openai-rocm:latest bash # Inside container: clone and install the PR branch git clone https://github.com/vecheruk-amd/vllm.git -b fix/quark-w4a6-mxfp4-compat cd vllm && pip install -e . ``` **2. Set environment variables:** ```bash export VLLM_ROCM_USE_AITER=1 ``` **3. Serve the model (TP=8, eager mode):** ```bash vllm serve /workspace/models/DeepSeek-V3-Quark-MXFP4-v4-w4a6 \\ --tensor-parallel-size 8 \\ --block-size 1 \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.90 \\ --max-model-len 32000 \\ --enforce-eager \\ --port 8000 ``` **4. Test inference:** ```bash curl http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{\"model\": \"/workspace/models/DeepSeek-V3-Quark-MXFP4-v4-w4a6\", \"messages\": [{\"role\":\"user\",\"content\":\"Explain quantum computing\"}], \"max_tokens\": 256}' ``` **Without this PR:** Step 3 crashes with \"Unsupported kernel config for moe heuristic dispatch\" (QuantType issue). **With this PR:** Both short and long generation produce correct, coherent output. ### Test Result Both short and long generation produce correct, coherent output with `--enforce-eager`. ### Known Issue CUDA graph mode segfaults during decode with the W4A6 emulation path on PyTorch 2.10+ / ROCm 7.2+ (works on v0.18.0 / PyTorch 2.9 / ROCm 7.0). `--enforce-eager` is the workaround. A separate issue will be filed. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/37390", "createdAt": "2026-03-18T06:05:13Z", "updatedAt": "2026-08-13T02:14:07Z", "timestamp": "2026-08-13T02:14:07Z", "metrics": { "reactions": 1, "comments": 7 }, "labels": [ "stale" ], "author": "vecheruk-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8d301389ed28c6fe2be1", "signalId": "github:vllm-project/vllm:issue:38988", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:38988", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Qwen 3.5 27B Prefix Caching", "text": "### Proposal to improve performance _No response_ ### Report of performance regression There have been discussions if Qwen 3.5 (27B) supports prefix caching in vLLM. - https://www.reddit.com/r/LocalLLaMA/comments/1rplb3r/vllm_prefix_caching_cannot_be_used_with_qwen_35/ - #36010 I would appreciate information on this issue. ### Misc discussion on performance _No response_ ### Your current environment (if you think it is necessary) ```text The output of `python collect_env.py` ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38988", "createdAt": "2026-04-04T17:00:25Z", "updatedAt": "2026-08-13T02:13:59Z", "timestamp": "2026-08-13T02:13:59Z", "metrics": { "reactions": 4, "comments": 7 }, "labels": [ "performance", "stale" ], "author": "NilsHellwig", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5b2236361171a41a7cfa", "signalId": "github:vllm-project/vllm:issue:38994", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:38994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "Qwen-3.5 9B often producing repetitive/garbled output with Intel Backend", "text": "### Your current environment This was the output from running python collect_env.py (running in docker image) ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : version 4.3.1 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+xpu Is debug build : False CUDA used to build PyTorch : None ROCM used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Mar 3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.17.0-20-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : False CUDA runtime version : No CUDA CUDA_MODULE_LOADING set to : N/A GPU models and configuration : No CUDA Nvidia driver version : No CUDA cuDNN version : No CUDA HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. Model name: AMD Ryzen Threadripper PRO 3945WX 12-Cores BIOS Model name: AMD Ryzen Threadripper PRO 3945WX 12-Cores Unknown CPU @ 4.0GHz BIOS CPU family: 107 CPU family: 23 Model: 49 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU(s) scaling MHz: 66% CPU max MHz: 4427.7739 CPU min MHz: 560.1400 BogoMIPS: 8000.53 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es Virtualization: AMD-V L1d cache: 384 KiB (12 instances) L1i cache: 384 KiB (12 instances) L2 cache: 6 MiB (12 instances) L3 cache: 64 MiB (4 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-23 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flake8==7.3.0 [pip3] mypy==1.20.0 [pip3] mypy_extensions==1.1.0 [pip3] numpy==2.2.6 [pip3] pyzmq==27.1.0 [pip3] sentence-transformers==5.3.0 [pip3] torch==2.10.0+xpu [pip3] torchaudio==2.11.0+xpu [pip3] torchvision==0.25.0+xpu [pip3] transformers==4.57.6 [pip3] triton-xpu==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.19.1rc1.dev1+gfa9e68022 (git sha: fa9e68022) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled GPU Topology: Could not collect ============================== Environment Variables ============================== VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_TARGET_DEVICE=xpu LD_LIBRARY_PATH=/opt/intel/oneapi/ccl/2021.15/lib/:/opt/intel/oneapi/tcm/1.4/lib:/opt/intel/oneapi/umf/1.0/lib:/opt/intel/oneapi/tbb/2022.3/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/0.16/lib:/opt/intel/oneapi/mpi/2021.17/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.17/lib:/opt/intel/oneapi/mkl/2025.3/lib:/opt/intel/oneapi/dnnl/2025.3/lib:/opt/intel/oneapi/debugger/2025.3/opt/debugger/lib:/opt/intel/oneapi/compiler/2025.3/opt/compiler/lib:/opt/intel/oneapi/compiler/2025.3/lib:/opt/intel/oneapi/ccl/2021.15/lib/:/tmp/ucx_install/lib:/opt/intel/oneapi/tcm/1.4/lib:/opt/intel/oneapi/umf/1.0/lib:/opt/intel/oneapi/tbb/2022.3/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/0.16/lib:/opt/intel/oneapi/mpi/2021.17/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.17/lib:/opt/intel/oneapi/mkl/2025.3/lib:/opt/intel/oneapi/dnnl/2025.3/lib:/opt/intel/oneapi/debugger/2025.3/opt/debugger/lib:/opt/intel/oneapi/compiler/2025.3/opt/compiler/lib:/opt/intel/oneapi/compiler/2025.3/lib:/opt/intel/oneapi/ccl/2021.17/lib/:/usr/local/lib/ PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root Additionally I am running on 2xB70 Battlemage GPU's. ### 🐛 Describe the bug TLDR: I am running Qwen-3.5 9B on Intel Backend - and its often producing long thinking loops that sometimes degenerate into garbled output. Sometimes it works with a very high number of output tokens but this seems a bit arbitrary. I triple checked my input parameters - but if you see any errors in my setup you are welcome to mention it. I am starting the image and server with these settings: docker run -d --restart=always \\ --name vllm \\ --net=bridge \\ -p 8000:8000 \\ --group-add=video \\ --ipc=host \\ --privileged \\ --device /dev/dri:/dev/dri \\ -v /dev/dri/by-path:/dev/dri/by-path \\ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \\ -v /home/ejer/llm/local_models/Qwen-3.5-9B-original:/app/model:ro \\ --entrypoint /bin/bash \\ vllm-intel-03-04-2026 \\ -lc 'source /opt/intel/oneapi/setvars.sh --force && \\ vllm serve \\ --model /app/model \\ --host 0.0.0.0 --port 8000 \\ --served-model-name Qwen-3.5-9B \\ --no-enable-prefix-caching \\ --enable-chunked-prefill \\ --tensor-parallel-size 2 \\ --reasoning-parser qwen3 \\ --max-model-len 16384 \\ --enforce-eager \\ --disable-custom-all-reduce \\ --gpu-memory-utilization 0.9 \\ --max-num-seqs 128 \\ --attention-backend TRITON_ATTN \\ --dtype bfloat16 \\ --trust-remote-code' In the first example it reaches an answer but after a very long thinking loop (as if the presence penalty and repetition penalty are not taking effect). curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 5000, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.0, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Topic: Denmark.\\n * Format: One sentence.\\n * Content: Fun fact.\\n * Constraint: \\\"Give me a one-sentence fun fact about Denmark.\\\"\\n\\n2. **Brainstorming Facts about Denmark:**\\n * It has the lowest suicide rate in Europe (actually not true anymore, need to check).\\n * It's famous for hygge (coziness).\\n * It was a country of refugees during WWII (Auschwitz? No, Danish saved Jews by lying).\\n * The flag is officially the oldest national flag still in use (Dannebrog).\\n * They invented the first vending machine for newspapers.\\n * It has no nuclear weapons.\\n * It has more than 50 bicycles per 100 people (maybe too specific/changed).\\n * Famous for Lego.\\n * King Vagn? No, King Christian X had a toothache?\\n * Fact: The longest bridge is actually just a connection between Copenhagen and Zealand, not crossing water?\\n * Fact: Danish people don't have last names traditionally in old times?\\n * Fact: The Danish word \\\"hygge\\\" describes comfortable moments.\\n * Fact: Denmark was once home to the first public toilet.\\n * Fact: Their parliament is called Folketinget.\\n * Fact: It's a unitary monarchy.\\n * Fact: They drink tea in glass cups instead of mugs.\\n * Fact: It rains less often in Denmark? No.\\n * Fact: The Royal Guard wears red coats because that's what they look like in cartoons? No.\\n * Fact: Denmark is a landlocked? No, coastal.\\n * Fact: It has many islands (Danish islands).\\n * Fact: Copenhagen airport is very efficient?\\n * Fact: The Danes are known for their long nap time?\\n * Wait, let's pick a really solid one.\\n * Idea 1: \\\"The word 'hygge', which means cozy, was added to the Oxford English Dictionary in 2016.\\\" -> Good, trendy.\\n * Idea 2: \\\"Denmark was once ruled by the Vikings.\\\" -> Too obvious.\\n * Idea 3: \\\"The Danish flag is the oldest national flag still in existence.\\\" -> This is controversial. Is it the *oldest* flag or just one of them? The Jastarn flag is older. But it is widely cited as such.\\n * Idea 4: \\\"Denmark has the highest number of windmills per capita.\\\" -> True-ish.\\n * Idea 5: \\\"They have mandatory cycling?\\\" -> Mandatory bike lane?\\n * Idea 6: \\\"It rained less in Denmark recently\\\".\\n * Let's go with Hygge. \\\"The Danish concept of 'hygge' was so popularized that the word itself was added to the Oxford English Dictionary in 2016.\\\" -> This is a good fun fact.\\n * How about: \\\"Denmark has the world's largest percentage of households with bicycles.\\\" -> Maybe not accurate enough.\\n * Better fact: \\\"Denmark has more cows than people.\\\" -> That's Iceland or Sweden. Denmark has more sheep? No.\\n * Better fact: \\\"It has the most LEGO.\\\"\\n * How about this: \\\"In Denmark, Christmas trees must be decorated before sunrise on December 24th.\\\"\\n * Let's check \\\"Denmark is known for being a safe country.\\\"\\n * Actually, how about: \\\"Denmark holds the Guinness World Record for the smallest nation without any coastline.\\\" -> No, that's Monaco.\\n * Okay, let's focus on something culturally iconic.\\n * Fact: The word \\\"Hygge\\\" is in the dictionary.\\n * Fact: \\\"The Danish word 'smørrebrød' translates to buttered bread.\\\" -> Smørrebrød = open-faced sandwich.\\n * Fact: \\\"Denmark has more bridges than roads.\\\" -> Maybe not.\\n * Fact: \\\"It is impossible to walk from one end of Denmark to another.\\\" -> No.\\n * What about: \\\"Copenhagen has the longest nameplate in the world.\\\"\\n * Let's search for a verified fun fact.\\n * \\\"In Denmark, you can order coffee 'with creamer' only if you ask specifically.\\\"\\n * \\\"There are more islands than mainland areas.\\\"\\n * \\\"The capital city of Copenhagen is not an island.\\\" -> Yes, it's on Zeland.\\n * Fact: \\\"The Danish language has two different past tense verbs.\\\"\\n * Let's stick to the \\\"Hygge\\\" one.\\n * Actually, a very common one is: \\\"Denmark is a landless nation that owns no land.\\\" -> No.\\n * \\\"Every year, millions of people come to see the Great Dane.\\\" -> No.\\n * \\\"It has a flag that flies at sea.\\\"\\n * Let's try: \\\"Denmark has the official color code of its flag listed in the United Nations registry of countries.\\\" -> No.\\n * How about: \\\"Denmark has the smallest population density of all European nations.\\\" -> No.\\n * Fact: \\\"There are more than 50 islands in the Danish archipelago.\\\"\\n * Fact: \\\"Denmark is a flat country.\\\"\\n * Okay, I recall something about the Danish royal family having a cat named...\\n * \\\"The King of Denmark has never eaten sushi.\\\" -> No.\\n * \\\"The Queen of Denmark...\\\"\\n * Wait, I found this fact online earlier: \\\"Denmark was once called the 'land of happiness'.\\\" -> Not really.\\n * \\\"Denmark was part of Sweden for a while.\\\"\\n * Okay, here is a solid one: \\\"The word 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Or: \\\"Denmark has more cows than people.\\\" (No)\\n * \\\"The longest tunnel is in Odense.\\\"\\n * How about: \\\"Denmark has over 400 islands.\\\" -> There are 443 islands off the coast.\\n * Let's refine: \\\"Denmark consists of over 400 distinct islands within its territory.\\\" -> Yes, 429 islands total including mainland? No.\\n * \\\"It is the safest country in the world according to some ranking.\\\"\\n * \\\"The word 'hygge' is used to describe a feeling of comfort.\\\"\\n * Okay, I'll stick with that.\\n * Another option: \\\"In Denmark, people celebrate Valentine's Day on February 14th but also St. Valentine's day.\\\"\\n * Let's choose the \\\"Oldest Flag\\\" fact. It's widely claimed. \\\"The Danish flag is considered the world's oldest continuously used national flag.\\\" -> This is debated (Scandinavian flags existed earlier), but usually accepted as a trivia answer. But wait, \\\"the word hygge was adopted into the English language\\\" is stronger.\\n * Let's check \\\"The longest nameplate in the world\\\" - No.\\n * \\\"In Denmark, the word 'hush' is forbidden.\\\" -> No.\\n * \\\"Denmark has no king.\\\" -> It does.\\n * \\\"Queen Margrethe II is a feminist.\\\"\\n * Okay, let's craft a sentence about \\\"Hygge\\\".\\n * \\\"The concept of 'hygge' – meaning coziness and community comfort – was added to the Oxford English Dictionary in 2016, highlighting its cultural significance.\\\" -> A bit dry.\\n * How about: \\\"Denmark is famous for the term 'hygge,' which literally means cozy.\\\" -> Simple.\\n * Wait, I recall something about a Danish law requiring bicycles to stop when stopped.\\n * \\\"Denmark has the world's longest flagpole.\\\" -> No.\\n * \\\"The Danish government spends millions on a bicycle highway.\\\"\\n * Okay, let's go with: \\\"The Danish word for happiness, 'lykke,' is often replaced by 'hygge.'\\\" -> No.\\n * Let's make it punchy. \\\"Denmark is best known for the concept of 'hygge,' which refers to creating cozy, comfortable environments and experiences.\\\"\\n * Better: \\\"The Danish concept of 'hygge'—meaning coziness—was officially added to the Oxford English Dictionary in 2016.\\\"\\n * Even better: \\\"The Danish culture celebrates 'hygge' (coziness) as a vital part of daily life, and the word even made its way into the Oxford English Dictionary in 2016.\\\"\\n\\n * Check accuracy: Did \\\"hygge\\\" get added? Yes.\\n * Okay, let's draft: \\\"The Danish concept of 'hygge,' which describes cozy social interactions and well-being, was recently added to the Oxford English Dictionary.\\\" -> Accurate.\\n * How about: \\\"Denmark was once home to a Viking king who had a beard painted yellow.\\\"\\n * How about: \\\"Denmark uses the Euro currency.\\\" -> No, DKK.\\n * \\\"Denmark has the lowest crime rate in Europe.\\\"\\n * \\\"The Danes eat herring every Thursday.\\\" -> No.\\n * \\\"Denmark has more bicycles than cars.\\\" -> False.\\n * \\\"Denmark's population is mostly related to the Norsemen.\\\" -> True.\\n * \\\"Denmark has no mountains taller than 10 meters.\\\" -> True.\\n * \\\"All buildings in Denmark must face north-east.\\\" -> No.\\n * \\\"The Danish language is difficult.\\\"\\n * Okay, I think the \\\"Oxford English Dictionary 2016\\\" fact is the most robust \\\"fun fact\\\" style fact.\\n\\n * Alternative: \\\"Denmark has the world's largest collection of fairytales.\\\" -> No.\\n * \\\"It's famous for Tivoli Gardens opening night in 1843.\\\"\\n * Let's combine: \\\"Denmark is famous for adding the Danish word 'hygge' meaning cozy to the Oxford English Dictionary.\\\"\\n * Refinement: \\\"The concept of 'hygge' meaning coziness was added to the Oxford English Dictionary in 2016.\\\"\\n * How about: \\\"Denmark is known for being the country where the concept of 'hygge' was officially added to the Oxford English Dictionary.\\\"\\n * Or maybe: \\\"Denmark has the world's largest number of islands.\\\" -> No, Greece is larger.\\n * \\\"Denmark is flat.\\\"\\n * \\\"It's impossible to see the sun set on both sides of the equator there.\\\" -> No.\\n * \\\"The Queen of Denmark is known as the Queen Mother.\\\" -> No.\\n * \\\"Denmark has 16,000 lakes.\\\"\\n * Okay, I'll write about the dictionary entry.\\n * Draft: \\\"The Danish concept of 'hygge,' which represents coziness and contentment, was added to the Oxford English Dictionary in 2016.\\\"\\n * Can I make it shorter? \\\"The Danish concept of coziness, 'hygge,' entered the Oxford English Dictionary in 2016.\\\"\\n\\n * Another idea: \\\"Denmark has more bridges than tunnels.\\\" -> False.\\n * \\\"Denmark has 70% of its electricity from wind power.\\\" -> Maybe false.\\n * \\\"Denmark exports more energy.\\\"\\n * Okay, sticking to OED.\\n * Wait, another fun fact: \\\"Denmark has the shortest distance to the moon.\\\" -> No.\\n * \\\"It has the fastest train.\\\" -> No.\\n * \\\"It has a lot of castles.\\\"\\n * \\\"The Danish monarchy has been around since 1300s.\\\" -> No.\\n * \\\"The word 'hygge' was born in Denmark.\\\" -> No.\\n * Okay, I think the OED one is fine.\\n\\n * Let's double check facts.\\n * Wikipedia says: \\\"In 2016, the word 'hygge' was added to the Oxford English Dictionary.\\\"\\n * Okay, let's write it down.\\n\\n * Sentence: \\\"The Danish concept of coziness, known as 'hygge,' was added to the Oxford English Dictionary in 2016.\\\"\\n * Better flow: \\\"The Danish concept of 'hygge,' meaning coziness and contentment, was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n3. **Final Polish:** Ensure it meets \\\"one-sentence\\\" constraint strictly.\\n\\n * Proposed sentence: \\\"The Danish concept of 'hygge,' which refers to coziness and comfort, was added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Check alternative interesting fact: \\\"Denmark has the highest average income in Europe.\\\" -> Debatable.\\n * \\\"Denmark is famous for its Lego sets.\\\"\\n * \\\"The Danish word for 'love' is 'forelsket'.\\\"\\n * \\\"Denmark has the smallest island in the world?\\\" -> No.\\n * \\\"It is a small country.\\\"\\n * Okay, the \\\"Oxford English Dictionary\\\" one seems perfect.\\n\\n * Wait, is it the Danish concept or just \\\"hygge\\\"? The word itself might not be in there in 2016?\\n * Checking sources mentally: Many articles claim \\\"hygge\\\" entered in 2016. Some say it was in 2017. Let's assume the prompt doesn't require citation, just general knowledge.\\n * I'll stick with it.\\n\\n * Wait, did they add 'hyggelig'?\\n * Okay.\\n\\n * \\\"Denmark's capital city has a park built by a man who was killed in a car crash.\\\" -> No.\\n * \\\"It was voted the happiest country in 2003.\\\" -> Gallup World Poll.\\n * \\\"Denmark has 143 castles.\\\"\\n * Okay, final selection.\\n\\n * Draft: \\\"The Danish concept of 'hygge,' which describes a feeling of coziness and well-being, was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Let's check if there is a better one.\\n * \\\"Denmark is a flat country.\\\" -> Too simple.\\n * \\\"Denmark has a high standard of living.\\\"\\n * \\\"It has the world's first newspaper vending machine.\\\" -> Maybe.\\n * \\\"Denmark has the world's first traffic lights.\\\" -> No.\\n * \\\"The first traffic light was in Copenhagen.\\\"\\n * Okay, \\\"hygge\\\" feels safer.\\n\\n * Sentence: \\\"The Danish concept of 'hygge,' which means cozy and relaxed, was added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Or maybe the flag one is more visually descriptive.\\n * \\\"The Danish flag is the oldest national flag still in continuous use.\\\" -> \\\"continuous use\\\" implies it hasn't changed design significantly.\\n * I will stick with the dictionary one.\\n\\n * Wait, is it \\\"Oxford English Dictionary\\\" or just \\\"English Dictionary\\\"? Oxford.\\n * Sentence structure: \\\"In Denmark, the Danish word 'hygge,' which means cozy, was added to the Oxford English Dictionary in 2016.\\\"\\n * Correction: \\\"Hygge\\\" is the word.\\n * \\\"The word 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Is it Danish? Yes.\\n * Does it make sense as a fact? Yes.\\n\\n * Alternative: \\\"Denmark has more bicycles per person than cars.\\\" -> False.\\n * \\\"The Danish capital has a lot of bridges.\\\"\\n * Okay.\\n\\n * Wait, another popular fun fact is: \\\"The first person to live in a car.\\\"\\n * No.\\n * \\\"It's impossible to see the sun set on the equator in Denmark.\\\" -> No.\\n * \\\"It has no mountain higher than 1 meter.\\\" -> False, 126 meters max.\\n * Okay, the dictionary one is good.\\n\\n * Final choice: \\\"The Danish concept of coziness, known as 'hygge,' was officially added to the Oxford English Dictionary in 2016.\\\"\\n\\n * Let's refine for punchiness.\\n * \\\"Denmark's unique concept of 'hygge'—referring to cozy social comfort—was added to the Oxford English Dictionary in 2016.\\\"\\n * Okay.\\n\\n * Wait, is it the country or the concept? Both.\\n * Maybe \\\"The Danish concept of 'hygge' was added to the Oxford English Dictionary in 2016.\\\"\\n * Add \\\"in Denmark\\\"? -> \\\"In Denmark, the word 'hygge,' which means cozy, was added to the Oxford English Dictionary in 2016.\\\" -> Repetitive.\\n * Okay.\\n * Let's go with that.\\n\\n * Final sentence: \\\"The Danish concept of 'hygge,' meaning cozy and comfortable, was added to the Oxford English Dictionary in 2016.\\\"\\n * Check constraints again. One sentence.\\n * Done.\\n\\n4. **Wait, wait.**\\n * Are there other fun facts?\\n * \\\"The longest bridge is a footbridge.\\\"\\n * \\\"The shortest road.\\\"\\n * \\\"It has a statue of a queen.\\\" -> No.\\n * Okay, let's check \\\"Denmark is the land of happiness.\\\" -> \\\"The happiest place on earth.\\\"\\n * \\\"The Danes have the shortest haircuts.\\\" -> No.\\n * Okay, sticking with OED fact.\\n * Wait, the user asked for a *fun* fact.\\n * \\\"Denmark was once ruled by a king who liked cats.\\\" -> No.\\n * Okay, \\\"The word 'hygge' entered the Oxford English Dictionary in 2016.\\\"\\n\\n * Okay.\\n\", \"content\": \"\\n\\nThe Danish concept of 'hygge,' which refers to coziness and well-being, was officially added to the Oxford English Dictionary in 2016.\" In the next example I run with slightly higher repetition_penalty - and here output degenerates rapidly: curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 2048, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.2, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"Thinking Process:\\n\\n1. **Analyze the Request:**\\n * Topic: Denmark (country).\\n * Constraint: One sentence only.\\n * Tone: Fun fact.\\n\\n2. **Brainstorming Facts about Denmark:**\\n * Lego is Danish? Yes, but maybe too cliché. \\\"It was founded by Hans Christian Andersen?\\\" No, wrong name. It's known for windmills and hygge. Flat landscape with lots of beaches on some islands vs mountains elsewhere in Europe... Wait that doesn't sound right either. Actually most countries are low-lying due to sea level rise fears or something like Copenhagen has high tides sometimes called 'High Tide'. Or perhaps it rains every day except Sunday - this might be a joke rather than true.. Let me check facts.\\\"Denmark invented LEGO\\\" -> The toy company started as wooden models made up from various parts which they could put together\\\". But wait there were also more specific items. What kind would make someone smile then laugh out loud at their own stupidity when I said no such things exist?\\\". Ah yes! Here comes an idea; How many people speak English perfectly well here compared other Europeans because so much time spent watching TV programs broadcasted worldwide where everyone speaks multiple languages easily without learning any new words first!\\\" That isn't quite accurate though since language barrier does still exists even within Scandinavia itself alone today according studies show otherwise now actually!\\\". Okay let’s focus back on actual history related stuff regarding famous inventions happening around world before moving forward towards modern times later again next.\\\". Hmm okay good enough information provided above should suffice if written properly using simple vocabulary terms plus concise expression styles applied throughout writing process overall making sense easy reading possible anytime anywhere whenever needed soonest tomorrow morning afternoon evening night all hours round clock everyday forever until eternity ends someday sometime neveragain evermore whatsoever whateverwhereverwheneverwhywhenwhatifhowaboutthisandthatthensoyesnoormaybebutnotnorthernsouthernmidwesterneasterncentralinnerouterupperlowermiddlehighlowshortlongfar nearbig smallheavy lightdarkbrightdullclearcloudyfairstarkrainysunnywindygustyfoggyshoweryhailingsnowfallfreezingmeltingboilingcondensingevaporatingsublimationprecipitationtranspirationphotosynthesiscellularrespirationdigestionabsorptionmetabolismhomeostasisreproductiongrowthdifferentiationmaturationapoptosismitochondriaorganicinorganicacidbaseoxidizereducentelectronegativityconductivityspecificheatlatentheatspecificenergykineticpotentialmechanicalchemicalthermalnucleargravitationalhydrostaticbarometricatmosphericweatherclimateenvironmentecologybiodiversityspeciesdiversityecosystemhabitatnichepopulationcommunityinteractioncompetitionpredationsymbiosisparasitismmutualismandcommensalismcooperationcollaborativeconsensusdemocracyliberalauthoritarianautocraticanarchisticfascisttotalitariumcommunistsocialismanarcho-capitalismeconomicinequalitypovertywealthresourceallocationdistributionconsumptionprofitmarketdemandsupplypriceinterestratevalueexchangeutilityscarcitysatisfactionneedwantsdesirepreferenceschoicesdecisionsplanningbudgetfinancesinvestmentsavingsassetsliabilitiesdebtcreditriskguaranteewarrantylifecycleownershipmaintenancerepairreplacementretirementpensioninsurancebenefitscompensationrewardrecognitionappreciationsuccessachievementaccommodationcomfortluxuryconvenienceeaseaccessibilityavailabilityresponsivenessservicecustomerclientsupportfeedbackloopimprovementefficiencyproductivityperformanceoutputinputthroughputconversionraterevenuecostspricingstrategymarketsharebrandawarenessimageperceptionpositioningattractivelocalglobalnationwideinternationalworldwidemulticulturalcrossculturalinterracialinterfaithintrareligiousethnocentricxenophobicracialnationalreligioussecularaestheticbeautyartcreativestyleformdesignstylefunctionalsignificancemeaningpurposeintentmessagecommunicationlanguageexpressionvoicearticulationtonepitchvolumeclaritydistortionnoisebackgroundambienceacousticsonicsoundaudiovisualsensorysynesthesiachromaticcolorfulnesshuebrightnesssaturationcontrastshadingtexturelightshadowreflectanceopacitytranslucencydensitymassvolumeweighthumiditytemperaturepressureforcevelocityaccelerationmomentuminertiafrictionviscosityelasticmodulusstrengthdurabilityhardnessexactitudetransparencyimpuritiesdefectsflawsweakpointsfailuresbreakagescratchesdentchipsdingsbruiseabrasionscorrosiorrusttarnishwearouterosivegrindingpolishingcuttingslicingcarvingengravingdrillingboringreamingtappingthreadinggasketsealingvalvesleakagepipeslinesductssprinklerirrigationwaterworksfiltratesedimentstorageaccumulateloadbearingcompressionexpansioncontractionflexuretorsioneccentrictangentialradialnormalcurvatureangledirectionorientationinclinationdeclinationlatitudelongitudealtitudeelevationheightdepthleveldistanceproximityclosenessnearbyremoteisolatedremoteseparateconnectedlinkedbondedsimilaridenticalsametypekindnatureessencesubstancecompositionstructureorganizationarrangementpatternorderchaosdisorgandomesmessynescrambledmixedupconfusedjumbledentangledknottedtwistedcrookedstraightforwarddirectindirectambiguousunclearobfuscatelegraphiccryptographicephermerencesectionsparagrapbspageschaptersshelfsectiontopicdivisionpartsegmentportionfractionquantitynumbercountmeasuremetricstandardunitssystemmeasurementweighthingscalingdimensionalgeometricaltopologicalspatialtemporalchronologicaleventsequentialtimelineperiodintervaltimefrequencydurationlengthwidththicknessareaavolumeamountweightcapacitycapabilitypossiblerequirementexpectancyhopeanticipationpredictionforecastprojectionspeculationguessestimatejudgmentopinionassumptionbeliefconfidencecertaintyprobabilityoddslikelihoodchanceriskgamblebetwinninglossbreakingmakinglosingpayingchargingbillingfeepremiumsurchargefeeassessmenttaxlevydutytariffimportexportcustomsdraftbillstatuteregulationsordinancemarxlawofeconomicdevelopmentlawsregulatorylegallegislationconstitutionjurisdictioncourtsjudgejusticecrimefelonyoffensepunishmentsentencepenaltyfineforfeitforbiddenillegalillicitunauthorizedundesirabledangerousunsafeimmovablemobilemovabletransferportabledispersedistributedstoreduploadeddownloadsharedpublicprivatepersonalproprietaryintellectualpropertycopyrighttrademarkservice trademark service trade secret proprietary intellectual property copyright patent design application invention creation development innovation technology science research education health care social services public policy government administration bureaucracy governance democracy liberty equality justice freedom human rights universal values international norms global standards local rules national laws regional agreements intergovernmental cooperation multilateral collaboration bilateral partnership tri-lateral alliance coalition treaty pact accord protocol understanding memorandum agreement commitment declaration statement position view opinion belief thought feeling emotion attitude perspective angle direction aspect factor element component part segment portion fraction quantity number count measure metric standard unit system measurement weighing scaling dimensional geometrical topological spatial temporal chronological sequential timeline period interval duration length width thickness area volume amount weight capacity capability possibility requirement expectation hope anticipation prediction forecast projection speculation guess estimate judgment assumption condition necessity impossibility constraint limitation restriction rule regulation law principle axiom theorem postulate hypothesis conjecture theory proposition assertion claim argument debate discussion conversation dialogue monologue narrative story tale legend myth folklore tradition custom culture heritage legacy ancestry descent bloodline genetic inheritance familial lineage genealogical tree family pedigree ancestors descendants progeny offspring children grandchildren great-grandchildren etceterasiblingspartnersfriendsrelativesneighborsextendedfamilynetworksocialsupportcirclesrelationshipsconnectionslinksbridgestiesetieabroadlocaldomesticforeignoverseacontinentalhemisphereplanetsolarsystemgalaxyuniversmultiverseeverythingallnothingnonexistnullvacuumvoidemptinesssilencequietstillnesstopausebreatherrestslacksettlementresolutionclosurecompletionterminationendingfinishdonepastpresentfuturehistorymemoryrecallforgetforgivenoignoreneglectdismissrejectacceptapproveadoptchooseselectpicktakeholdgethavekeepownlosegivegrantdonatenextnextstepfollowingbeforeafterduringwhileuntilsincefromtointoontouptoadatabooktextdocumentfilepaperpageprintwordletteremailnotememoentryrecordlogdatabaseinformationdataknowledgewisedomainspecificationsrequirementsstandardsrulesguidelinespolicyproceduremanualinstructionguidebookhelpdeskassistanceconsultantexpertadviceaidcomfortcarehealthwellbeingfitnessactivityexerciselearningtrainingteachingeducationschoolcollegedeskclassroomlaboratoryequipmentmachinerytoolinstrumentgeardeviceappliancedigitalanaloguehybriddesktopworkstationservercomputerprocessorchipmicroelectronicssemiconductorphotolithographyetchingoxygenizingnitrogenpurgingargonshieldingasphygasaircompressorradiowavesmicrowavesmillimeterswaveinfaredultravioletvisiblelightspectrumradiolightelectriccurrentvoltammeterskilometersvoltageluxlumensohmsampsjoulescoulombssecondsminuteshoursdaysweeksmonthsyearsdecennialsseptenaryoctennialquinquagenarianvigintiarytri-centurycircularquadricularenglandswedenbelgiumgermanyfrancenetherlandsdenmarksaxtoniaitaliashriyenafreezelandmondoitaliaeuromonacoathenaolympusmontpelierspainnylonsteelwoodbrickglassstonecementplasticonlyerubberfibermetalceramicmineralsoilwatervineyardcornfieldforestmarshmeadowwetlandsgrasslaandleisurezoneurbanareasbuildingconstructionhousingfacilityinstallationfactoryplantmachinevehicletransportshipboattrainbusmetrotramcartvantruckmotorcycleautomobilemotoauto-carjeetcarsvehicletypetypesmodelsmakebrandsmanufacturerproducercreatorfabricatorinventormakercraftsmanbuilderartisanartistwritercomposeractorperformerpresentertalentstarcelebritystudiosceneplayfilmcinemaauditoriumtheaterconcertorchestraoperamusichallsoundsystemsplaylisttracklistlibrarycollectionmusicgenretypecategorylabelpublishereditorjournalmagazineradiochannelsstationsbroadcastsignaltelecommunicationsmediainternetwebonlinevirtualcyberspaceblockchainbitcoinethereumaltcoinscryptoassetdigitalcoinfiattokenledgerwallettransactioncontractsmartcontractscodealgorithmprotocolframeworklayerstacktechnodearchitectureengineeringstructurescomponentspartsmodulesfunctionsoperationsmethodsprocessestrategiesplansprogramsprojectsinitiativescampaignsstoriesideasthemesconceptsthoughtmodelsprincipledesignsbusinessenterprisesorganizationsindustriessectorsfieldsdomainsnicheexpertise\", \"content\": null } With running docker server with --tensor-parallel-size 1 \\ (without \"--enforce-eager\") I get this garbled output: curl http://localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{ \"messages\": [ {\"role\": \"user\", \"content\": \"Hello! Give me a one-sentence fun fact about Denmark.\"} ], \"stream\": false, \"max_tokens\": 2048, \"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"presence_penalty\": 1.5, \"repetition_penalty\": 1.2, \"min_p\": 0.0, \"include_reasoning\": true }' | jq -r '.choices[0].message | {reasoning, content}' { \"reasoning\": \"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\", \"content\": null } ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/38994", "createdAt": "2026-04-04T19:51:48Z", "updatedAt": "2026-08-13T02:13:57Z", "timestamp": "2026-08-13T02:13:57Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "bug", "stale" ], "author": "AlexanderValentini", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a95d898326d6c66afe08", "signalId": "github:vllm-project/vllm:pull_request:39452", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:39452", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Use physical device id for get_device_uuid", "text": "This PR updates the get_device_uuid to translate the device_id parameter to a physical gpu id before retrieving the gpu handle. `get_device_uuid` is used in the Verl application PPO and Fully Async uses cases which executes the code below when the vllm backend is being used: def get_device_uuid(device_id: int) -> str: from vllm.platforms import current_platform # Convert torch.npu.current_device to its corresponding ASCEND_RT_VISIBLE_DEVICES. if is_npu_available: if os.getenv(\"ASCEND_RT_VISIBLE_DEVICES\") is not None: npu_visible_devices = os.environ[\"ASCEND_RT_VISIBLE_DEVICES\"].split(\",\") assert device_id < len(npu_visible_devices), f\"device_id {device_id} must less than {npu_visible_devices}\" return \"NPU-\" + npu_visible_devices[device_id] else: return f\"NPU-{device_id}\" else: return current_platform.get_device_uuid(device_id)", "url": "https://github.com/vllm-project/vllm/pull/39452", "createdAt": "2026-04-09T21:01:17Z", "updatedAt": "2026-08-13T02:13:53Z", "timestamp": "2026-08-13T02:13:53Z", "metrics": { "reactions": 1, "comments": 6 }, "labels": [ "rocm", "stale" ], "author": "tmm77", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0808ed5efc5502b2c803", "signalId": "github:vllm-project/vllm:issue:50765", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50765", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug][P/D]: NIXL handshake failure with asymmetric TP in PD disaggregation (pTP2 + dTP4)", "text": "### Your current environment - vLLM version: `0.1.dev1+g51f799c1a` - NVIDIA GPU: H200 (140GB) - CUDA: 13 - NIXL transport: UCX/RDMA (RoCE) ### Model google/gemma-4-26B-A4B (MoE, 26B total, ~4B active) ### Describe the bug NIXL KV transfer handshake fails with `NIXL_ERR_BACKEND` when using asymmetric tensor parallelism in PD disaggregation (prefill TP=2, decode TP=4). The failure occurs during `loadRemoteMD` — the RDMA metadata exchange — before any actual KV transfer happens. The prefill pods (2×TP2) start and serve correctly. The decode pod (1×TP4) loads the model and begins serving, but crashes when the first request triggers a NIXL handshake with a prefill pod. The block count mismatch between prefill (`num_remote_blocks: 113`) and decode (`num_local_blocks: 211`) causes the handshake to fail. After the decode pod crashes and restarts, subsequent handshakes continue to fail (possibly related to #38840). Symmetric TP configurations (e.g., pTP4 + dTP4) work correctly. ### Configuration ``` Prefill: 2 pods × TP2 Decode: 1 pod × TP4 Total GPUs: 8 (single node) enable_bidirectional_kv: true ``` ### Error logs (decode pod) ``` (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] NIXL transfer failure: handshake_failed | Context: {'failure_type': 'handshake_failed', 'request_id': 'cmpl-39ab6c4d-b1df-473c-a398-b6c02944f8c8-0-ada18e77', 'engine_id': '48174aad-7e0e-48bf-9494-00877081f134', 'remote_engine_id': '7cd7cde3-e7cc-4347-a286-92f4f2581f70', 'remote_host': '10.131.1.160', 'remote_port': 5600, 'num_local_blocks': 211, 'num_remote_blocks': 113, 'local_block_ids_sample': [2474, 2475, 2476]} (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] Traceback (most recent call last): (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 764, in request_ready (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] f.result() (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 733, in done_callback (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] self._remote_agents[eid] = f.result() (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 581, in _nixl_handshake (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] remote_agent_name = self.add_remote_agent( (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm-source/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py\", line 1309, in add_remote_agent (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] remote_agent_name = self.nixl_wrapper.add_remote_agent( (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] File \"/opt/vllm/lib64/python3.12/site-packages/nixl_cu13/_api.py\", line 832, in add_remote_agent (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] agent_name = self.agent.loadRemoteMD(metadata) (Worker_TP1_EP1 pid=2749) ERROR 08-02 19:43:44 [worker.py:767] nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND ``` ### Related issues - #21576 — NIXLConnector does not support P TP > D TP (closed as stale, not fixed) - #22663 — Fix heterogeneous TP (merged, fixed KV cache layout to HND, but doesn't fix `loadRemoteMD` handshake failure) - #38840 — Handshake race when same-node workers re-register with new engine IDs", "url": "https://github.com/vllm-project/vllm/issues/50765", "createdAt": "2026-08-02T19:58:11Z", "updatedAt": "2026-08-13T02:12:38Z", "timestamp": "2026-08-13T02:12:38Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "bbenshab", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:35964e94755299bcec91", "signalId": "github:vllm-project/vllm:pull_request:51967", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51967", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][DSV4] Optimize global top-k index kernel with compile-time constants", "text": "## Purpose Optimize global top-k index kernel with compile-time constants ## Test Plan ``` vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --tensor-parallel-size 8 --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --no-enable-prefix-caching --max-num-batched-tokens 16384 ``` ``` vllm bench serve --backend vllm --base-url http://localhost:8000 --model deepseek-ai/DeepSeek-V4-Flash-0731 --dataset-name random --random-input-len 1024 --random-output-len 64 --num-prompts 128 --num-warmups 8 --request-rate inf --ignore-eos --temperature 0 --seed 701 --save-result ``` ## Test Result ## Serving Benchmark Results | Version | Mean Output Throughput | Mean TPOT | Completed Requests | Relative Change | | --- | ---: | ---: | ---: | ---: | | main | 634.97 tokens/s | 103.33 ms | 128/128 | Baseline | | this pr | 638.12 tokens/s | 102.32 ms | 128/128 | Throughput **+0.50%**, TPOT **-0.98%** | ## Kernel Microbenchmark ```python import torch import triton from vllm.models.deepseek_v4.common.ops.cache_utils import ( _compute_global_topk_indices_and_lens_kernel, ) num_tokens = 512 topk = 512 block_size = 256 triton_block_size = 1024 global_topk_indices = torch.empty( (num_tokens, topk), device=\"cuda\", dtype=torch.int32 ) global_topk_lens = torch.empty(num_tokens, device=\"cuda\", dtype=torch.int32) topk_indices = torch.randint( 0, 8192, (num_tokens, topk), device=\"cuda\", dtype=torch.int32 ) block_table = torch.arange( 8192, device=\"cuda\", dtype=torch.int32 ).reshape(num_tokens, -1) def run(): _compute_global_topk_indices_and_lens_kernel[(num_tokens,)]( global_topk_indices, global_topk_indices.stride(0), global_topk_lens, topk_indices, topk_indices.stride(0), topk, block_table, block_table.stride(0), block_size, TRITON_BLOCK_SIZE=triton_block_size, ) for _ in range(20): run() torch.cuda.synchronize() latency_us = triton.testing.do_bench(run, warmup=100, rep=500) print(f\"Latency: {latency_us:.3f} us\") ``` | Kernel Version | Latency | Relative Change | | --- | ---: | ---: | | Original implementation | ~7.50 us | Baseline | | Compile-time-constant optimization | ~6.37 us | **15.1% faster** | ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51967", "createdAt": "2026-08-12T09:06:50Z", "updatedAt": "2026-08-13T02:10:36Z", "timestamp": "2026-08-13T02:10:36Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [], "author": "chaunceyjiang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e54180937635d06e8e19", "signalId": "github:vllm-project/vllm:pull_request:52064", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52064", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Mirror external test assets in vLLM S3", "text": "## Summary - mirror externally hosted video, image, and GSM8K test assets into the public vLLM S3 bucket - update affected tests to use the shared `VLLM_S3_BUCKET_URL` constant ## Why [Buildkite build 83608](https://buildkite.com/vllm/ci/builds/83608) had multiple test failures caused by direct dependencies on third-party hosts, including `RemoteDisconnected` while fetching OpenCV and Bogotobogo video fixtures. The mirrored objects are publicly readable from `vllm-public-assets` in `us-west-2`, and each object was verified byte-for-byte against its source. ## Duplicate work I searched open PRs for the affected video fixtures and GSM8K/JinaVL S3 mirroring and found no overlapping change. #52043 handles separate missing hybrid-dependency wheel failures and is intentionally outside this PR. ## Validation - full pre-commit selection against `origin/main...HEAD` passed, including Ruff, formatting, mypy, and repository validation hooks - `git diff --check` - all 7 mirrored test assets fetched from public S3 and matched their source SHA256 checksums - OpenCV decoded all mirrored videos successfully: - `Megamind.avi`: 270 frames at 23.976 fps - `slow_traffic_small.mp4`: 914 frames at 29.970 fps - `vtest.avi`: 795 frames at 10 fps - GSM8K train and test mirrors passed line-by-line JSON parsing - targeted pytest collection was attempted, but the local test environment is missing `numpy` and fails while loading `tests/conftest.py` Model evaluations were not run because this changes CI asset locations only and does not affect model behavior. ## AI assistance AI assistance was used to investigate the CI failures and prepare this change. The human submitter is responsible for reviewing every changed line and validating the change before merge.", "url": "https://github.com/vllm-project/vllm/pull/52064", "createdAt": "2026-08-12T22:55:52Z", "updatedAt": "2026-08-13T02:08:59Z", "timestamp": "2026-08-13T02:08:59Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "ready", "ci/build", "multi-modality" ], "author": "khluu", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:daa6384b120c973bfcef", "signalId": "github:vllm-project/vllm:pull_request:51823", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51823", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(pooling): validate BGE-M3 combined task ownership", "text": "## Purpose Fixes #51801. BGE-M3 can resolve to the internal `embed&token_classify` task. That combined task belongs to the sparse IO-processor plugin, which also converts the combined model output into the public dense/sparse response. Registering a generic pass-through processor would avoid the original `KeyError`, but it would not perform that conversion. This revision keeps the combined task plugin-owned: - fail at startup with an actionable error when `embed&token_classify` is selected without an IO processor plugin; - register only the plugin processor when the plugin is available; - pin the documented sparse server to `token_classify` and require a restart with `token_embed` for ColBERT. This does not restore dynamic per-request task switching or change model weights, kernels, pooling math, or output values. I checked the issue Development section and searched open PRs for `#51801`, `embed&token_classify`, and BGE-M3 pooling processor changes. No overlapping PR was found; #44530 is about asymmetric ColBERT encoding and is unrelated. AI assistance was used for investigation, code drafting, test scaffolding, and review. I reviewed every changed line, reproduced the factory behavior, and understand the change well enough to maintain it. ## Test Plan - `.venv\\Scripts\\python.exe -m pytest tests/entrypoints/pooling/test_factories.py -q --confcutdir=tests/entrypoints/pooling` - `.venv\\Scripts\\python.exe -m ruff check vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `.venv\\Scripts\\python.exe -m ruff format --check vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `.venv\\Scripts\\python.exe -m py_compile vllm/entrypoints/pooling/factories.py vllm/entrypoints/pooling/pooling/io_processor.py tests/entrypoints/pooling/test_factories.py` - `git diff --check` Model evaluation is not applicable because the revision changes factory validation and documentation only; it does not affect model execution or numerical outputs. ## Test Result - Focused factory regression: `2 passed` - Ruff check: passed - Ruff format check: passed - Python compile check: passed - Diff check: passed ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose and linked issue are described. - [x] The test plan includes the commands used. - [x] Test results are included. - [x] Documentation is updated for the fixed-task serving contract.
", "url": "https://github.com/vllm-project/vllm/pull/51823", "createdAt": "2026-08-11T13:22:26Z", "updatedAt": "2026-08-13T02:00:49Z", "timestamp": "2026-08-13T02:00:49Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "frontend" ], "author": "030611", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1d7cf22006f6cf48eb53", "signalId": "github:vllm-project/vllm:pull_request:51886", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51886", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KVConnector] Add retention interval to OffloadingConnector", "text": "## Purpose Add retention interval support to `OffloadingConnector`. The Mooncake KV connector supports retention interval support via the `envs.VLLM_PREFIX_CACHE_RETENTION_INTERVAL` env variable. It is used to avoid storing/loading blocks that are outside of the sliding window of the attention mechanism. This is implemented via block masks. The implementation in `OffloadingConnector` uses the same block mask mechanism to implement retention interval support. cc @varun-sundar-rabindranath Note: parts of the code generated with claude ## Test Plan - Run kv_offloading unit tests - Add new unit tests - run gptoss w/retention interval enabled. ## Test Result gpt-oss-20b results ``` Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-20b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":4294967296,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-20b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.315 │ 0.185 │ 36.5 │ 1284.8 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.305 │ 0.145 │ 33.4 │ 1434.2 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.325 │ 0.170 │ 25.2 │ 1840.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.325 │ 0.160 │ 29.8 │ 1577.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.345 │ 0.200 │ 22.3 │ 2138.7 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.320 │ 0.165 │ 28.1 │ 1678.6 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` gpt-oss-120b results ``` ● gpt-oss-120b — Retention Interval Eval Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-120b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-120b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.540 │ 0.145 │ 58.9 │ 725.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.540 │ 0.100 │ 51.7 │ 851.9 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.575 │ 0.085 │ 57.0 │ 769.3 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.555 │ 0.080 │ 50.9 │ 870.1 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.570 │ 0.095 │ 40.3 │ 1059.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.570 │ 0.080 │ 50.9 │ 851.3 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` DeepSeekV4 results showing reduction in memory stores for different intervals. ``` DeepSeek-V4-Flash (GSM8K, 200 questions, 8-shot, max_model_len=16384) Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve deepseek-ai/DeepSeek-V4-Flash \\ --enforce-eager \\ --max-model-len 16384 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --block-size 256 Where INTERVAL is unset (baseline), 512 (sparse), or 0 (most sparse). Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 8 \\ --host http://localhost \\ --port 8001 Accuracy: ┌──────────────────────────┬────────┬──────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.960 │ 48.5 │ 371.1 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.970 │ 53.0 │ 345.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ fresh │ 0.965 │ 54.3 │ 331.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ reload │ 0.955 │ 47.9 │ 380.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.970 │ 52.6 │ 343.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.960 │ 46.3 │ 389.6 │ └──────────────────────────┴────────┴──────────┴─────────────┴──────────┘ Prometheus Offload Metrics: ┌──────────────────────────┬─────────────────────────┬───────────────────────┐ │ Configuration │ Store Bytes │ Reduction vs Baseline │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Baseline (no interval) │ 4,610,304,000 (4.61 GB) │ — │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (interval=512) │ 192,430,080 (192 MB) │ 95.8% │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Most sparse (interval=0) │ 104,232,960 (104 MB) │ 97.7% │ └──────────────────────────┴─────────────────────────┴───────────────────────┘ ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51886", "createdAt": "2026-08-11T23:11:05Z", "updatedAt": "2026-08-13T01:45:26Z", "timestamp": "2026-08-13T01:45:26Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kv-connector" ], "author": "bnellnm", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:94bdbc1d3f2e7afa5e7c", "signalId": "github:vllm-project/vllm:pull_request:52014", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52014", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KVConnector][MoRIIO] Fix MoRIIO port collisions for deployments using both DP and TP", "text": "## Purpose Fix MoRIIO port collisions for deployments using both DP and TP. Port offsets now use `dp_rank * tp_size + tp_rank`, where `tp_size` belongs to the listening peer. This also handles heterogeneous TP, asynchronous WRITE completion, and the unknown-TP sentinel correctly. ## Test Plan Run the focused MoRIIO unit tests ## Test Result Unit tests passed ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52014", "createdAt": "2026-08-12T15:51:54Z", "updatedAt": "2026-08-13T01:42:15Z", "timestamp": "2026-08-13T01:42:15Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "kv-connector" ], "author": "zzaebok", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4d0bbf54e0df07213a4c", "signalId": "github:vllm-project/vllm:pull_request:51437", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51437", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Kimi-K3 latent-MoE: overlap the shared all-reduce with the routed up-projection", "text": "## Purpose Kimi-K3's latent MoE combines a **routed** expert output — produced in a compressed latent space, then up-projected back to hidden dim — with a **shared** expert output. Under tensor parallelism both need an all-reduce. Today the ROCm runner (`ROCmLatentMoERunner`) only implements the column-parallel tail (`_shard_up_proj_tail`, \"Tier 2\"), which folds a sharded up-projection into the shared partial before a single reduce. That tail is efficient for prefill-sized batches but gives up all overlap: the reduce must follow the accumulate. This PR adds a token-count-dispatched overlap tail so decode-sized batches stop leaving the shared all-reduce on the critical path: - **New Tier 1 — `_overlap_allreduce_tail`.** All-reduce + RMSNorm the latent, then run the full replicated up-projection GEMM on the default stream while the shared-expert all-reduce runs on the aux stream (`maybe_execute_in_parallel` + `aux_stream()`), and add. For batches under `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` the shared all-reduce is hidden behind the up-proj GEMM. - **Tier selection — `_select_tail_tier`.** Small batches take the overlap tail; larger batches keep the column-parallel tail; when the up-projection rows do not divide evenly across ranks (or the routed scale is non-unit) the runner falls back to the overlap tail, which is correct at any shape. - **`_allreduce_norm_latent_out`.** Fused AR+RMSNorm for the latent — collapses the latent all-reduce and its RMSNorm into a single AITER op, with a plain `all_reduce` + `RMSNorm` fallback. This helper is backed by a small addition to the base runner (`moe_runner.py`): a module-level `_aiter_fused_ar_rmsnorm` binding (via `rocm_aiter_ops.get_fused_allreduce_rmsnorm_op()`) plus a `_get_zero_residual` buffer helper. The underlying AITER op already exists in `_aiter_ops.py`; the base runner just did not expose it before. The native (non-fused) path is unchanged: the fused path is still gated on `tp_size > 1`, an un-reduced combine output, a shared expert, no sequence parallelism, and unit routed scale — anything else defers to the base `MoERunner.forward`. No existing base-runner behavior changes; the only base-runner edit is the additive AR+RMSNorm helper above, covered by a new unit test. Relates to #50682 (ROCm/AMD Kimi-K3 gap and roadmap tracking → Performance Optimization). Tracked as PR #51437. ### Files changed - `vllm/models/kimi_k3/amd/latent_moe_runner.py` — the overlap tier, tier selection, and `_allreduce_norm_latent_out`. - `vllm/model_executor/layers/fused_moe/runner/moe_runner.py` (+90/−7) — additive `_aiter_fused_ar_rmsnorm` binding + `_get_zero_residual` helper (no behavior change to existing paths). - `tests/models/kimi_k3/test_amd_latent_moe_runner.py` — tier-selection unit tests + multi-GPU parity for both tails. - `tests/model_executor/test_moe_runner_fused_ar_rmsnorm.py` — unit test for the new base-runner helper. ### This feature has no kernel dependency The overlap and column-parallel tails contain **no activation** — they are reductions, an up-projection GEMM, and an add. The `situ`/`situv2` activation lives entirely in the stage-1 expert GEMM (`_forward_entry`), which runs **before** tier selection and is identical across Tier 1, Tier 2, and the upstream baseline. So this PR does not depend on any activation-kernel change, and the unit + multi-GPU parity tests — which feed synthetic routed/shared tensors straight into the tail methods and never invoke the expert GEMM — pass regardless of the AITER build. **Environment note (not a feature dependency):** serving Kimi-K3 int4 on gfx942 at all requires **ROCm/aiter#4471** (\"[FlyDSL] Support SiTUv2 in the packed-int4 MoE stage1 epilogue\"), because Kimi-K3 uses `hidden_act = situ` and the current packed-int4 FlyDSL stage1 otherwise raises `NotImplementedError: split-K stage1 activation supports only 'silu', got 'situv2'` (or silently computes SiLU). This affects the **upstream Tier-2 baseline (#51253) exactly as much as this PR** — it is a prerequisite for reproducing the *end-to-end serving* numbers below, not something this feature introduces or relies on. On an AITER without split-K SiTUv2 support, force the non-split-K stage1 with `AITER_KSPLIT=1`. ### Relationship to #50657 #50657 (\"[ROCm][Perf] Fuse and FP8-pack Kimi-K3 latent MoE output tail\") targets the same boundary but with a different mechanism: a dedicated AITER FlyDSL fused-tail kernel plus optional FP8 packing of the up-projection weight, which depends on unmerged AITER PRs (ROCm/aiter#4496, #4503). This PR is complementary — it schedules existing in-tree collectives/GEMMs across two streams and adds an AITER fused AR+RMSNorm helper to the base runner (backed by an op already present in `_aiter_ops.py`, so no new kernel). The two overlap only on the `runner_cls=` wiring line in `amd/linear.py` (already present on `main`); if both land, that single line needs reconciling to a runner that can select between the two strategies. ## Test Plan Unit (no GPU) — tier-selection and fused-path gates: ```bash .venv/bin/python -m pytest -q \\ tests/models/kimi_k3/test_amd_latent_moe_runner.py \\ -k \"not tp4 and not tp8\" ``` Multi-GPU arithmetic parity — each tail must equal the replicated up-projection: ```bash # 4- and 8-GPU ROCm host .venv/bin/python -m pytest -q tests/models/kimi_k3/test_amd_latent_moe_runner.py ``` The multi-GPU suite covers, at TP4 and TP8: - `test_overlap_tail_tp{4,8}_matches_replicated_projection` — new Tier 1 output equals `shared + up_proj(rms_norm(all_reduce(routed)))`. - `test_shard_tail_tp{4,8}_matches_replicated_projection` — Tier 2 unchanged. - `test_shard_tail_tp4_writes_only_its_own_shard` — the column-parallel tail writes only this rank's hidden shard before the final collective. End-to-end serving parity (8× MI355X, `gfx950`, `moonshotai/Kimi-K3`): ```bash vllm serve /model --served-model-name moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 --trust-remote-code --moe-backend auto \\ --gpu-memory-utilization 0.9 --max-num-seqs 32 \\ --max-num-batched-tokens 4096 --reasoning-parser kimi_k3 ``` Compare the candidate against `main` (baseline = Tier 2 only). Exercise both tiers by sweeping batch size around `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` (small decode → overlap tier; large prefill → column-parallel tier) and force the column-parallel path with `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1`. Note: EP must be **off** for the fused tail to engage — with expert parallel enabled the MoE combine already reduces the fused output, so the runner takes the native path and neither tail runs. ## Test Result _All numbers below are placeholders — fill from the runs described in the Test Plan. Environment: 8× MI325X (`gfx942`), `moonshotai/Kimi-K3`, `int4_per_group_32`, TP=8, EP off. Serving requires an AITER that handles SiTUv2 (ROCm/aiter#4471), or `AITER_KSPLIT=1` to force the non-split-K stage1 — an environment prerequisite for K3 on gfx942, not specific to this feature._ **Correctness** - Unit (`-k \"not tp4 and not tp8\"`): _TODO PASS/FAIL_ (`N passed`). - Multi-GPU parity (TP4 + TP8, overlap & shard tails): _TODO PASS/FAIL_. - Accuracy parity vs `main` (fused tail must not change outputs within tolerance): _TODO — e.g. gsm8k / small lm_eval, baseline vs candidate._ **Decode latency / throughput** — batch below `VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD` so the overlap tail (Tier 1) is active. Baseline = `main` (Tier 2 only); Candidate = this PR. | Batch (tokens) | Tail tier | Baseline TPOT (ms) | Candidate TPOT (ms) | Δ % | Baseline tok/s | Candidate tok/s | Δ % | |---:|:--|---:|---:|---:|---:|---:|---:| | 1 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 8 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 32 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 128 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 256 | overlap | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | | 4096 (prefill) | column-parallel | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | **Overlap evidence** — trace excerpt showing the shared-expert `all_reduce` running on the aux stream concurrently with the routed up-projection `mm` on the default stream (confirms the reduce is hidden, not merely reordered): _TODO — paste kernel timeline._ ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/51437", "createdAt": "2026-08-07T19:43:05Z", "updatedAt": "2026-08-13T01:41:58Z", "timestamp": "2026-08-13T01:41:58Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "mpashkovskii", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:14e06f00f22fc0c1dff6", "signalId": "github:vllm-project/vllm:pull_request:49171", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Skip logits and sampling for unfinished prefills", "text": "## Purpose Model Runner V2 currently produces one sampling-logit row for every request in the common non-speculative path, including chunked-prefill requests whose prefill does not finish in the current step. Although these sampled tokens are discarded later by setting `num_sampled` to zero, the model has already performed the vocabulary projection and the sampler has already applied logits processors and sampling kernels. This introduces unnecessary GPU work, especially for workloads with multiple concurrent long prompts and a small `max_num_batched_tokens`. This PR avoids that work by: - Determining whether each request finishes prefill before constructing the logits batch. - Assigning zero logits rows to unfinished chunked-prefill requests. - Compacting mixed prefill/decode batches so output projection and sampling run only for requests that can produce an output token. - Short-circuiting `compute_logits` and sampling when the entire batch consists of unfinished prefill chunks. - Expanding compact sampled-token and NaN-count results back to the original per-request layout with a small Triton kernel, preserving existing downstream interfaces. - Preserving the existing pure-decode fast path. The change is limited to Model Runner V2's common non-speculative path. Speculative decoding remains unchanged. LoRA batches retain the existing dense logits layout because the per-request mapping is reused by prompt-logprob projection. This should reduce unnecessary vocabulary-projection and sampling work for chunked-prefill-heavy serving workloads without changing generated outputs. ## Test ### start server ``` VLLM_USE_V2_MODEL_RUNNER=1 \\ vllm serve Qwen3.5-4B \\ --served-model-name Qwen3.5-4B \\ --host 0.0.0.0 \\ --port 8000 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 2048 \\ --max-num-seqs 32 \\ --gpu-memory-utilization 0.90 \\ --disable-log-stats ``` ### vllm bench ``` vllm bench serve \\ --backend vllm \\ --host 127.0.0.1 \\ --port 8000 \\ --endpoint /v1/completions \\ --model Qwen3.5-4B \\ --tokenizer Qwen3.5-4B \\ --dataset-name random \\ --random-input-len 8192 \\ --random-output-len 32 \\ --random-range-ratio 0 \\ --ignore-eos \\ --request-rate inf \\ --max-concurrency 32 \\ --num-warmups 16 \\ --num-prompts 256 \\ --seed 0 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,99 ``` ## Test Result ### before \"IMG_3103\" ### after \"IMG_3104\" - Output token throughput increased from **174.72 tok/s** to **191.97 tok/s**, an improvement of **9.87%**. - Mean TTFT decreased from **4311.56 ms** to **3904.83 ms**, a reduction of **9.43%**. - Mean TPOT decreased from **43.97 ms** to **40.15 ms**, a reduction of **8.69%**. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49171", "createdAt": "2026-07-20T08:40:18Z", "updatedAt": "2026-08-13T01:37:05Z", "timestamp": "2026-08-13T01:37:05Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend", "speculative-decoding", "ci/build", "v1", "multi-modality", "nvidia", "mrv2", "rust" ], "author": "jesse996", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4b2e04ae6d26675e1d51", "signalId": "github:vllm-project/vllm:pull_request:41567", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:41567", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine", "text": "Wire factory and ec_transfer config; add two-process e2e test, EPD full-pipeline script, and README notes. ## Purpose - Add **`ECMooncakeConnector`**: encoder-cache (EC) transfer over **Mooncake TransferEngine** (HTTP registry + ZMQ coordination + pull path), for disaggregated setups where consumers load EC tensors without relying on shared filesystem. - Register the connector in **`ECConnectorFactory`** and document **`ECTransferConfig.ec_connector`** options (`ECExampleConnector` vs `ECMooncakeConnector`, extra config expectations). - Add **integration coverage / ops glue**: - **Two-process CUDA e2e** (`test_ec_mooncake_transfer_e2e.py`): producer on `cuda:0`, consumer on `cuda:1`, registry + tensor equality check. - **EPD full-pipeline script** (`run_epd_mooncake_ec_full_pipeline.sh`): baseline vs 1E+1PD with Mooncake EC + proxy (optional/heavy path). - **README** section for how to run the Mooncake smoke test and dependencies. ## Test Plan **Lightweight (connector transfer only)** - Requires: **2+ CUDA GPUs**, `mooncake-transfer-engine`, `pyzmq`, `httpx`, `fastapi`, `uvicorn`, and a **built** vLLM (`import vllm` / `vllm._C` available). - From repo root: ```bash PYTHONPATH=. MOONCAKE_EC_PROTOCOL=tcp python tests/v1/ec_connector/integration/test_ec_mooncake_transfer_e2e.py", "url": "https://github.com/vllm-project/vllm/pull/41567", "createdAt": "2026-05-03T18:28:57Z", "updatedAt": "2026-08-13T01:35:19Z", "timestamp": "2026-08-13T01:35:19Z", "metrics": { "reactions": 5, "comments": 4 }, "labels": [ "documentation", "v1", "kv-connector" ], "author": "stmatengss", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c0051fdd5d5d1548a944", "signalId": "github:vllm-project/vllm:pull_request:51821", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51821", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][CI] Restore the DeepSeek-V4 input GEMM override point", "text": "## Purpose The GSM8K accuracy job for `amd/DeepSeek-V4-Flash-NVFP4` on gfx950 reports 0.0000 against a threshold of 0.92. It is not a near miss: the server comes up clean and then answers every one of the questions with an unparsable run of repeated tokens, so the failure is in the numerics rather than in the harness. A bisect over the window in which the job turned red lands on 79c865b838e34f7a98a936771284773819d79c8f, \"[Perf] Narrow DeepSeek V4 eager CUDA graph region\" ([#51430](https://github.com/vllm-project/vllm/pull/51430)). Alongside narrowing the graph region, that commit deleted `DeepseekV4Attention._fused_wqa_wkv_gemm` and inlined `self.fused_wqa_wkv(hidden_states)[0]` at its only call site. The deleted method is the override point the ROCm attention layer implements. On ROCm the `fused_wqa_wkv` weight is preshuffled in place when the model loads and its block scale is kept to the side, so the only thing that can multiply it correctly afterwards is AITER's preshuffled-B GEMM. With the call inlined, the input projection of every attention layer runs the plain linear over a shuffled weight, and the model produces garbage from the first layer on. The same removal leaves the ROCm override's own fallback calling a `super()` method that no longer exists, so any path that took the fallback would raise `AttributeError`. Nothing about this is visible on CUDA, and the ROCm override still reads as if it were connected, which is why the removal went through. ## Relationship to the other #51430 follow-ups Two open PRs address #51430, both for a different symptom on NVIDIA, and neither covers this one. [#51750](https://github.com/vllm-project/vllm/pull/51750) is an auto-generated full revert opened after the B200 nightly saw GSM8K fall to 0.0432 with MTP acceptance collapsing to 4.1%; it would undo this bug as a side effect, but it is still a draft and the author of #51430 has proposed a narrower fix-forward instead. That fix-forward, [#51768](https://github.com/vllm-project/vllm/pull/51768), attributes the NVIDIA failure to the legacy V1 model runner's breakable PIECEWISE CUDA graph path and guards that combination in `vllm/config/vllm.py`, leaving the attention implementation exactly as #51430 left it. If #51768 is the one that lands, ROCm stays broken. The two failures are independent. This one does not need a model runner, a graph mode or MTP to show up: it reproduces at compilation mode 0 with no graph captured, and at `FULL_DECODE_ONLY`, which #51768 explicitly keeps allowed. It also cannot appear on CUDA, since nothing preshuffles the weight there. This is not an alternative to either of them. On CUDA the restored method computes exactly what the inlined call computed, so this change is a no-op there and does nothing for the B200 failure — #51768 is still needed for that. In the other direction, if #51750 lands after all, this one becomes redundant and I will close it. Searches for open PRs mentioning `fused_wqa_wkv`, and for open PRs referencing #51430, turned up nothing else that restores this method. ## The change Restore `_fused_wqa_wkv_gemm` on the base class as a plain call into `fused_wqa_wkv`, and send the call site back through it. That is the whole change, eight lines. The narrowed CUDA graph region and everything else from #51430 are untouched. ## Test Plan gfx950 (MI355X), with the model, server arguments and environment taken verbatim from `tests/evals/gsm8k/configs/DeepSeek-V4-Flash-NVFP4.yaml`: tensor parallel 8, fp8 KV cache, compilation mode 3 with `FULL_DECODE_ONLY`, AITER enabled. GSM8K at 8 shots, shortened to 400 questions to keep the turnaround reasonable. The same evaluation was run before the change on the unmodified checkout and after it. To rule out the parts of #51430 that the title points at, the failure was also reproduced at tensor parallel 1 and at compilation mode 0, where no graph is captured at all. ## Test Result Before the change, on current main, accuracy is 0.0000 with an invalid rate of 1.000 — no answer is parsable. It stays 0.0000 at tensor parallel 1 and at compilation mode 0, so neither sharding nor graph capture is what breaks it. After the change, accuracy is 0.953 with an invalid rate of 0.000, comfortably above the 0.84 floor the threshold and its tolerance imply. Reverting #51430 in full gives an equivalent result, which confirms the method removal is the only part of that commit involved. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51821", "createdAt": "2026-08-11T13:05:48Z", "updatedAt": "2026-08-13T01:33:53Z", "timestamp": "2026-08-13T01:33:53Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "rocm", "deepseek" ], "author": "stefankoncarevic", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:8a9f58d1c1d9442a794f", "signalId": "github:vllm-project/vllm:pull_request:52043", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52043", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Force source builds for hybrid dependencies", "text": "Hybrid language-model CI began failing before pytest while `mamba-ssm` and `causal-conv1d` probed guessed GitHub release-wheel URLs. The requested CUDA/Torch and ROCm/Torch wheels do not exist, so the normal path is a caught HTTP 404 followed by a source build. During today's intermittent GitHub connectivity problems, some requests instead ended with `Remote end closed connection without response`, which escaped the installers' narrow exception handling and aborted the jobs. This change bypasses the fragile release probe and directly performs the same source builds used by successful runs. - Set `MAMBA_FORCE_BUILD=TRUE` and `CAUSAL_CONV1D_FORCE_BUILD=TRUE` for all affected language-model dependency installs. - Cover the active Hybrid, Granite L4, Extended Generation, and AMD mirror jobs, plus the analogous MI300 and MI355 jobs in `.buildkite/test-amd.yaml`. - Motivated by the [AMD Hybrid failure](https://buildkite.com/vllm/ci/builds/83591/canvas?jid=019ff759-6a91-445b-b5e2-55e2a28240a3&tab=output) and [NVIDIA Hybrid failure](https://buildkite.com/vllm/ci/builds/83591/canvas?jid=019ff75c-a52f-450b-a9d7-28c4a6e02bc2&tab=output), both of which stopped during dependency setup before pytest.", "url": "https://github.com/vllm-project/vllm/pull/52043", "createdAt": "2026-08-12T20:29:46Z", "updatedAt": "2026-08-13T01:33:30Z", "timestamp": "2026-08-13T01:33:30Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:62dbf2a99341a463848f", "signalId": "github:vllm-project/vllm:issue:52065", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52065", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DeepSeek-V4-Flash-0731 + DSpark fails to start on vLLM 0.27.0 / H100 sm90 (DeepGEMM CUDA_ERROR_ILLEGAL_ADDRESS); works on 0.26.0", "text": "### Your current environment - **GPU**: 8× NVIDIA H100 80GB HBM3 (sm90a), single node - **vLLM**: 0.27.0 (pip install into a clean venv) - **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` (local copy of the HF repo) - **Parallelism**: TP=8, expert-parallel, fp8 KV cache - **Also installed / compared against**: vLLM 0.26.0 in a separate venv on the *same* image and the *same* node Serve command (DSpark arm): ```bash vllm serve /models/DeepSeek-V4-Flash-0731 \\ --served-model-name DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --enable-expert-parallel \\ --max-model-len 65536 \\ --gpu-memory-utilization 0.90 \\ --kv-cache-dtype fp8 \\ --max-num-batched-tokens 8192 \\ --max-num-seqs 128 \\ --seed 0 \\ --host 0.0.0.0 --port 8080 \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"greedy\"}' ``` ### 🐛 Describe the bug On vLLM **0.27.0**, `DeepSeek-V4-Flash-0731` with **DSpark speculative decoding** never reaches `Application startup complete` on sm90. The DSpark draft model loads successfully, and then roughly 8 minutes later (during engine init / graph capture) **all 8 workers** die with a DeepGEMM illegal memory access: ``` INFO [dspark.py:491] DSpark draft model loaded: 96 params ... ERROR [multiproc_executor.py:1018] WorkerProc hit an exception. RuntimeError: CUDA driver error (/workspace/.deps/deepgemm-src/csrc/apis/../jit_kernels/impls/../../jit/handle.hpp:154): 700 (CUDA_ERROR_ILLEGAL_ADDRESS, an illegal memory access was encountered) ``` **The identical configuration works on 0.26.0.** Same node, same 8 GPUs, same container, same model files, same flags — only the venv (and therefore the vLLM version) differs. On 0.26.0 the server reaches `Application startup complete` and serves `/v1/chat/completions` correctly. Reproduced **3/3** on 0.27.0 across independent runs, with prefix caching both enabled and disabled (it makes no difference to this failure): | vLLM | DSpark | prefix caching | result | |---|---|---|---| | 0.27.0 | on | off | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.27.0 | on | on | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.27.0 | on | off | `CUDA_ERROR_ILLEGAL_ADDRESS` (DeepGEMM `handle.hpp:154`) | | 0.26.0 | on | off | serves OK | ### Possibly a broader 0.27.0 / sm90 problem, not only DSpark Worth flagging for triage: on 0.27.0 this model also fails to start **with DSpark disabled**, but with a *different* signature — a FlashMLA sparse **decode** tensormap assertion: ``` Assertion `res == CUresult::CUDA_SUCCESS` failed (/workspace/.deps/flashmla-src/csrc/sm90/decode/sparse_fp8/instantiations/../splitkv_mla.cuh:742) Engine core initialization failed ``` That arm also serves fine on 0.26.0. So on sm90, 0.27.0 appears unable to start this model in either configuration; DSpark just fails in a different subsystem than the non-DSpark path. I did not bisect between 0.26.0 and 0.27.0. Setting `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` does **not** help 0.27.0 (same `splitkv_mla.cuh:742` assertion). On 0.26.0 that env var is a valid workaround for an unrelated flashinfer JIT issue and the server still starts. ### Related issues - #51822 — same model + DSpark on 0.27.0, but H200 / TP=4 and a different failure: `device-side assert triggered` surfacing in `take_draft_token_ids` → `copy_event.synchronize()`. Possibly the same root cause manifesting asynchronously; filing separately because the signature, hardware and TP differ. - #50660 — same model on **0.26.0**: FlashMLA sparse **prefill** tensormap assertion (`sm90/prefill/sparse/.../phase1.cuh:614`) that I can reproduce 3/3 when DSpark is combined with prefix caching. Different version and different kernel, but the same `Assertion res == CUresult::CUDA_SUCCESS` family — these tensormap asserts guard `cuTensorMapEncodeTiled`, which returns `CUDA_ERROR_INVALID_VALUE` on a zero/degenerate tensor dimension. ### Before submitting a new issue... - [x] I searched existing issues (#51822, #50660, #50576, #51041) and believe this signature is not yet reported.", "url": "https://github.com/vllm-project/vllm/issues/52065", "createdAt": "2026-08-12T22:56:56Z", "updatedAt": "2026-08-13T01:26:31Z", "timestamp": "2026-08-13T01:26:31Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [], "author": "DiegoCao", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:fadb38a755cf5a050fcd", "signalId": "github:vllm-project/vllm:pull_request:51218", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51218", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Report FULL_ATTENTION for uniform-base UniformTypeKVCacheSpecs groups instead of UNKNOWN", "text": "## Problem `get_kv_cache_spec_kind()` returns `KVCacheSpecKind.UNKNOWN` for a `UniformTypeKVCacheSpecs` whose members have more than one *inner* kind: ```python if isinstance(kv_cache_spec, UniformTypeKVCacheSpecs): inner_kinds = {get_kv_cache_spec_kind(spec) for spec in kv_cache_spec.kv_cache_specs.values()} if len(inner_kinds) == 1: return next(iter(inner_kinds)) return KVCacheSpecKind.UNKNOWN # <-- here ``` But such a group is **only ever formed when all members share one registered `uniform_type_base_spec`** — that is the merge condition itself (`UniformTypeKVCacheSpecs.is_uniform_type` → `KVCacheSpec.is_uniform_with_collection`). So `UNKNOWN` discards information the merge already established. ## Why it matters KV-cache events are tagged with this kind (`v1/core/kv_cache_manager.py`), and consumers that filter on it drop `UNKNOWN` outright. Affected models therefore lose **all** cache-aware routing credit — a KV-aware router scores them as though nothing were cached. Concrete case: a model that registers a main `FullAttentionSpec` K/V cache **and** a differently-shaped `MLAAttentionSpec` index cache. Both register `uniform_type_base_spec=FullAttentionSpec` (`v1/core/single_type_kv_cache_manager.py`), so they merge into a single group — which then reports `UNKNOWN`: ```python >>> get_kv_cache_spec_kind(UniformTypeKVCacheSpecs({\"main\": FullAttentionSpec(...), ... \"idx\": MLAAttentionSpec(...)})) KVCacheSpecKind.UNKNOWN >>> KVCacheSpecRegistry.get_uniform_type_base_spec(FullAttentionSpec(...)) >>> KVCacheSpecRegistry.get_uniform_type_base_spec(MLAAttentionSpec(...)) # same base -> the group IS uniform ``` ## Fix Fall back to the registered base spec, upgrading to `FULL_ATTENTION` only when **every** member resolves to `FullAttentionSpec`. Sliding-window, Mamba and other shapes keep `UNKNOWN` exactly as before, so this is deliberately conservative. ## Validation Verified on a live 2-worker deployment with CPU KV offloading enabled, reading the router's own `router_kv_hit_rate` metric (\"Predicted KV cache hit rate at routing time\") for one 78,163-token prefix: | step | before | after | |---|---:|---:| | cold (nobody has it) | 0.2 % | 0.0 % | | prefix warm in GPU | 99.8 % | **99.8 %** (no regression) | | prefix evicted to host, **served from host** | **0.2 %** | **74.9 %** | In the third row the worker demonstrably served the prefix from host RAM (77,952 external-prefix-cache hits, 1,066 ms vs 7,863 ms cold), yet before the fix the router scored it identically to \"nobody has it\". 74.9 % ≈ **0.75 × 99.8 %**, i.e. exactly the router's host-cache-hit weight applied to a full-prefix match — the credit path engaging at its configured multiplier. Before the fix, the group's `UNKNOWN` kind also meant that enabling self-describing KV events would stamp `UNKNOWN` on **GPU-tier** events, which the same filter drops; GPU-tier routing survived only because the field was absent from the wire. This change makes both tiers report a real kind. ## Notes - Behaviour is unchanged for every group that already returned a concrete kind. - No new imports at module scope (the registry import is local to avoid a cycle with `kv_cache_spec_registry`, which imports from this module). - Happy to add a unit test asserting `FULL_ATTENTION` for a mixed `FullAttentionSpec` + `MLAAttentionSpec` group if you'd like one — pointers to the preferred location welcome.", "url": "https://github.com/vllm-project/vllm/pull/51218", "createdAt": "2026-08-06T02:30:49Z", "updatedAt": "2026-08-13T01:25:24Z", "timestamp": "2026-08-13T01:25:24Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "bug", "ready" ], "author": "yifjiang", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:3490749f3bf597a73b73", "signalId": "github:vllm-project/vllm:pull_request:48804", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48804", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode][V1] Warm Eagle and DFlash/DSpark spec-decode Triton kernels at startup", "text": "## Summary Triton specializes integer arguments whose runtime value is 1 into compile-time constants, producing a separate cubin per \"which params are 1\" combination. Triton also specializes `tl.constexpr` arguments (one cubin per distinct value) and `tl.num_programs` grid axes when they equal 1. Without warmup, the first request in each shape pays a JIT latency spike. This PR adds two standalone warmup modules under `vllm/model_executor/warmup/` that enumerate all relevant parameter combinations for the spec-decode Triton kernels: ### Eagle warmup (`eagle_spec_decode_warmup.py`) 1. **`eagle_prepare_next_token_padded_kernel`** — 2³ int combos × len(BLOCK_SIZES) cache entries 2. **`eagle_prepare_inputs_padded_kernel`** — 2 entries (single-req vs multi-req) 3. **`_mtp_shared_head_rmsnorm_kernel`** — 1 entry (only `tl.constexpr` params, grid dim is not specialized) 4. **`eagle_step_slot_mapping_metadata_kernel`** — len(n_blocks_candidates) × 2² int combos, covering CP × hybrid block factors No-op when Eagle spec decoding is not configured (`num_speculative_tokens` is `None` or 0) or when not on CUDA. ### DFlash/DSpark warmup (`dflash_spec_decode_warmup.py`) 5. **`_prepare_dflash_inputs_kernel`** — shared by the DFlash and DSpark speculators. Its cache key is driven by: - `BLOCK_SIZE` constexpr — computed at runtime as `min(256, next_power_of_2(max_tokens_per_req))`, so it varies with batch composition (small for pure decode, 256 for prefill chunks). This is the main source of multiple cubins; without warmup the first prefill-heavy request pays a JIT latency spike (the reported JIT warning references `BLOCK_SIZE=256, SAMPLE_FROM_ANCHOR=True` for DSpark). - `SAMPLE_FROM_ANCHOR` constexpr — `False` for DFlash, `True` for DSpark. - `PAD_SLOT_ID` constexpr — always `-1`. - Grid `(num_reqs, num_blocks)` — `tl.num_programs` is specialized when an axis equals 1. The remaining i32 scalars (`block_size`, `block_table_stride`, `num_speculative_steps`, `num_query_per_req`, `parallel_drafting_token_id`, `max_num_reqs`, `max_num_tokens`, `max_model_len`) are constant per deployment, so Triton only specializes them when their value happens to be 1 — passing the configured value covers both branches automatically. The warmup enumerates `BLOCK_SIZE` powers of two `{1,2,4,8,16,32,64,128,256}` crossed with the four grid axis==1 combinations `(1,1)/(1,8)/(50,1)/(50,8)` = 36 entries, reading deployment-fixed values from the live `DFlashSpeculator` (covers `DSparkSpeculator` via subclassing). No-op when DFlash/DSpark is not configured. ## Design Both warmups are called from the central `kernel_warmup()` dispatcher in `kernel_warmup.py`, each wrapped in its own `try/except` so a failure does not prevent other warmups from running. Each kernel invocation inside the modules is also individually wrapped in `try/except` so a single shape failure does not abort the rest. Deployment-fixed values are read from the live speculator so the warmup matches the runtime specialization exactly. ## Supersedes #48393 #48393 embedded the warmup logic inside `LLMBaseProposer.dry_run_helper_kernels()` and was stacked on #41481 (still open). This PR takes a different approach: - **Standalone module** under `vllm/model_executor/warmup/` instead of proposer class methods - **No dependency on #41481** — imports kernels directly from `vllm.v1.spec_decode.utils` and `vllm.v1.worker.gpu.spec_decode.dflash.speculator` - **Broader coverage** — also warms `eagle_step_slot_mapping_metadata_kernel` (not in #48393) and adds the DFlash/DSpark warmup - **Central dispatch** — called from `kernel_warmup()` alongside other warmups, consistent with mHC / sparse-MLA / DeepGEMM patterns ## Test commands ```bash .venv/bin/python -m pytest tests/v1/spec_decode/test_helper_kernel_warmup.py -v ``` DFlash/DSpark model evals should be run to confirm the `jit_monitor` no longer reports `_prepare_dflash_inputs_kernel` JIT compilation during inference (the reported warning referenced `BLOCK_SIZE=256, SAMPLE_FROM_ANCHOR=True` for DSpark). ## Pre-commit `ruff check`, `ruff format`, `mypy`, `check-torch-cuda-call`, `check-spdx-headers`, `check-forbidden-imports`, and `signoff-commit` all pass on the new files. ## AI assistance This PR was prepared with AI assistance (opencode). Every changed line was reviewed by a human.", "url": "https://github.com/vllm-project/vllm/pull/48804", "createdAt": "2026-07-16T03:42:03Z", "updatedAt": "2026-08-13T01:17:04Z", "timestamp": "2026-08-13T01:17:04Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "speculative-decoding", "needs-rebase", "v1", "cpu", "mrv2" ], "author": "leihuang-sketch", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:51c4ad14a17bcbc9fc9f", "signalId": "github:vllm-project/vllm:pull_request:50366", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50366", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "DCP: consume owner-sharded Top-K candidates through symmetric memory", "text": "# DCP: consume owner-sharded Top-K candidates through symmetric memory ## Summary This change replaces the sparse-indexer candidate AllGather with direct consumption from one symmetric-memory candidate shard per DCP rank. Each producer stores only its local `(rows, candidates, score_and_token)` shard. Symmetric-memory rendezvous gives every consumer a peer tensor view for each owner's allocation. The stable Top-K kernel reads those views and selects globally without first constructing a full gathered candidate tensor. This is the symmetric-memory counterpart to the generic CUDA VMM design in #50010. It preserves #50010 as the backend for asymmetric peer-addressable objects; this PR tests whether the existing PyTorch symmetric-memory dependency offers a narrower integration for the uniform DCP Top-K case. ## Data flow ```text rank 0 local candidates --\\ rank 1 local candidates ---+-> stable Top-K reads owner shards directly rank 2 local candidates ---+-> global token ids rank 3 local candidates --/ ``` There is no `world_size x local_candidates` inbox on each rank. At DCP4, the logical candidate payload remains one shard per rank instead of four shards per rank. ## Synchronization Each owner publishes a device-side write epoch after packing its shard. A consumer waits until all owner epochs are visible, reads all shards, and then publishes its read epoch. Before reusing an owner shard, its producer waits until every consumer has acknowledged the prior epoch. The protocol is CUDA-graph compatible and fail-closed. Once selected, it does not fall back to AllGather, the full symmetric-memory inbox, or VMM. ## Scope - Owner-local symmetric-memory allocation and peer-view discovery. - Direct peer-shard loads in the existing stable Top-K selection kernel. - Bounded decode dispatch behind an experimental environment flag. - Explicit phase/shape policy outside that bound; no error recovery fallback. - Distributed exactness, ties, empty-shard, prefill-indexing, and changing-input CUDA-graph tests. ## Validation Validated on 4x NVIDIA GB200: - Exact match with explicit AllGather: passed. - Changing-input CUDA-graph replay: passed. - Rows `1, 17, 64, 96`, ties, an empty owner shard, and prefill indexing: passed. - Repository checks including Ruff, formatting, mypy, and SPDX: passed. The full distributed test reports `1 passed`. ### Performance The matched DCP4 GLM-5.2 DSA sweep used 2048 candidates per rank, 4096 score columns, fixed 1800 MHz SM clocks, 100 CUDA-graph replays per sample, 30 samples, maximum-rank timing, and three fresh process runs. Against explicit candidate AllGather: | Rows | Explicit (us) | Symmetric peer shards (us) | Saved (us) | Improvement | |---:|---:|---:|---:|---:| | 1 | 44.429 | 41.042 | 3.387 | 7.624% | | 8 | 48.581 | 41.392 | 7.189 | 14.800% | | 32 | 56.431 | 43.247 | 13.184 | 23.363% | | 64 | 64.690 | 45.891 | 18.799 | 29.060% | The symmetric path's process CV was 0.021%-0.130%. Against the unchanged generic VMM implementation from #50010 at commit `5198c756b31f963c821a0c7752e50245d0da3a93`: | Rows | #50010 VMM (us) | Symmetric peer shards (us) | Symmetric improvement | |---:|---:|---:|---:| | 1 | 41.049 | 41.042 | 0.018% | | 8 | 41.467 | 41.392 | 0.183% | | 32 | 44.017 | 43.247 | 1.750% | | 64 | 46.014 | 45.891 | 0.267% | At 64 rows, explicit AllGather materializes 4,194,304 candidate bytes per rank. This path stores 1,048,576 candidate bytes and allocates 1,048,624 bytes including synchronization metadata. The VMM path allocates 2,097,152 bytes for the same candidate payload because of VMM allocation granularity. This candidate proceeds because it keeps the one-shard-per-rank property, wins against explicit AllGather at every measured row count, matches or slightly beats #50010, and halves the actual allocation relative to #50010 at this bound. It remains a separate DSA contribution; Kimi-K3 does not exercise this Top-K path. This is an operator and workspace result; the PR does not claim an end-to-end serving gain. Head commit: `a3db29b465cf540197bd8d2b20dc75bd5c71a435`.", "url": "https://github.com/vllm-project/vllm/pull/50366", "createdAt": "2026-07-30T02:48:12Z", "updatedAt": "2026-08-13T01:13:02Z", "timestamp": "2026-08-13T01:13:02Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "needs-rebase", "v1" ], "author": "foraxe", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6ec1c7266ee2bf02a776", "signalId": "github:vllm-project/vllm:pull_request:52067", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52067", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Forward ownership in KV cache events", "text": "## Purpose Add optional ownership information to KV cache events so events emitted by a KV secondary tier can be distinguished from framework-owned events. No conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py", "url": "https://github.com/vllm-project/vllm/pull/52067", "createdAt": "2026-08-12T23:23:46Z", "updatedAt": "2026-08-13T01:11:25Z", "timestamp": "2026-08-13T01:11:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "mkhazraee", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e616787ad87d3dd66eb0", "signalId": "github:vllm-project/vllm:pull_request:47708", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47708", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Gigachat 3.5 support", "text": "## Purpose GigaChat-3.5-432B-A28B is a 432B-total / 28B-active MoE on a DeepSeek-V3-style backbone (MLA + DeepSeek MoE) with a hybrid attention stack: most layers use a Qwen3-Next Gated-Delta-Net (GDN) linear-attention block, and a periodic subset keeps full MLA. It adds gated zero-centered RMSNorm (low-rank gating bottleneck, four-norm sandwich around attention and MLP), gated attention before the output projection, learnable MLA query/key scales (`alpha_q` / `alpha_kv`), and Multi-Token Prediction (NextN) for speculative decoding. Supports bf16 and block-fp8. The implementation reuses vLLM's DeepSeek-V3 MLA, `FusedMoE`, and Qwen3-Next GDN blocks. ## Test Plan Setup: 8×H100 + TP=8 + EP Checkpoint: `ai-sage/GigaChat3.5-432B-A28B` **Serve**: ```bash vllm serve ai-sage/GigaChat3.5-432B-A28B \\ --tensor-parallel-size 8 --enable-expert-parallel \\ --gpu-memory-utilization 0.9 --max-model-len 8192 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' ``` **MATH-500 accuracy**: ```bash CKPT=ai-sage/GigaChat3.5-432B-A28B lm_eval --model local-chat-completions \\ --model_args \"model=$CKPT,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=32,tokenizer=$CKPT,trust_remote_code=True,tokenized_requests=False,max_gen_toks=2048\" \\ --apply_chat_template --fewshot_as_multiturn --tasks minerva_math500 --batch_size 32 ``` **MTP benchmark** (ShareGPT×300, concurrency 32, greedy): ```bash SRC=\"https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json\" DST=\"./ShareGPT_V3_unfiltered_cleaned_split.json\" [ -f \"$DST\" ] || wget -c -O \"$DST\" \"$SRC\" CKPT=ai-sage/GigaChat3.5-432B-A28B vllm bench serve --model $CKPT --base-url http://localhost:8000 \\ --backend vllm --endpoint /v1/completions \\ --dataset-name sharegpt --dataset-path \"$DST\" \\ --num-prompts 300 --max-concurrency 32 --temperature 0 ``` ## Test Result **MATH-500 accuracy** | Metric | no-MTP | MTP-2 | HF card | |---|---|---|---| | MATH-500 (`math_verify`) | **87.0% ± 1.5** | **84.8% ± 1.6** | **86%** | Matches the model card, the no-MTP and MTP-2 scores agree within noise (overlapping 95% CIs). **MTP performance** | Config | Output tok/s | Speedup | req/s | Mean TPOT | Accept. len | Draft acceptance | |---|---|---|---|---|---|---| | no-MTP | 1059 | 1.00× | 4.76 | 27.1 ms | — | — | | MTP-1 | 1457 | **1.38×** | 6.55 | 20.1 ms | 1.81 | 81.5% | | MTP-2 | 1600 | **1.51×** | 7.20 | 18.9 ms | 2.28 | 64.2% (81.9% / 46.5%) | Tool calling (`--tool-call-parser gigachat35`) verified end-to-end — unit tests pass. ---
Essential Elements of an Effective PR Description Checklist - [x] Purpose — new model support (`GigaChat35ForCausalLM` + MTP draft + `gigachat35` tool parser). - [x] Test plan — serve + lm-eval (MATH-500) + `vllm bench serve` commands. - [x] Test results — MATH-500 accuracy (vs HF card) + MTP acceptance / throughput. - [x] Documentation — `docs/models/supported_models.md`, `docs/features/tool_calling.md`.
", "url": "https://github.com/vllm-project/vllm/pull/47708", "createdAt": "2026-07-06T07:57:53Z", "updatedAt": "2026-08-13T01:10:42Z", "timestamp": "2026-08-13T01:10:42Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "new-model", "speculative-decoding", "needs-rebase", "v1", "tool-calling" ], "author": "perkyfever", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9ad95550a7431f3ec2c5", "signalId": "github:vllm-project/vllm:pull_request:52077", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52077", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Handle empty YAML config in `--config` parsing", "text": "## Description Passing an empty or comments-only YAML file to `--config` caused `yaml.safe_load()` to return `None`, which then crashed with `AttributeError: 'NoneType' object has no attribute 'items'` in `FlexibleArgumentParser.load_config_file`. This PR: - Treats `None` (empty/comments-only YAML) as an empty configuration so the parser proceeds with defaults and any explicit CLI args. - Validates that the loaded YAML is a mapping at the top level. For lists, scalars, or other non-mapping types, raises a clear `ValueError` with an actionable message instead of an obscure internal `AttributeError`. ## Fix In `vllm/utils/argparse_utils.py` `load_config_file()`: ```python if config is None: config = {} if not isinstance(config, dict): raise ValueError( \"Config file must contain a YAML mapping at the top level. \" f\"Got {type(config).__name__} instead in {file_path}.\" ) ``` ## Tests Added 5 regression tests in `tests/utils_/test_argparse_utils.py`: - `test_empty_yaml_config` — empty YAML file treated as empty config - `test_comments_only_yaml_config` — comments-only YAML treated as empty config - `test_empty_yaml_config_with_cli_overrides` — empty YAML + CLI args keep CLI values - `test_yaml_top_level_list_rejected` — top-level YAML list raises `ValueError` - `test_yaml_top_level_scalar_rejected` — top-level YAML scalar raises `ValueError` All existing tests continue to pass. Closes #51618", "url": "https://github.com/vllm-project/vllm/pull/52077", "createdAt": "2026-08-13T00:58:25Z", "updatedAt": "2026-08-13T00:59:01Z", "timestamp": "2026-08-13T00:59:01Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "veerareddyvishal144", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1e8aca166e2b9cc7ed6b", "signalId": "github:vllm-project/vllm:pull_request:52073", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52073", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Test extract_hidden_states on NemotronH hybrid models", "text": "## Purpose `NemotronHForCausalLM` is the architecture vLLM's own contributor docs point to as the reference implementation for Mamba2 + attention hybrids, and it already implements the `EagleModelMixin` / `SupportsEagle3` hooks that `extract_hidden_states` consumes. It has no coverage in the `extract_hidden_states` integration tests. The only hybrid verifier currently covered there is Qwen3.5 (`test_extract_hidden_states_qwen35_hybrid_smoke`), whose stack is Mamba2 + full attention. NemotronH differs in two ways that both land on the arithmetic this feature is sensitive to: - It builds its layer stack from `hybrid_override_pattern`, interleaving **MLP and MoE** layers with Mamba2 and attention. A given aux layer id is therefore not necessarily an attention or Mamba layer, and the stack contains two block types the current tests never exercise. - `NemotronHForCausalLMConfig.verify_and_update_config` forces `mamba_ssm_cache_dtype` to `float32`, so its Mamba page size is materially larger than a model whose SSM state stays in the model dtype. Both feed the Mamba page size, and the hidden-state cache group's block size is derived from the common page it has to align to (`get_kv_cache_groups`, the `HiddenStateCacheSpec` branch). That is exactly the arithmetic that regressed silently for hybrid verifiers in #46301, where shapes stayed correct while every extracted value read back as zero — a failure mode that yielded a draft model with ~0% acceptance and \"converged\" cleanly in training before anyone noticed. **This is coverage for a supported-but-untested combination, not a bug fix.** Before writing the test I verified on an A100-80GB against `nvidia/Nemotron-H-8B-Base-8K` that extraction works today and is numerically correct, and that the resolved geometry genuinely differs from the covered case: | | attention block size | mamba page padding | |---|---|---| | Qwen3.5-0.8B (covered today) | 544 | 1.49% | | Nemotron-H-8B | **1056** | **1.15%** | With real weights and aux ids `[10, 26, 40]` over a 13-token prompt, the extracted states are distinct and deepen as expected — i.e. correct, not a repeated buffer: | pair | cosine | max abs diff | |---|---|---| | L10 vs L26 | 0.974 | 33.1 | | L10 vs L40 | 0.318 | 1866.2 | | L26 vs L40 | 0.334 | 1882.5 | `abs_mean` per layer: 0.084 → 0.377 → 2.268. Follows the precedent of #39949 (hybrid support) and #46301 (hybrid block-size fix). ## Test Plan Adds `test_extract_hidden_states_nemotron_h_hybrid_smoke`, built on a tiny synthetic `NemotronHConfig` rather than a real checkpoint, so it fits the 18 GB device the `Extract Hidden States Integration` CI job runs on. The pattern `\"M-M*M-ME*E-ME\"` covers all four layer types, and the aux layer ids `[4, 8, 12]` capture the outputs of an attention layer, an MoE layer and a Mamba2 layer respectively (aux id `k` is the residual stream after layer `k-1`). Assertions come from the existing `get_and_check_output` helper: shape, token-id alignment, and states-not-all-zero. ```bash export VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s v1/kv_connector/extract_hidden_states_integration/test_extraction.py \\ -k \"nemotron_h or qwen35\" ``` ## Test Result Both the new test and the existing Qwen3.5 hybrid test pass, on one A100-80GB, against this branch: ``` tests/.../test_extraction.py::test_extract_hidden_states_qwen35_hybrid_smoke PASSED tests/.../test_extraction.py::test_extract_hidden_states_nemotron_h_hybrid_smoke PASSED =========== 2 passed, 2 deselected, 18 warnings in 622.10s (0:10:22) =========== ``` The Qwen3.5 case is included as a no-regression control. Lint on the changed file: ``` ruff check -> All checks passed! ruff format -> 1 file already formatted ``` Worth noting that the tiny config lands in its own alignment regime rather than mirroring a real checkpoint's, which is useful coverage in itself: ``` Using auxiliary layers from speculative config: (4, 8, 12) Setting attention block size to 48 tokens to ensure that attention page size is >= mamba page size. Padding mamba page size by 29.73% to ensure that mamba page size and attention page size are exactly equal. ``` (against 1056 / 1.15% for the real Nemotron-H-8B, and 544 / 1.49% for Qwen3.5.) ## Note on what this test does and does not assert Worth stating explicitly, because it bounds what the test can catch. Under `load_format=\"dummy\"` the weights are drawn from `U(-1e-3, 1e-3)`, and NemotronH's MLP activation is relu², so each block contributes on the order of `1e-10` of the residual and all captured layers read back as a bfloat16 rounding of the embeddings. (Measured: `abs_mean` identical to the last digit across all three aux slots under dummy weights, matching a standalone simulation of the embedding magnitude to three significant figures; the real-weights numbers above are the contrast. Qwen3.5 does not show this because SwiGLU stays linear near zero.) So this test asserts the plumbing — shapes, token alignment, non-zero states through a Mamba2 + attention + MoE stack — and deliberately does not assert per-layer values, which `test_extract_hidden_states_with_predictable_dummy_model` already covers with a model that emits known per-layer outputs. The docstring says the same, so a passing run isn't later read as a stronger guarantee than it gives. Prompts are passed as explicit `TokensPrompt` ids rather than text: the tiny config declares `vocab_size=1000` while the borrowed TinyLlama tokenizer emits ids up to ~18.9k, which indexes past the embedding table and trips a device-side assert. 🤖 Generated with [Weave Router](https://router.workweave.ai)", "url": "https://github.com/vllm-project/vllm/pull/52073", "createdAt": "2026-08-13T00:18:21Z", "updatedAt": "2026-08-13T00:58:37Z", "timestamp": "2026-08-13T00:58:37Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "aminsamir45", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:def4fd492d661c39cba4", "signalId": "github:vllm-project/vllm:pull_request:52074", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52074", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] extract_hidden_states supports chunked prefill", "text": "## Summary The `extract_hidden_states` docs state: > Chunked prefill is not compatible with this feature and must be disabled. That appears to be stale. The feature's own integration test exercises chunked prefill deliberately, in `tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py`: ``` 2. **Chunked prefill**: max_num_batched_tokens=128 with ~500-token prompts so each is split across multiple scheduler iterations — verifies hidden states are reassembled correctly. ``` It runs with `max_num_batched_tokens=128` against ~500-token prompts, so each prompt is split across scheduler iterations, and the test asserts the states are reassembled correctly. I found this while enabling the feature on a NemotronH hybrid model, where the engine also ran with chunked prefill enabled and returned correct per-layer hidden states. ## Test Plan Docs-only change; no code paths touched. The behavioural claim is backed by the existing test above. ## Test Result n/a — documentation only.", "url": "https://github.com/vllm-project/vllm/pull/52074", "createdAt": "2026-08-13T00:19:33Z", "updatedAt": "2026-08-13T00:58:27Z", "timestamp": "2026-08-13T00:58:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "aminsamir45", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:52109468c9d1514dd637", "signalId": "github:vllm-project/vllm:pull_request:47808", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47808", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] DSpark confidence-scheduled verification", "text": "Adaptively sizes the DSpark draft-verification budget from per-request confidence instead of always verifying every drafted token. Motivation: fixed-k speculation collapses at high concurrency — once the GPU saturates, verifying 7 drafts per request burns more compute than the accepted tokens return, dropping **below** non-speculative decoding (see table). ## Design - A Triton kernel ranks draft slots by survival probability (cumprod of per-position confidence) and admits the prefix maximizing estimated accepted tokens per millisecond. - Step costs come from curves profiled with dummy steps at startup: a step function at/below the cudagraph capture limit (padding is physical), linear interpolation above it. - Batch-level budget sizing runs on CPU from double-buffered *stale* confidences (no sync); per-request allocation runs on GPU from live values. - Per-request confidence is smoothed with an EMA (`adaptive_verification_ema_alpha`, default 0.8 — measured bias-zero crossing across batch sizes). - Decode cudagraphs become varlen: captured on the token grid, dispatched via `max_query_len`, every captured slot non-empty. Requires `AttentionCGSupport.ALWAYS`, which the DSV4 backends report on SM100. ## Results DeepSeek-V4-Flash-DSpark, TP=4, SM100 (B300), `speed_bench` 256 prompts, 512 output tokens; throughput measured at `32881ec177`; subsequent HEAD commits are audit fixes off the captured-graph hot path (decode `max_q_len` sourcing, CPU upper-bound restoration, non-adaptive request-ordering revert to main), accuracy re-validated at HEAD. Output tok/s: \"pareto\" Adaptive matches fixed-k within ±3% at c≤64 and preserves the spec-decode win at high concurrency where fixed-k goes underwater (at c=256, fixed 7-token is 33% *below* no-spec). Acceptance length falls from ~3.9 to 3.5 at c=256 — the budget deliberately trims drafts as verification tokens become expensive. Note: the no-spec arm ran with a co-tenant job on the other half of the node; its numbers are directionally correct but modestly depressed. Adaptive/fixed arms ran on an idle node. Accuracy (re-measured at HEAD after the audit fixes below): GSM8K **0.945** (c=16) / **0.951** (c=64), 0 invalid (bar 0.84); MTBench **80/80** completed, completions manually checked for repetition/degeneration — none found.
Reproduction commands **Server** (all measurements; ablations via the noted config deltas): ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \\ --tokenizer-mode deepseek_v4 --trust-remote-code \\ --dtype bfloat16 --max-model-len 8192 \\ --tensor-parallel-size 4 --enable-expert-parallel --block-size 256 \\ --gpu-memory-utilization 0.5 --kv-cache-dtype fp8 \\ --max-num-batched-tokens 16384 --max-num-seqs 256 \\ --compilation-config '{\"max_cudagraph_capture_size\":1024}' \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"deepseek-ai/DeepSeek-V4-Flash-DSpark\",\"attention_backend\":\"FLASH_ATTN\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"enable_adaptive_verification\":true}' ``` - fixed 7-token baseline: `\"enable_adaptive_verification\":false` - no-spec baseline: omit `--speculative-config` **Benchmark harness** — `vllm bench serve` (the Python implementation; the CLI falls back to it automatically when the optional Rust `vllm-rs` binary is absent). If your install ships a Rust binary that rejects any flag below, invoke the Python implementation directly — still repo-only: ```bash python -c 'from vllm.benchmarks.serve import add_cli_args, main from vllm.utils.argparse_utils import FlexibleArgumentParser p = FlexibleArgumentParser(); add_cli_args(p); main(p.parse_args())' ``` **GSM8K** (in-tree runner; 1319 questions, 5-shot, temperature 0): ```bash python tests/evals/gsm8k/gsm8k_eval.py --port 8000 \\ --num-questions 1319 --num-shots 5 --max-tokens 256 \\ --temperature 0 --max-concurrency 16 # and 64 ``` **MTBench coherence** (80 prompts, temperature 1; inspect the saved completions for repetition/degeneration, not just the completion count): ```bash vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \\ --base-url http://127.0.0.1:8000 --model dspark-full \\ --tokenizer deepseek-ai/DeepSeek-V4-Flash-DSpark --tokenizer-mode deepseek_v4 \\ --dataset-name hf --dataset-path philschmid/mt-bench \\ --num-prompts 80 --no-oversample --hf-output-len 256 \\ --max-concurrency 16 --request-rate inf --temperature 1 \\ --save-result --save-detailed ``` **Throughput sweep** (per concurrency; one warmup pass at `--speed-bench-output-len 256 --num-prompts 64 --max-concurrency 32` first): ```bash for c in 1 16 32 64 128 256; do vllm bench serve \\ --backend openai-chat --base-url http://127.0.0.1:8000 \\ --endpoint /v1/chat/completions --model dspark-full \\ --tokenizer deepseek-ai/DeepSeek-V4-Flash-DSpark --tokenizer-mode deepseek_v4 \\ --dataset-name speed_bench --dataset-path \\ --speed-bench-dataset-subset qualitative \\ --skip-chat-template --disable-shuffle --temperature 1.0 \\ --speed-bench-output-len 512 --num-prompts 256 --max-concurrency $c \\ --save-result --result-filename adaptive_on_c${c}.json done ``` `--disable-shuffle` plus the fixed prompt set gives every arm identical prompts in identical order; `output_throughput` from the result JSON is the tok/s reported above. The prompt set is an internal speed-bench \"qualitative\" subset; any fixed prompt set of comparable length distribution reproduces the relative behavior, e.g. `--dataset-name random --random-input-len 1024 --random-output-len 512`.
## Limitations - Output logprobs are rejected when confidence-based verification is enabled (verification compacts logits after the forward pass). - Non-SM100 falls back to PIECEWISE decode graphs: correct, but without the varlen-FULL-graph throughput win. ## Tests ``` pytest tests/v1/spec_decode/test_adaptive_verification.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py tests/v1/cudagraph/ -q ``` All pass; full pre-commit green. GSM8K eval config added at `tests/evals/gsm8k/configs/DeepSeek-V4-Flash-DSpark-confidence-TP4.yaml`. ## Not a duplicate Canonical PR for DSpark confidence-based verification; no other open PR touches adaptive draft budgeting. --- AI assistance was used for this change (see commit trailers). 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/47808", "createdAt": "2026-07-07T03:21:26Z", "updatedAt": "2026-08-13T00:57:35Z", "timestamp": "2026-08-13T00:57:35Z", "metrics": { "reactions": 12, "comments": 41 }, "labels": [ "documentation", "performance", "speculative-decoding", "ready", "ci/build", "v1", "qwen", "nvidia", "mrv2" ], "author": "LucasWilkinson", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1be010715b2f45d1ac7f", "signalId": "github:vllm-project/vllm:pull_request:52041", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52041", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Skip broadcasting mm tensor data to workers for prefix-cache-covered items", "text": "## Purpose Fixes #52040. ## Background: why this gap exists and who it affects The EngineCore->TP-worker broadcast was designed when request inputs were token IDs (kilobytes), so shipping them unconditionally was both correct and free. Multimodal grafted large tensors onto the same path, but for the classic workload (a fresh image per chat request) the workers genuinely need every tensor, so unconditional shipping remained necessary. What changed recently is multimodal **prefix caching**: there now exists a state -- \"this item's tokens are fully KV-covered\" -- in which the shipped tensors are provably never consumed, and the broadcast becomes pure per-request CPU overhead. The workload that exposes this is the emerging **streaming / agentic multimodal** pattern: an assistant that re-sends a rolling window of frames every turn (a live camera or video feed with a position-stable layout, so prefix caching covers all history images). For that pattern the cost is linear in window size on every request -- ~19 ms x 25 images = ~0.5 s of EngineCore CPU per turn in our deployment -- while the tensors themselves (fixed-size fp32 pixel patches, ~15.5 MB per image at our processor settings) are re-serialized into the ring for workers that will never touch them. As this workload class grows (streaming VLM assistants, video agents, multi-turn visual memory), the benefit of skipping compounds: it turns fully-cached multimodal TTFT from linear-in-images into flat. When a multimodal item's placeholder span lies entirely within the prefix-cache-covered region, no encoder run can ever be scheduled for it — yet its processed tensors are still serialized into the EngineCore→TP-worker broadcast, per unique item, on every request. For workloads that re-send a rolling window of images (fully prefix-cached), this makes TTFT grow linearly with attached image count: we measured ~19 ms per unique image at TP=4 with 15.5 MB `pixel_values` per image (evidence, flamegraphs, and a duplicate-image control experiment in #52040). This PR extends the existing `MultiModalFeatureSpec.data = None` convention (already used to skip API-server→EngineCore IPC on processor-cache hits) one hop further, to the worker broadcast: - `NewRequestData.from_request` strips `data` from items whose span is fully inside `num_computed_tokens` (new helper `strip_covered_mm_data`, same file). - The scheduler-side `Request` keeps the full features, so nothing is lost engine-side. - **Preemption safety**: requests resumed from preemption re-ship their features via a new `CachedRequestData.resumed_mm_features` field, re-stripped against the post-eviction computed-token count — so items uncovered by eviction regain their data before their encoder runs are scheduled. The worker applies these in its resumed-request branch. - The worker's encoder-input batching already tolerates `data is None` (existing `continue` in `_batch_mm_inputs_from_scheduler`), and the v2-runner path folds resumed requests through `from_request`, so it re-evaluates the strip condition naturally. For the rolling-window workload this removes ~25 tensor broadcasts per request; the fully-cached TTFT floor becomes flat in image count (~4 ms/image of parse+hash remains, which is intrinsic to content addressing). ## Test Plan - Added unit tests for `strip_covered_mm_data` in `tests/v1/core/test_output.py`: fully-covered and boundary items stripped, uncovered kept, already-None preserved, input list not mutated, zero-computed no-op. - Logic additionally verified standalone; I was not able to run the full test suite locally (no GPU dev box) — relying on CI here, and happy to iterate. - Live-workload validation available on our deployment (TP=4, Gemma-4-31B vision, 26-image rolling window): expected effect is the fully-cached floor dropping from ~630 ms to near the 2-image floor. Can report numbers once a build with this change is deployable. ## Test Result Unit tests pass locally at the logic level; awaiting CI for the full suite. ## Notes for reviewers - The strip is deliberately conservative: it only removes data for items that *cannot* have encoder runs scheduled given current `num_computed_tokens` at ship time; every path that changes that quantity downward (preemption/resume) re-ships. - An alternative, larger design would move mm tensor transport to `scheduled_encoder_inputs` time entirely (ship exactly when an encoder run is scheduled). Happy to pursue that instead if preferred — this PR takes the minimal-diff route.", "url": "https://github.com/vllm-project/vllm/pull/52041", "createdAt": "2026-08-12T19:53:25Z", "updatedAt": "2026-08-13T00:55:11Z", "timestamp": "2026-08-13T00:55:11Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "verified" ], "author": "sseanliu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8b51e71a90e657c49eb7", "signalId": "github:vllm-project/vllm:pull_request:47664", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47664", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature][Whisper] Native word-level timestamps (cross-attention + DTW)", "text": "Whisper accepts `timestamp_granularities[]=word` today, but `words` always comes back `null`. This fills it in. ```bash vllm serve openai/whisper-large-v3-turbo --enable-word-timestamps curl localhost:8000/v1/audio/transcriptions \\ -F file=@mary_had_lamb.ogg -F model=openai/whisper-large-v3-turbo \\ -F response_format=verbose_json -F 'timestamp_granularities[]=word' ``` **Before** ```json { \"text\": \" The first words I spoke in the original phonograph...\", \"words\": null } ``` **After** ```json { \"text\": \" The first words I spoke in the original phonograph...\", \"words\": [ {\"word\": \"The\", \"start\": 0.52, \"end\": 0.88}, {\"word\": \"first\", \"start\": 0.88, \"end\": 1.60}, {\"word\": \"words\", \"start\": 1.60, \"end\": 2.02}, {\"word\": \"I\", \"start\": 2.02, \"end\": 2.26}, {\"word\": \"spoke\", \"start\": 2.26, \"end\": 2.66} ] } ``` Off by default; nothing runs without `--enable-word-timestamps`. ## How Whisper never emits word boundaries as tokens — they live in the decoder's cross-attention. Instead of asking a fused backend for its attention probabilities, this records **Q** (and the encoder **K** at prefill) just before the fused call, then recomputes `softmax(Q·Kᵀ)` for the `alignment_heads` only, median-filters it and runs a DTW. Same method as OpenAI Whisper and WhisperX. No second model, no second forward pass, no attention-backend change, and it runs with CUDA graphs and `torch.compile` on. The recording is a registered custom op writing into static buffers, so it stays opaque to Dynamo and safe under graph replay. Code lives in `vllm/v1/worker/gpu/word_align.py` and attaches to the V2 model runner by composition, like `gpu/kv_connector.py`. The runner side is four lifecycle calls. The V1 runner is untouched. Each request gets a capture slot from a bounded pool, and a request that finds the pool full gets no timestamps. Only the layers that actually carry alignment heads are allocated, and the pool is sized by a 2 GiB budget rather than a fixed count, so checkpoints with few alignment layers cover a full batch while the larger ones stay inside the budget: | model | decoder layers | layers with alignment heads | slots within budget | | --- | --- | --- | --- | | `whisper-large-v3-turbo` | 4 | 2 | 215 | | `whisper-large-v3` | 32 | 10 | 43 | | `whisper-small` | 12 | 4 | 179 | The pool is also capped at `max_num_reqs`, so it never allocates for more requests than the server can run. ## Benchmark H200, a `large-v3-turbo`-architecture checkpoint, 40 short clips (median ~2 s) replayed at concurrency 32. `verbose_json` is the baseline because word timestamps require it — it is ~25% slower than plain `json` on its own, with or without this feature. Each row is the mean of two passes after a discarded warm-up pass; runs repeat to within ±2%. | server | request | KV cache | throughput | | --- | --- | --- | --- | | flag off | `verbose_json` | 89,033 tok | 61.6 req/s | | flag on | `verbose_json` | 85,767 tok | 38.6 req/s | | flag on | `+ word` | 85,767 tok | 37.4 req/s | Single-request latency is unchanged within measurement error (~70 ms either way); the cost only shows up under concurrency. Two things to note. Asking for words is nearly free once the flag is on (38.6 → 37.4). Turning the flag on costs ~39% throughput even for requests that never ask for words, because the capture op is compiled into every decode step. Disabling the copies inside the op does not recover it, so the cost is the op's presence in the compiled graph rather than the work it does — making the opt-in per-request instead of per-server is the obvious follow-up. ## Accuracy - vs the OpenAI/HF `find_alignment` reference: mean |Δstart| **0.019 s** over 399 words. - vs an independent forced aligner over ~36k words: median |Δstart| **0.104 s**, identical word order. Close to a dedicated aligner, not a replacement for one. - The change is additive: a new response field, no effect on sampling or on the generated text. ## Tests ```bash pytest tests/models/multimodal/generation/test_whisper_word_align.py # 15 pytest tests/entrypoints/speech_to_text/transcription/test_transcription_word_timestamps.py # 4 ``` Both pass on the default runner with CUDA graphs and `torch.compile` on. The unit tests pin the median filter and the DTW **bit-identical** to transformers' `_median_filter` and `_dynamic_time_warping`, plus the capture op's slot routing and the slot pool. The e2e tests cover populated and monotonic `words[]` that reconstruct the transcript, the per-request opt-in, and concurrent clips each keeping their own timeline. Added to the `Model Runner V2 Core Tests` step so CI runs them under `VLLM_USE_V2_MODEL_RUNNER=1`. Also checked on `whisper-large-v3`, where alignment heads span 10 layers instead of 2: same word count, monotonic, in bounds. Pool exhaustion: 100 concurrent requests against 64 slots returned 0 errors, 98 with timestamps and 2 without, none malformed. ## Notes Requested in [#13400](https://github.com/vllm-project/vllm/issues/13400), [#25750](https://github.com/vllm-project/vllm/issues/25750), [#24302](https://github.com/vllm-project/vllm/issues/24302), [#19556](https://github.com/vllm-project/vllm/issues/19556). [#24209](https://github.com/vllm-project/vllm/pull/24209) added *segment* timestamps and left word-level as future work. Not a duplicate: #13400 was closed *not planned* and no open PR implements this. One open question: carrying the per-token times from the worker to the API needs a `word_align` field on `ModelRunnerOutput` → `EngineCoreOutput` → `CompletionOutput`, ~19 lines of plumbing. I followed how `routed_experts` does it — is there something lighter I missed? Developed with AI assistance (Claude); I reviewed every line and ran every test and benchmark above.", "url": "https://github.com/vllm-project/vllm/pull/47664", "createdAt": "2026-07-05T23:16:23Z", "updatedAt": "2026-08-13T00:52:07Z", "timestamp": "2026-08-13T00:52:07Z", "metrics": { "reactions": 9, "comments": 12 }, "labels": [ "documentation", "frontend", "needs-rebase", "ci/build", "v1", "multi-modality", "mrv2" ], "author": "yusufani", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1650b36f117214cdb5ce", "signalId": "github:vllm-project/vllm:pull_request:49639", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49639", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fix(kernel): restore scalar_t RMSNorm intermediate rounding boundary (#49616)", "text": "## Purpose Fixes #49616. ### Problem and root cause The RMSNorm CUDA kernels kept the normalized value in `float` through the weight multiplication. That changed the rounding boundary relative to the unfused `scalar_t` path and produced bit-level differences during greedy NGRAM/speculative verification. In our nightly comparison, the unpatched path differed in 1,208 of 4,096 elements; the patched path matched all 4,096. ### Solution The kernels now cast the normalized value to `scalar_t` before multiplying by the native-dtype weight in both the regular and FP8-quantized fused paths. This restores the execution-order boundary used by the unfused RMSNorm path. ## Test plan `tests/kernels/core/test_rmsnorm_precision.py` now calls `vllm._custom_ops.rms_norm` directly, compares the weighted kernel output with the scalar_t-rounded composite reference, covers both `float16` and `bfloat16`, and skips cleanly when CUDA/ROCm is unavailable. Local checks: ```text uv run --offline --no-sync ruff check tests/kernels/core/test_rmsnorm_precision.py uv run --offline --no-sync ruff format --check tests/kernels/core/test_rmsnorm_precision.py PYTHONPATH=. uv run --offline --no-sync pytest tests/kernels/core/test_rmsnorm_precision.py -v --noconftest ``` Ruff and formatting passed. The local CPU-only environment collected all 18 cases and skipped them because no CUDA/ROCm device is available; the CUDA validation is left to vLLM CI. The earlier nightly comparison above provides the GPU evidence for the regression. AI assistance disclosure: I used an AI coding assistant to help inspect the existing kernel/test paths and draft the focused regression-test correction. I reviewed and verified the final diff, test behavior, and commit contents myself.", "url": "https://github.com/vllm-project/vllm/pull/49639", "createdAt": "2026-07-23T19:55:06Z", "updatedAt": "2026-08-13T00:50:13Z", "timestamp": "2026-08-13T00:50:13Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [], "author": "Hasnaathussain", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a41f1cf48df8204593c5", "signalId": "github:vllm-project/vllm:pull_request:51649", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51649", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix extreme case in pythonic parser argument json safety", "text": "# [Bugfix] Pythonic tool parsers: argument JSON safety and per-call salvage ## Purpose Follow-up to #48171. Nine defects in the pythonic tool-call family, all found by running real LFM2 agent traces through the parsers. | Model output | Before | After | |---|---|---| | `calc(x=1e999)` | `{\"x\": Infinity}` — invalid JSON | call rejected | | `search(**{'query': 'cats'})` | `{\"null\": {\"query\": \"cats\"}}` | `{\"query\": \"cats\"}` | | `get_weather('Paris', unit='celsius')` | `{\"unit\": \"celsius\"}` — `'Paris'` gone | call rejected | | bad call next to a good one | good call dropped too | only the bad one skipped | | broken quote next to a good call | whole block dropped | good call survives | | two calls, each with a nested quote | one call, argument = the other call's text | both calls, intended values | | a call still arriving mid-stream | an exception with a traceback logged on **every chunk** | quiet wait for more text | | `[foo(x=1])]` — brackets the model got wrong | same, and no log names the offending text | one warning carrying the block | | olmo3 `get_weather(...)` — a **correct** call | one traceback per token of the name | nothing logged | **Provenance**, since the nine are not equally my own: - Rows 1-3 predate #48171 in the shared `handle_single_tool` — they behave identically on the commit before that PR merged. They were deferred from it because fixing them changes all four of `pythonic`, `llama4_pythonic`, `olmo3` and `lfm2`. - Rows 4-5 are long-standing conservative behavior: the block yields no calls at all. Nothing wrong is emitted, so these are robustness, not correctness. - **Row 6 is a further fix for the problems in #48171.** Before that PR the ambiguous-quote block yielded no calls; the nested-quote recovery turned that into a call with corrupted arguments. - Row 7 is log hygiene, and also my own doing: the per-call salvage in row 4 treated \"nothing convertible yet\" as a failure. Nothing was emitted wrongly and no call was lost — it is purely a server-side noise fix. - Rows 8-9 are log hygiene too, and both predate #48171. Row 8 is the same block re-parsed on every chunk, raising at the same bracket every time; skipping it is correct and unchanged, only the reporting is. Row 9 needs no malformed input at all — it fires on every well-formed olmo3 tool call. The five groups are separable — say the word and I will split the PR. ### Rows 5-9 in detail Rows 5 and 6 are block-level, and LFM2 emits several calls in one block often enough for that to matter: in a pinchbench run with LFM2.5-2.6B, 11% of tool-emitting messages carry more than one call (72 of 639). **Row 5** — one broken call takes the block down with it: ``` [read(path='/etc/hosts'), bash(command='grep 'e' a.log', timeout='30s')] ^^^^^^^^^^^^^^^^^^^^^^^ perfectly valid ^^^^^^^ unescaped quotes ``` `ast.parse` sees one syntactic unit, so the whole block fails and `read` is dropped with it — the agent loop gets no tool result at all. The per-call salvage cannot help: it needs a parsed call list, which does not exist. Now the block is split at top-level commas and each segment is parsed on its own, so `read` survives and only the broken call is lost. Splitting runs both a string-aware and a bracket-only scan. A broken quote desynchronizes the former and hides every later separator; string arguments always sit at bracket depth >= 1, so the latter never splits inside them (`exec(command='echo a, b')` stays one segment). A wrongly split segment simply fails to parse and is dropped, so this can only under-recover, never attribute arguments to the wrong call. **Row 6** — the further recovery for #48171 picks a reading that corrupts: ``` [bash(command='sed -n '1,5p' a.py'), bash(command='sed -n '9,12p' b.py')] ^ ^ the only reading that parses closes here, absorbing the entire second call into the first one's command value ``` The client received one call whose `command` was `sed -n '1,5p' a.py'), bash(command='sed -n '9,12p' b.py`, which the agent then executes verbatim, and the second `sed` never happened. That is the defect: not a call that failed to parse, but a call reported as successful carrying an argument the model never wrote. Counting brackets is immune to the broken quote, so the block's call count is now the invariant: a reading that loses calls is rejected as a swallow. That alone restores the pre-#48171 outcome — the block yields no calls, and nothing wrong is emitted. The block then falls through to segment splitting, and each segment is exactly the single-call shape whose nested quote #48171 already recovers (`bash(command='sed -n '1,5p' a.py')`), so both calls come back with their intended values through the existing path rather than a new guess. **Row 7** — the salvage in row 4 mistook a normal streaming state for a failure: ``` [fetch(url='http://x/?a=1'), bash(cm <- all that has arrived ^^ parsed as bash(cm): a positional argument, because 'cmd=' is only half streamed ``` `make_valid_python` withholds a completion while a parameter name is still arriving, but it recognizes that state by counting `=` against `,` in the text before the open paren. An earlier string value that itself contains `=` — a URL query, a `sed` script, an env assignment — defeats the count, and the half-typed name completes to a bare `Name`. Three chunks of the block above land in that state; it parses perfectly the moment `=` arrives. A call the model genuinely wrote with a positional argument (rejected as of row 3) is in that state on *every* one of its chunks — 13 of them for `[read('/etc/hosts')]`. Pre-fix both raised, the streaming handler caught it and logged a full traceback: over a thousand in one production run, while not a single tool call was lost. `salvage_tool_calls` no longer decides policy — it returns what it managed to convert, and the caller decides what an empty result means: - complete block (non-streaming): raise and fall back to content, as before - mid-stream: return quietly and wait for the next chunk **Row 8** — the same shape of noise from a different failure, and older than #48171. A block whose brackets are structurally wrong cannot be completed at any prefix, and streaming re-parses from the start of the block on every chunk, so `make_valid_python` raises at the same offset until the block ends: ``` [foo(x=1])] 3 tracebacks [bash(cmd='ls -la'], timeout=30)] 6 [bash(cmd='find . -name \"*.py\" ...'], timeout=30, retries=3, ...) 25 ``` The count follows how much of the call arrives after the bad bracket, so a long argument list is the worst case, and none of the tracebacks named the text responsible. The failure is now handled where it happens: silence while the block is partial (its end has not arrived, so the requote recovery has not run yet), and one warning carrying the block once the end token is present. Which blocks are skipped does not change. The three sibling parsers report at a different moment: they have no reliable terminator (`pythonic` has none, and the tags the other two strip are optional), so they log the first text that failed — which ends at the bad bracket — once per parser instance, and a parser is constructed per request. **Row 9** — found while writing row 8's tests, and the only row that needs no malformed input. olmo3 calls are newline separated rather than bracketed, so its streaming path wraps the completed text in `[...]` itself: ``` get_weather <- name here, \"(\" not yet ^^^^^^^^^^^ wrapped into [get_weather]: parses, but the element is a Name, not a Call -> rejected ``` Every tool call passes through the state where part of its name has arrived and its `(` has not, so **every healthy olmo3 tool call logged a traceback per token of its name**: 1 for `f`, 2 for `get_weather`, 7 for `search_the_internet_for_recent_news`. The call itself was always emitted correctly, which is why this went unnoticed. Mid-stream the state is now a wait; the non-streaming path, where a bare name really is malformed output, still rejects it. Not duplicating existing work: no open PR touches `handle_single_tool` argument handling (checked via search); #43984 fixed the same non-finite class on the JSON side (`coerce_to_schema_type`) and is the precedent for rejecting non-finite arguments. ## Test Plan ```bash pytest tests/tool_parsers/ ``` Every fix has utils-level regression tests plus end-to-end cases through the lfm2 parser, and the three sibling parsers gain sibling-survival regressions. Each new test was verified to fail without its fix. ## Test Result **38 new regression cases**, each verified to fail without its fix: | File | Tests | |---|---| | `test_utils.py` | 195 | | `test_lfm2_tool_parser.py` | 57 | | `test_pythonic_tool_parser.py` | 24 | | `test_llama4_pythonic_tool_parser.py` | 26 | | `test_olmo3_tool_parser.py` | 25 | All pass, and the rest of `tests/tool_parsers/` is unaffected: the whole directory runs with **zero failures** (1013 passed; the 3 skips and 34 xfails are pre-existing markers in other parsers' suites, unchanged by this PR). pre-commit (ruff check/format, typos, mypy 3.10) and the pinned mypy 1.20.2 with the repo config: clean on all changed files. ## Notes for reviewers Block splitting is deliberately non-streaming: a partial streaming block is legitimately unparsable and must keep waiting for more text rather than be split into calls that are not finished yet. The recovery in rows 4-6 changes behavior rather than only fixing wrong output: a block that loses one of several calls can execute a dependent sequence partially (`cd` then `ls`). I judge that better than the block-level drop it replaces — the model gets an environment response and can correct — but it is a judgement call, not a pure bugfix, and I am happy to drop it if you disagree. The swallow guard in row 6 is not part of that judgement call: it stops a corrupted argument from being executed, and without the recovery it simply restores the pre-#48171 behavior of yielding no calls. Skipped calls are logged server-side with the offending AST, so the rate of malformed model output stays measurable rather than being silently absorbed. ---", "url": "https://github.com/vllm-project/vllm/pull/51649", "createdAt": "2026-08-10T09:32:18Z", "updatedAt": "2026-08-13T00:42:28Z", "timestamp": "2026-08-13T00:42:28Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "tool-calling", "llama" ], "author": "fatday", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:de100e68cfc2cd2bb552", "signalId": "github:vllm-project/vllm:pull_request:51346", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51346", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU] Add sequence parallelism support for DeepSeek V4", "text": "## Purpose Add sequence parallelism (SP) support to the XPU DeepSeek V4 model path. Attention activations are kept sharded along the sequence dimension across TP ranks, so each rank only materializes `num_tokens / tp_size` rows for the MoE and hyper-connection stages. This reduces activation memory and the volume moved by the post-attention collective on XPU. Implementation notes: - Reuses the existing `vllm.models.common.ops.sequence_parallel` helpers (`sp_shard` / `sp_all_gather` / `sp_reduce_scatter` / `sp_padding_mask`) rather than introducing an XPU-specific mechanism. - `_use_sequence_parallel()` gates SP on `pp == 1 and enable_expert_parallel and tp > 1 and dp > 1`, following the existing upstream convention for this family of models. - Inside the decoder layer, the attention input is all-gathered and the attention output is reduce-scattered; `wo_b.reduce_results` is disabled so the reduce-scatter replaces the all-reduce instead of duplicating it. - `is_sequence_parallel` is propagated to `FusedMoE` and the shared experts. - The `VLLM_MOE_SKIP_PADDING` `forward_context.is_padding` mask is sharded consistently with the activations. - Covers the main model, MTP, and DSpark paths.", "url": "https://github.com/vllm-project/vllm/pull/51346", "createdAt": "2026-08-07T02:31:41Z", "updatedAt": "2026-08-13T00:35:33Z", "timestamp": "2026-08-13T00:35:33Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "intel-gpu", "deepseek" ], "author": "majian4work", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:49016139a5870bc4371c", "signalId": "github:vllm-project/vllm:pull_request:52075", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52075", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Improve context-parallel backend error guidance", "text": "## Summary Improve context-parallel compatibility errors so they give users concrete recovery steps. - tell DCP users to select another backend with `--attention-backend`, or disable DCP with `--decode-context-parallel-size 1` - add equivalent actionable guidance to the PCP compatibility error - add regression tests for both messages ## Related issue Closes #28407 ## Testing - Python syntax checked for both changed files - Full pytest execution was not available in the local environment because PyTorch is not installed", "url": "https://github.com/vllm-project/vllm/pull/52075", "createdAt": "2026-08-13T00:33:24Z", "updatedAt": "2026-08-13T00:35:28Z", "timestamp": "2026-08-13T00:35:28Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "veerareddyvishal144", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:02058fca534afb3a0b51", "signalId": "github:vllm-project/vllm:pull_request:50592", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50592", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3][AMD] Return KDA and MLA projection outputs directly", "text": "## Purpose Kimi-K3's AMD attention paths copied each output projection into a caller-owned buffer even though the projection already returns a tensor with the required shape, dtype, layout, and lifetime. This PR removes that redundant post-projection allocation and copy from both attention families: - 69 KDA layers - 24 MLA layers - 93 post-attention-projection copies per model step in total The functionality from #50847 is now folded into this PR. #50654 is separate and complementary: it optimizes the earlier KDA recurrent/conv/RMSNorm path, while this PR removes the later copy after the output projection. ## Implementation - Return KDA output-projection storage directly from both the shared KDA implementation and the dedicated full-rank Kimi-K3 implementation selected by current `main`. - Return MLA output-projection storage directly. - Use one return-value contract in `KimiDecoderLayer` for KDA and MLA. - Preserve attention arithmetic, state updates, normalization, projection, tensor-parallel communication, and HIP-graph replay behavior. ## Validation ```text Targeted ownership tests: 3 passed Ruff 0.14.0 check: passed Ruff 0.14.0 format check: passed git diff --check: passed DCO: passed ``` The tests cover KDA dispatch ownership, the concrete full-rank Kimi-K3 KDA forward path, and MLA dispatch ownership. In each case, the returned tensor is the exact projection tensor and has the same `data_ptr()`. ## Fixed-cohort decode A/B ### Setup ```text GPU: 8 x AMD Instinct MI355X (gfx950) Parallelism: TP8 Model: Kimi-K3, BF16 activations / MXFP4 weights Decode: non-speculative Prefix caching: enabled Prompt: 63,911-token shared prefix + 4,089-token suffix Output: 1,024 tokens per request, ignore EOS Concurrency: C16 and C24 fixed cohorts HIP graph: FULL, capture sizes 1/16/24 Sampling: greedy, seed 42 Runs: 1 warmup + 3 measured runs per concurrency and variant Success rate: 240/240 measured requests ``` Both variants used image `localhost/kimi-pp8-upstream-clean` (vLLM `beca88e59`, AITER `99733dc00`). The baseline was the unmodified image. The candidate mounted only this PR's AMD runtime files from `43d81ba3e`. The relevant AMD Kimi-K3 files in the image are identical to the rebased `main` baseline before this PR. ### Results Means over three measured runs. TPOT and ITL are end-to-end serving metrics, not the reviewer's pure-GPU trace period. | Cohort | Metric | Baseline | Combined PR | Delta | |---|---|---:|---:|---:| | C16 | Mean TPOT | 73.141 ms | 72.428 ms | **-0.713 ms (-0.97%)** | | C16 | Median ITL | 70.488 ms | 70.186 ms | **-0.302 ms** | | C16 | Output throughput | 193.575 tok/s | 195.555 tok/s | **+1.02%** | | C24 | Mean TPOT | 74.958 ms | 74.457 ms | **-0.501 ms (-0.67%)** | | C24 | Median ITL | 71.311 ms | 70.916 ms | **-0.396 ms** | | C24 | Output throughput | 278.245 tok/s | 280.452 tok/s | **+0.79%** | Mean-TPOT standard deviation across the three runs was 0.184 ms / 0.348 ms for baseline C16/C24 and 0.019 ms / 0.122 ms for candidate C16/C24. ### Interpretation The reviewer's trace found that the complete 93-copy group costs approximately 0.495 ms at C16 and 0.563 ms at C24. The combined A/B removes all 93 copies and measures a 0.713 ms C16 and 0.501 ms C24 TPOT reduction. C24 closely matches the trace-derived removable cost. C16 is modestly larger, which is plausible because returning projection storage directly also changes allocator and lifetime behavior; the baseline's run-to-run variance is also higher. The absolute TPOT values should not be compared directly with the reviewer's 32.82/35.98 ms pure-GPU periods because TPOT includes serving and scheduling overhead. The A/B delta is the relevant same-harness comparison. ## Scope summary ```text #50592: remove 69 KDA + 24 MLA post-projection copies #50847: functionality absorbed by this PR #50654: separate earlier KDA recurrent-output/fusion optimization ```", "url": "https://github.com/vllm-project/vllm/pull/50592", "createdAt": "2026-07-31T16:26:50Z", "updatedAt": "2026-08-13T00:33:02Z", "timestamp": "2026-08-13T00:33:02Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "rocm", "kimi", "k3" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:da78766c19d3b4ec4a69", "signalId": "github:vllm-project/vllm:pull_request:51647", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51647", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Pad non-aligned AITER MLA heads", "text": "## Summary - pad non-16-aligned ROCm AITER MLA query heads to the next supported multiple of 16 and slice padding from the output - size dense and sparse persistent metadata for the padded launch shape - enable Kimi-K3 TP4's 24 heads/rank to use AITER MLA instead of falling back to Triton MLA ## Performance 8x MI355X, Kimi-K3 TP4/DP2/EP8, 100,000 input tokens (95,911 shared prefix + 4,089 suffix), OSL 1024, one API server, three runs per cohort. | Cohort | Triton MLA TPOT | AITER padded TPOT | TPOT change | Triton output tok/s | AITER output tok/s | Throughput change | |---|---:|---:|---:|---:|---:|---:| | C16 | 62.696 ms | 41.013 ms | -34.6% | 205.40 | 286.48 | +39.5% | | C24 | 76.061 ms | 46.816 ms | -38.5% | 256.32 | 373.99 | +45.9% | Source base: vLLM `61c1dd0966fd8d91954003cf6b7ab746448a46f8`, AITER `ca68b4f3501762c15c550cb920a5516e9710cf89`. ## Test plan - [x] targeted ROCm AITER MLA tests: 32 passed, 2 skipped - [x] Ruff check and format check on all changed files - [x] GSM8K-100: 99/100, 0 malformed outputs, 0 invalid numeric answers - [x] TP4/DP2/EP8 full-graph server startup and 100K C16/C24 decode benchmark Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/51647", "createdAt": "2026-08-10T09:20:08Z", "updatedAt": "2026-08-13T00:30:29Z", "timestamp": "2026-08-13T00:30:29Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "rocm", "verified" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bb27caeae4c5dbea4ba7", "signalId": "github:vllm-project/vllm:pull_request:51520", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51520", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RL] Add nccl-m2n sharding-aware weight transfer (worker side)", "text": "## Purpose Adds `nccl_m2n`, the inference-side half of a sharding-aware weight transfer backend built on [NCCL M2N](https://github.com/NVIDIA/nccl-extensions), per [RFC #46439](https://github.com/vllm-project/vllm/issues/46439). The broadcast NCCL backend assumes both sides hold the same layout, so a trainer that shards its parameters (FSDP, EP) must all-gather each one to a full tensor before sending. `nccl_m2n` moves each parameter with a single `reshard` that redistributes it between two disjoint meshes inside one communicator — trainer ranks `[0, T)`, inference workers `[T, T + N)`, which is exactly the interval vLLM's existing `rank_offset` convention already produces. The trainer sends its local shards and never materializes a full tensor. This PR is the **worker side only**. Each worker receives the whole tensor and hands it to `load_weights`, which is what the broadcast backend does for every parameter — so the two are directly comparable, and the destination layout is the same for every parameter, derived from the rank split with no negotiation. An external trainer can drive it today through the existing four-phase control plane (HTTP or Ray). Two follow-ups, already written and stacked behind this: 1. the in-tree trainer engine (`M2NWeightSource` / `DTensorModuleSource`, `M2NTrainerWeightTransferEngine`) plus a runnable FSDP → TP example; 2. per-parameter destination layouts, so a worker receives only its own shard. Selected with `--weight-transfer-config '{backend: nccl_m2n}'`. Default behavior is unchanged, and the runtime is imported lazily, so vLLM is unaffected unless the backend is selected. ## Not a duplicate Checked per `AGENTS.md`: ```bash gh pr list --repo vllm-project/vllm --state open --search \"46439 in:body\" # none gh pr list --repo vllm-project/vllm --state open --search \"m2n\" # none relevant gh pr list --repo vllm-project/vllm --state open --search \"weight transfer backend\" ``` No open PR references RFC #46439 or adds an m2n backend. The two adjacent ones solve a different problem: - **#40828 (WPI)** — a Kubernetes-native zero-copy transport for moving weights from external trainers. It is a transport substitution; it does not reshard between differing parallelism layouts, which is the entire point here. - **#43375 (RDT weight sync, draft)** — a WIP prototype for RDT sharded transfer. Overlapping in goal, but a different transport and still a prototype with no test plan. If it lands first, the two backends coexist in the factory registry the same way `nccl` / `ipc` / `sparse_nccl` do. ## Test Plan ```bash python -m pytest tests/distributed/test_weight_transfer_m2n.py -v ``` The suite covers how a transfer is *described*, not the transfer itself: layout encoding (including the size-1-axis spelling of full replication), the dtype and tensor-rank limits, init/update-info validation, and backend registration. Those are the failures worth catching cheaply — a bad layout or a ragged plan puts the two sides into mismatched collectives, which hangs rather than raises, so everything is validated during `init_transfer_engine` before any rendezvous. The transfer path needs the `nccl-extensions` runtime and multiple GPUs, so it is exercised out of tree against a real trainer. ## Test Result - `ruff check` / `ruff format --check` — clean on all touched files. - Layout invariants and the replicated-destination derivation verified against a stubbed `torch`, confirming both sides compute the same destination layout independently. - `pytest tests/distributed/test_weight_transfer_m2n.py` — **14/14 passed** (first run; this suite had never been executed before). Python 3.12.13, pytest 9.1.1, vLLM editable install on branch `nccl-m2n-worker` @ `1e7b332a6`. ## Model evaluation This PR has only the worker side change. We will need the trainer side change as well to make an end-to-end model run. The latter will be submitted in a stack PR. ## Known limitations Inherited from the m2n build and documented in `docs/training/weight_transfer/m2n.md`: - tensor rank 1..3; no fp4; `Partial` placements unsupported - the trainer's device mesh must cover a contiguous rank interval from 0, 1-D or 2-D - m2n caps per-destination-shard fan-in at 16 source shards (`MAX_SOURCES`, a compile-time bound in `reshard_limits.h`; raising it needs a library rebuild). This is not a rank limit in general — with a sharded destination each shard is fed by only a few sources. It binds here because a *replicated* destination is a single shard fed by every source shard, so a trainer that shards its parameters is limited to 16 ranks until the sharded-destination follow-up lands. A replicated trainer is unaffected at any scale. - requires `nccl-extensions` (NCCL ≥ 2.30.5) and `VLLM_NCCL_SO_PATH` pointing at the same `libnccl.so` that `libnccl_m2n.so` was linked against — torch bundles an older NCCL, so this is load-bearing rather than cosmetic. ## AI assistance This PR was written with AI assistance (Claude Opus 5 via Claude Code). Every line has been reviewed by the submitter, who understands and can defend the change end to end.", "url": "https://github.com/vllm-project/vllm/pull/51520", "createdAt": "2026-08-08T21:29:42Z", "updatedAt": "2026-08-13T00:17:40Z", "timestamp": "2026-08-13T00:17:40Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "needs-rebase" ], "author": "kwen2501", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:41a37e8bf5b03914bd19", "signalId": "github:vllm-project/vllm:pull_request:51938", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51938", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Register sparse DeepSeek V3.2 MTP", "text": "## Purpose Register the sparse DeepSeek V3.2 MTP implementation and select it for `glm_moe_dsa` speculative decoding configs. GLM MoE DSA previously inherited the generic `DeepSeekMTPModel` override even though its draft model needs the sparse DeepSeek V3.2 implementation. This PR: - maps the config to `DeepseekV32MTPModel`; - registers that architecture with the model registry; and - preserves its `(logits_hidden, feedback_hidden)` return contract in the legacy proposer. This PR is independent of the quantization optimization PRs split from #51936. ## Duplicate-work check No issue number was provided. I searched open PRs for `DeepseekV32MTPModel registry`; the only matching PR was the superseded draft #51936 from which this focused change was extracted. ## Tests ```bash .venv/bin/python -m pytest \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/v1/spec_decode/test_llm_base_proposer_sampling.py -q # 12 passed pre-commit run --files \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/models/registry.py \\ tests/v1/spec_decode/test_llm_base_proposer_sampling.py \\ vllm/config/speculative.py \\ vllm/model_executor/models/registry.py \\ vllm/v1/spec_decode/llm_base_proposer.py # all applicable hooks passed ``` ## AI assistance disclosure This change was developed with OpenAI Codex assistance. This is a draft PR; the human submitter must review every changed line and confirm they understand and can defend the change before marking it ready.", "url": "https://github.com/vllm-project/vllm/pull/51938", "createdAt": "2026-08-12T07:31:57Z", "updatedAt": "2026-08-13T00:13:05Z", "timestamp": "2026-08-13T00:13:05Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "new-model", "speculative-decoding", "needs-rebase", "deepseek" ], "author": "WoosukKwon", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:31c0c6322612332c6734", "signalId": "github:vllm-project/vllm:issue:49548", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:49548", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Dynamic speculative decoding (num_speculative_tokens_per_batch_size) causes catastrophic aggregate-throughput collapse under concurrency at the batch-size threshold (MTP, V1/PIECEWISE)", "text": "### Report of performance regression Enabling **dynamic speculative decoding** via `num_speculative_tokens_per_batch_size` produces two effects on a Qwen3.5-122B MTP (k=2) setup: 1. **Expected:** the documented `FULL_AND_PIECEWISE → PIECEWISE` cudagraph downgrade, which costs ~14% single-stream (59.6 → 50.0 tok/s). This is fine and matches the warning. 2. **Unexpected:** a **catastrophic aggregate-throughput collapse under concurrency** that looks like a pathological stall, *not* just \"no speculation at high batch.\" With the schedule `[[1,4,2],[5,512,0]]` (MTP k=2 for batch ≤ 4, spec **disabled** for batch ≥ 5), 8 concurrent 180-token requests dropped from **~232 tok/s aggregate** (static `num_speculative_tokens: 2`) to **24–157 tok/s**, with **wall-clock 40–60 s** for a workload that completes in **~6.3 s** under the static config — same model, same prompts, same token counts. Single-stream under the *same* dynamic config was a coherent 50 tok/s (the expected PIECEWISE level), so the collapse is specific to concurrency crossing the schedule's batch-size threshold. The naive expectation is that at batch ≥ 5 (K=0) throughput should fall back to roughly the **non-spec** decode rate. Instead it is ~1.5–10× *worse* than a plain non-spec run would be. That gap is what this issue is about. #### Environment - vLLM **0.23.1rc1.dev1302+ge765bbc97** (eugr/spark-vllm nightly) - **NVIDIA GB10 / DGX Spark**, sm_121a, unified memory (single device) - Model: **Qwen3.5-122B-A10B INT4 (AutoRound)**, native **MTP** speculator, `num_speculative_tokens: 2` - Attention backend: **FLASH_ATTN** - `enforce_eager=false`, cudagraph_mode requested `FULL_AND_PIECEWISE` - `VLLM_USE_V2_MODEL_RUNNER` **not** set (V1 runner) #### Speculative config Static (baseline, ~232 tok/s agg8, ~6.3 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2} ``` Dynamic (collapses to 24–157 tok/s, 40–60 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2, \"num_speculative_tokens_per_batch_size\": [[1, 4, 2], [5, 512, 0]]} ``` #### Numbers | Metric | static `k=2` | dynamic `[[1,4,2],[5,512,0]]` | |---|---:|---:| | Single-stream tok/s | 59.6 | 50.0 (expected PIECEWISE downgrade) | | Aggregate tok/s, 8 concurrent × 180 tok | ~232 | **24–157** | | Wall-clock, 8 concurrent | ~6.3 s | **40–60 s** | | Output token counts | identical | identical | #### Startup warning (as documented / expected) ``` Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` We have **not** yet tested `VLLM_USE_V2_MODEL_RUNNER=1` with the dynamic schedule (there is a separate open report, #48494, that V2 + `num_speculative_tokens_per_batch_size` + MTP crashes at speculator decode capture, so we held off). ### Misc discussion on performance — code-reading hypotheses We can't attach a profiler run yet (the box is a shared production node), so we read `main` (`f83de6d`) to make this actionable. Candidate mechanisms, strongest first: **H1 — At K=0 the MTP drafter still runs a full draft-model forward every step.** `SpecDecodeBaseProposer.propose` short-circuits on K=0 only **after** the first-pass draft forward has already executed: - `vllm/v1/spec_decode/llm_base_proposer.py:580-590` runs `self.model(**model_kwargs)` (the MTP-layer forward), - then `vllm/v1/spec_decode/llm_base_proposer.py:606-618` returns an empty tensor with the explicit comment *\"The prefill forward pass above already ran to keep the drafter KV cache in sync.\"* The runner also has **no K==0 guard** — the drafter is invoked whenever `spec_config is not None` and `input_fits_in_drafter` (`vllm/v1/worker/gpu_model_runner.py:4570-4601` and the MTP branch at `:5096-5212`, which calls `drafter.propose(num_speculative_tokens=0, ...)`). So \"spec disabled at batch ≥ 5\" is **not** equivalent to a non-spec run: every decode step at high batch still pays a full MTP forward plus all input prep (`set_inputs_first_pass`, attention-metadata build) for zero drafts. Combined with the per-step spec-decode overhead already noted in #47277, this alone can erase — and invert — the expected fallback throughput. **H2 — Dynamic SD disables the uniform-spec padding that preserves full cudagraphs, forcing PIECEWISE + variable per-step shapes.** The decode-padding path that pads to uniform spec size is explicitly gated off whenever a dynamic schedule is present: `vllm/v1/core/sched/scheduler.py:822-838` (`(self.num_spec_tokens > 0 and self.dynamic_sd_lookup is None)`). This is the documented `FULL_AND_PIECEWISE → PIECEWISE` downgrade (`vllm/config/vllm.py:819-836`). Under concurrency, per-step query lengths vary as K flips at the batch-4↔5 boundary and as requests enter/leave, so the target forward (`gpu_model_runner._determine_batch_execution_and_padding`, `vllm/v1/worker/gpu_model_runner.py:3876`) and the drafter forward (`vllm/v1/spec_decode/llm_base_proposer.py:1780`) see a churn of shapes, each dispatched/padded separately with attention metadata rebuilt every step. The dispatcher pads up rather than recapturing, so this isn't naive recapture — but PIECEWISE replay + eager glue over a constantly-shifting shape set, at batch 8, on a 273 GB/s unified-memory device, compounds. **H3 — K is keyed off the raw scheduled-request count, so the threshold thrashes.** `vllm/v1/core/sched/scheduler.py:1116-1120` selects `num_spec_tokens_to_schedule = self.dynamic_sd_lookup[len(num_scheduled_tokens)]`. `len(num_scheduled_tokens)` counts **all** scheduled requests including prefills/chunks, so a single prefill joining 4 decodes bumps the count to 5 and silently disables spec for the whole step. As the running set oscillates around the threshold, K flips 2↔0 step-to-step, changing the scheduled verify-token count and therefore both forward shapes, defeating cudagraph reuse and forcing repeated pad/dispatch/metadata rebuilds. This interacts badly with H2. **H4 — KV lookahead is over-reserved at K=0.** `num_lookahead_tokens` is fixed to the static `num_spec_tokens` at init (`vllm/v1/core/sched/scheduler.py:234, 246-258`) and passed to `allocate_slots` as `effective_lookahead_tokens` (`:886-889, 912-925`) — it is **not** lowered to the dynamic K. So at batch 8 with K=0 the scheduler still reserves `num_spec_tokens` (=2) lookahead slots per request every step. Reserved lookahead slots are normally transient, but they still count against the free-block admission check, so under KV pressure this could trigger earlier can't-schedule/preemption events than a true non-spec run — which would present exactly as a throughput stall. (Lowest-confidence; flagging for completeness.) H1 and H2 are the most defensible and, together, would explain why K=0-at-high-batch is materially worse than plain non-spec rather than equal to it. ### Offer We have a **GB10 / DGX Spark (sm_121a)** available and are happy to run follow-up diagnostics if a maintainer wants them — e.g. a torch profiler / nsys trace of the 8-concurrent dynamic run, `vllm:spec_decode_*` counters at batch ≥ 5, per-step cudagraph dispatch mode logging, or an A/B with the H1 wasted-forward path short-circuited. Just say which knobs/logging you'd like enabled. ### Your current environment ```text vLLM 0.23.1rc1.dev1302+ge765bbc97 (eugr/spark-vllm nightly) NVIDIA GB10 / DGX Spark, sm_121a, 128 GB unified memory (~273 GB/s) Model: Qwen3.5-122B-A10B INT4 AutoRound, native MTP, num_speculative_tokens=2 Attention backend: FLASH_ATTN (collect_env.py output can be attached on request from the box) ``` ### Before submitting a new issue... - [x] Searched existing/past issues. Closest are #47277 (native MTP per-step overhead, static K, single-stream), #42505 (DFlash slower at concurrency > 8, static K), #49369 (DSpark aggregate halved, static K), and #48494 (`num_speculative_tokens_per_batch_size` + MTP crashes full-cudagraph capture under V2). None cover this dynamic-schedule **concurrency-stall** signature (single-stream fine, aggregate collapses at the batch threshold under V1/PIECEWISE).", "url": "https://github.com/vllm-project/vllm/issues/49548", "createdAt": "2026-07-23T07:25:02Z", "updatedAt": "2026-08-13T00:08:33Z", "timestamp": "2026-08-13T00:08:33Z", "metrics": { "reactions": 0, "comments": 7 }, "labels": [], "author": "tobby168", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c785d6084e7b34490ebb", "signalId": "github:vllm-project/vllm:pull_request:51714", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51714", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][DSV4] Opt-in AITER gluon kernel for sparse-MLA decode on gfx950", "text": "## Purpose Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair (`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). ## Changes Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): - `VLLM_ROCM_DSV4_SPARSE_GLUON` (**default off**). Read once at import — this sits on the per-step decode path, so re-reading the environment per call would be pure overhead. With the flag unset the decode path is byte-for-byte unchanged. - The gluon entrypoint is resolved lazily behind `functools.cache`, so builds whose AITER lacks the kernel are unaffected. - A failed launch latches a process-wide flag and degrades to Triton for the rest of the process, rather than taking the server down mid-request. - The first successful dispatch logs once, so enabling the flag is observable rather than silent. - Additionally gated on `_ON_GFX950`. ## Requires ROCm/aiter#4673 The kernel needs [ROCm/aiter#4673](https://github.com/ROCm/aiter/pull/4673) to be correct on large KV pools. Without it the gluon driver decides `use_buffer_load` from `nelement() * element_size()`, which undercounts a strided pool whose addressable span exceeds a 32-bit offset; past 2 GiB the `buffer_load` offsets wrap and the gather silently returns zeros. An earlier revision of this PR refused to run under DP-attention, citing [ROCm/aiter#4667](https://github.com/ROCm/aiter/issues/4667) (gsm8k 1.00 → 0.30 on EP8 + DP-attention). That guard has been removed: #4667 is the overflow above, and DP-attention was only ever the thing that grew the pool far enough to cross the boundary — not a property of the kernel. Gating on `data_parallel_size` therefore disabled the kernel on exactly the configuration it was written for. With #4673 applied, DP-attention runs clean. ## Test Plan - DeepSeek-V4 on MI355X (gfx950) with `VLLM_ROCM_DSV4_SPARSE_GLUON=0` and `=1`, comparing decode outputs for parity and measuring decode throughput. - Cover both `swa_only` (single-segment) and two-loop layers. - Confirm the fallback latch by forcing a launch failure. - Cover DP-attention (DEP8) with aiter#4673 applied. ## Test Result | Concurrency | Output tok/s (base → +PR) | Δ | Median TPOT (base → +PR) | Δ | |---|---|---|---|---| | 8 | 284.79 → 291.33 | **+2.30%** | 24.67 → 24.16 ms | −2.04% | | 16 | 483.69 → 488.59 | **+1.01%** | 29.32 → 29.02 ms | −1.03% | | 32 | 708.51 → 712.89 | **+0.62%** | 40.78 → 40.52 ms | −0.65% | | 64 | 955.23 → 958.04 | **+0.29%** | 62.55 → 62.36 ms | −0.30% | ### AgentX agentic replay (fast mode, TP=8, concurrency 32, MTP on) | Metric | base (Triton) | +PR (gluon) | Δ | |---|---|---|---| | Output tok/s | 430.76 | 453.23 | **+5.22%** | | Output tok/s/GPU | 53.84 | 56.65 | **+5.22%** | | Per-user tok/s | 32.72 | 39.96 | **+22.12%** | | ITL avg | 39.83 ms | 34.60 ms | **−13.11%** | | ITL p50 | 35.87 ms | 26.44 ms | **−26.30%** | | Time-to-2nd-token p50 | 38.19 ms | 21.79 ms | **−42.95%** | | TTFT p50 | 656.7 ms | 700.6 ms | +6.68% (regression) | ### DP-attention, with aiter#4673 applied DEP8 (DP-attention + EP8) + MegaMoE + gluon, MI355X, agentic replay at concurrency 96: **12,244 tok/s/GPU**. All eight DP ranks log the gluon path engaging. This is the configuration the removed guard used to refuse.", "url": "https://github.com/vllm-project/vllm/pull/51714", "createdAt": "2026-08-10T18:00:26Z", "updatedAt": "2026-08-13T00:05:11Z", "timestamp": "2026-08-13T00:05:11Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "jiacao-amd", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:5fbc9aa55935f0384540", "signalId": "github:vllm-project/vllm:issue:51993", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51993", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: security: bump minimum setuptools version in requirements/common.txt", "text": "### Your current environment The output of python collect_env.py ```text OS: Linux (Termux / Android) Python Version: 3.11+ vLLM Repository: main branch audit ### 🐛 Describe the bug A security audit on `requirements/common.txt` identified known security advisories affecting the pinned version of `setuptools`: - **setuptools (`77.0.3` → Fix: `>=83.0.0`):** Affected by `PYSEC-2025-49` and `PYSEC-2026-3447`. ### Proposed Fix Update the lower bound in `requirements/common.txt`: ```text setuptools>=83.0.0 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51993", "createdAt": "2026-08-12T13:34:01Z", "updatedAt": "2026-08-13T00:00:45Z", "timestamp": "2026-08-13T00:00:45Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug" ], "author": "ev227166-spec", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:68ce0ff048a956f0dde1", "signalId": "github:vllm-project/vllm:pull_request:52072", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52072", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Apply suppress_tokens on the Gemma 4 MTP sparse path", "text": "## Purpose When `use_ordered_embeddings` is set on a Gemma 4 MTP drafter, `Gemma4Proposer._greedy_sample` samples through `get_top_tokens` rather than `compute_logits`: ```python # vllm/v1/spec_decode/gemma4.py:113 return self.model.get_top_tokens(hidden_states) ``` That reaches `Gemma4MTPMaskedEmbedder.get_top_tokens`, which takes a sparse argmax directly over the centroid-selected candidates: ```python logits, indices = self._select_and_score(hidden_states, lm_head_weight) return indices.gather(-1, logits.argmax(-1, keepdim=True)).squeeze(-1) ``` `compute_logits` — the only place `suppress_tokens` is applied — is never called on this path, so **the drafter can propose tokens the generation config marks as suppressed**. The same applies to the CUDA-graph-captured variant, since `_setup_centroids_cuda_graphs` captures `masked_emb.get_top_tokens` directly. This is a correctness gap rather than a crash, so it is silent: the suppressed token is proposed by the drafter and then either accepted or rejected by the verifier depending on the target model's own distribution. ## Approach Give `Gemma4MTPMaskedEmbedder` a boolean mask over the vocabulary, populated from the draft model's generation config in `Gemma4MTP.__init__`, and apply it to the candidate logits before the argmax. The mask lives on the embedder rather than on `Gemma4MTP` because there are two call sites — `Gemma4MTP.get_top_tokens` and the proposer's direct `masked_emb.get_top_tokens` during graph capture — and only module-level state covers both. It is registered with `persistent=False`: it is derived from the generation config, not a checkpoint weight, and a persistent buffer would surface as an unexpected key during weight loading. The dense path is untouched — `forward` still returns unsuppressed full-vocab logits and `compute_logits` masks them, which the test asserts explicitly. ## Test plan Adds `tests/model_executor/test_gemma4_mtp_sparse_suppress.py` (CPU-only, `@pytest.mark.cpu_test`). `Gemma4MTPMaskedEmbedder` has no `VllmConfig` dependency, so these construct it for real with a deterministic centroid layout: an identity `token_ordering`, fixed centroid weights so the selected candidate set is always tokens 0–7, and an `lm_head_weight` giving an unambiguous ranking (token 5 > 2 > 6). - suppressing the best candidate returns the second-best, and suppressing both returns the third; - `[]` and `None` are no-ops; - `suppress_mask` stays out of `state_dict` while `token_ordering` remains in it; - suppression holds across a batched `hidden_states`; - `forward()` output is byte-identical with and without suppression configured. ## Known degenerate case, deliberately not matched If *every* centroid-selected candidate is suppressed, the sparse argmax returns an arbitrary suppressed candidate, whereas the dense path returns the lowest-id non-selected token — non-selected positions are filled with `finfo.min`, which compares above `-inf`. Reaching this requires `suppress_tokens` to cover every token in the selected clusters. I left the behavior rather than encoding a somewhat arbitrary dense-path artifact as a contract, but happy to match it if reviewers prefer. ## Relationship to existing work #48693 by @dumko2001 also touches sparse-path parity as part of a broader Gemma 4 refactor; it has been in draft with merge conflicts since July. None of its code is used here. This is deliberately scoped to the sparse suppression gap alone so it can be reviewed independently. Depends on nothing, but is adjacent to the separate CUDA-graph fix for `compute_logits` on the dense path (#48503). ## Validation caveat No GPU access here, so this has not been exercised on a real Gemma 4 MTP run, and the test file has not been executed against a real vLLM install. What was verified: the assertions above all pass against the actual patched `Gemma4MTPMaskedEmbedder` class source executed standalone against real torch, and `ruff check` / `ruff format` (pinned v0.14.0) are clean. Hardware confirmation would be welcome. ## AI assistance disclosure Parts of this change and its test were drafted with AI assistance (Claude). I have reviewed every changed line and validated the behavior described above.", "url": "https://github.com/vllm-project/vllm/pull/52072", "createdAt": "2026-08-12T23:54:08Z", "updatedAt": "2026-08-12T23:54:44Z", "timestamp": "2026-08-12T23:54:44Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "aminsamir45", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8b2ddf2eedb6528f310d", "signalId": "github:vllm-project/vllm:pull_request:52070", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52070", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][Spec Decode] Disable EAGLE cache-hit block drop for always-K=0 dynamic DSD", "text": "## Purpose Opt-in skip of EAGLE-style last-block drop on prefix-cache hits when dynamic speculative decoding is **always K=0**. Default is off, so stock cache-hit behavior and observed greedy tokens are unchanged. Always-K=0 DSD (`num_speculative_tokens_per_batch_size=[[1,16,0]]`) produces no draft tokens, but `use_eagle=True` still drops the last matched KV block on a prefix-cache hit. That inflates scheduled-token volume on a long-context warm workload (stock: 4 steps / ~24k tokens per burst vs nospec ~1 step). This PR keeps `use_eagle=True` and only disables that drop when the operator sets `disable_eagle_cache_drop_for_k0=True` **and** all of: prefix caching on, no KV connector, `use_eagle()` (EAGLE / EAGLE3 / MTP / DFlash / DSpark), dense lookup `max(lookup[1:])==0`. Related: #49548 (always-K=0 slice only; not a universal fix). This is a **performance tradeoff**, not a correctness fix. Greedy outputs can change when the flag is on (measured divergence at generated pos 22 multi-request / pos 67 single-request vs stock K=0). ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/v1/core/test_eagle_cache_drop_k0.py \\ tests/v1/core/test_mamba_align_chunk_split.py \\ tests/v1/core/test_prefix_caching.py::test_eagle_enabled_removes_last_block \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_partial_blocks \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_sliding_window \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_combinations_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_swa_alignment_caches_extra_block \\ tests/v1/spec_decode/test_dynamic_sd.py -q ``` Enable (example): ```python speculative_config={ \"method\": \"mtp\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [[1, 16, 0]], \"disable_eagle_cache_drop_for_k0\": True, } # plus enable_prefix_caching=True; no KV connector ``` ## Test Result CPU unit tests above: **64 passed** (2026-08-12, local). ThinkingCap-Qwen3.6-27B-FP8, dual RTX 5090 TP=2, always-K=0 `[[1,16,0]]`, ~49k prompt with prefix-cache hit, 8 sessions × 2 rounds × 128 tokens, inproc. GPU validation used commit `da159d447d` overlaid into the production venv. The only extra local change was an uncommitted scheduler JSONL helper for the burst census; it does not change scheduling decisions. **Primary comparison (this flag, one outing):** C0 **82.6 t/s** (n=1) vs B2 **136.7 t/s** (n=3, stdev 0.3), about **1.65×**. Mixed C3 `[[1,2,2],[3,16,0]]` does not enable the flag. **Burst census (same outing):** | Config | Warm steps/burst | Warm Σ tok/burst | |---|---:|---:| | Stock always-K=0 (C0) | 4 | 24,238 | | This change (B2, flag on) | 2 | 11,429 | | Coarse `use_eagle=False` (B1, earlier run) | 2 | 11,429 | B2 matches B1’s 4→2 / 24k→11k shape. `use_eagle` stays True. **Earlier local prototype (not this flag; do not mix with the numbers above):** same workload family, util 0.90, one-rep: C0 89.4 t/s, local B2 (`VLLM_SCHED_WARM_BATCH`) 147.0 t/s, B1 147.7 t/s. Nearby nospec ~178 t/s. Residual vs nospec is not attributed. Greedy identity vs stock K=0 with the flag on: **FAIL** at pos 22 (multi) / pos 67 (single), 1 rep each. **Small accuracy probe** (not GSM8K): 8 grade-school integer problems, greedy, ~2k-token shared stem cached first so the last-block drop actually runs. C0 **7/8**, B2 **8/8**. The seven unambiguous items matched. The eighth is an ambiguous wording trap (capacity 80 vs 80−15=65); C0 answered 65, B2 answered 80. Do **not** read 8/8 vs 7/8 as a quality win. Cliff check only: the flag did not break the clear math items. No named model eval (GSM8K / `tests/evals/`) was run. ## Why this is not a duplicate No open PR implements an always-K=0 opt-in skip of `drop_eagle_on_cache_hit`. **#50897** is the general correctness approach (successor-aware hashes so the last block can be kept without changing greedy tokens). It is **not** a substitute we can cite for this workload today: - 43 files; **merge-conflicting** as of 2026-08-11. Collaborator review asked to **split** it (too pervasive; still keeps the old drop path, so core complexity goes up). A rebase alone does not address that. No LGTM. - We have **not** measured #50897 on the always-K=0 ~49k warm-burst bench. Same speed as this flag is a hypothesis. This PR is a small, default-off option for that measured always-K=0 volume defect. If successor-aware hashing lands and removes the defect, **deprecate and remove this flag**. The two do not conflict today (off by default; no hash-format change). Not discussed on #50897. | Related | Difference | |---|---| | #50897 | Correctness-preserving hash redesign (see above). Not merge-ready as-is; unmeasured here. | | #51769 | Warns when EAGLE-style methods cost a large prefix-cache hit. Diagnostic only. | | #51295 | Hybrid attention miss/corruption from eagle drop. Different bug. | | #48375 / #45614 | Mamba eagle-drop plumbing / hit accuracy. Not DSD K=0 scheduling volume. | | #51466 / #51575 | Already-open DSD bugfixes. Not bundled. | KV-connector / Mooncake Store eagle-drop paths are **out of scope** (gate requires `connector is None`). ## AI disclosure AI assistance was used to draft the implementation and tests. A human (Greg) is accountable for the PR and will review every changed line before merge. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52070", "createdAt": "2026-08-12T23:52:41Z", "updatedAt": "2026-08-12T23:53:11Z", "timestamp": "2026-08-12T23:53:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "Suppressor72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8fb0b0e9a35d0f1bfa1f", "signalId": "github:vllm-project/vllm:pull_request:52047", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52047", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][AMD] Annotate draft KV cache groups on the hybrid grouping path", "text": "## Purpose Serving **Kimi-K3** (`KimiLinearForCausalLM` — 93 layers: 24 full-attention MLA + 69 KDA linear-attention) with the **DSpark MTP draft** (`Inferact/Kimi-K3-DSpark`) and a CPU KV offload tier, the external prefix cache never serves anything: `vllm:external_prefix_cache_hits_total` stays pinned at exactly 0 for the life of the server while `vllm:kv_offload_store_bytes_total` grows without bound. The offload tier writes continuously and reads back nothing. The same latent defect affects any hybrid (attention + Mamba) model with an EAGLE-family drafter that reaches the general multi-group KV-cache grouping path. ## Root cause `is_eagle_group` is set in exactly one place in the tree: `_annotate_eagle_groups_deepseek_v4` (`vllm/v1/core/kv_cache_utils.py`). It is gated on `model_version == \"deepseek_v4\"` and is reachable only from the `group_and_unify_kv_cache_specs` branch of `get_kv_cache_groups`. Kimi-K3 is MLA + Mamba, so it lands on the general multi-group path and **no group is annotated at all**. Consumers respond to \"no group annotated\" by flagging *every* group as a draft group — both `HybridKVCacheCoordinator.find_longest_cache_hit` and the `OffloadingConnector` scheduler (`offloading/scheduler.py`, `eagle_groups = set(range(len(kv_cache_groups)))`). That fallback is documented as conservative, and it is — for full-attention groups. For a **Mamba** group it is not: 1. `get_sliding_window_size_in_chunks` returns `1` for a `MambaSpec` — \"Mamba depends on a single state\". 2. The eagle flag then applies `required_window += 1` → **2**. 3. `_sliding_window_lookup` reports a hit only on `consecutive_hits == sliding_window_size`, i.e. **two adjacent stored chunks**. 4. Under `mamba_cache_mode=\"align\"`, exactly one state checkpoint is materialized per scheduler step; the chunks in between are null blocks (`block_id == 0`) and are skipped on store. 5. Two adjacent mamba chunks are therefore **never both stored**, so the group can never report a hit. 6. `_lookup` takes the minimum across groups, so `num_hit_chunks == 0` zeroes the whole request. ## What this means concretely for Kimi-K3 + DSpark The Mamba page forces a **1536-token block**, and the draft's 5 MLA layers are the smallest spec bucket, so `group_size` collapses to 5 — giving **20 KV cache groups: 5 target-MLA, 14 Mamba, 1 draft**. (Consistent with the two page-padding warnings every DSpark run logs: `4.17% = 1/24` and `1.45% = 1/69`.) At `--max-num-batched-tokens 32768` a KDA checkpoint lands every **21 chunks = 32,256 tokens**. | | Groups flagged | Mamba group lookup | Result | |---|---|---|---| | **before** | 20 / 20 | needs 2 adjacent chunks at a 21-chunk stride | unsatisfiable → `get_num_new_matched_tokens` returns 0 for every prompt-shaped request | | **after** | 1 / 20 (the draft) | window stays 1, satisfied by a single checkpoint | lookup is no longer structurally zeroed; the 5 target-MLA groups also stop taking the eagle last-chunk drop | The failure is silent today. A vetoed lookup is not counted as a *query* either, so the hit **rate** stays empty rather than visibly dropping — there is no counter an operator can watch go wrong. Hence the second half of this patch. ## The fix Annotate **by spec rather than by model version**. `non_causal_multi_token_decode` is set exactly on drafter attention layers (`vllm/models/kimi_k3/nvidia/dspark_mla.py`), is declared on `MLAAttentionSpec` as \"Marks draft groups that flatten a non-causal query block into decode rows\", and is propagated through `MLAAttentionSpec.merge` via `any(...)`. So it identifies the draft group wherever grouping happens to land. Two functions, both called at the end of the general multi-group path: - `_annotate_eagle_groups_from_draft_spec` — flags groups whose merged spec carries the marker. - `_warn_if_unannotated_eagle_mamba` — warns when speculative decoding is on, no group could be identified, and Mamba groups are present. That combination still takes the flag-all path and should not fail silently. ## Why this is not a duplicate | PR | What it does | Why it does not cover this | |---|---|---| | **#47891** — Exclude DSpark draft KV-cache group from OffloadingConnector lookup poisoning | Adds `eagle_group_is_veto_exempt` so an *identified* draft group's miss does not veto the request | The flag is only ever set inside `_annotate_eagle_groups_deepseek_v4`. On an unannotated model the exempt set is empty and nothing changes. The PR says so explicitly: *\"Left empty in the 'flag all groups' fallback below … we conservatively keep the original behavior.\"* Kimi-K3 is precisely that case. | | **#48459** — core-tier half of #47891 | Same mechanism on `HybridKVCacheCoordinator` | Same single DSV4-gated setter; same gap. | | **#48375** — Honor `drop_eagle_block` in `MambaManager` | Fixes silent recurrent-state corruption on the GPU tier | Different bug. Does not touch group annotation or the offload lookup window. | | **#41640** — Add `is_eagle` in `AttentionSpec` | Generic annotation infrastructure that would attack the same root | Open and untouched since May 2026. If it lands, this patch should be rebased onto it. | No open PR mentions `required_window` or `_sliding_window_lookup` at all. **This patch is the missing precondition for #47891/#48459 rather than a competitor.** It stops the flag-all fallback from firing; they stop the identified draft group from vetoing. They compose, and on a hybrid model neither is sufficient alone. ## Test plan E2E KK3 test Four tests added to `tests/v1/core/test_kv_cache_utils.py`: - `test_draft_group_annotated_on_hybrid_general_path` — on an MLA + Mamba + draft-MLA fixture, exactly one group is flagged, it is the draft's, and no Mamba group is flagged. - `test_draft_group_not_annotated_without_spec_decode` — the marker alone flags nothing. - `test_unidentifiable_draft_with_mamba_warns` — the warning fires when no draft can be identified. - `test_no_warning_when_draft_group_is_identified` — and stays quiet when one can. ## Test result Unit tests: ``` tests/v1/core/test_kv_cache_utils.py::test_mla_draft_prefers_standard_layout_when_pages_can_be_unified PASSED [ 16%] tests/v1/core/test_kv_cache_utils.py::test_draft_group_annotated_on_hybrid_general_path PASSED [ 33%] tests/v1/core/test_kv_cache_utils.py::test_mamba_groups_never_flagged_even_when_draft_shares_a_group PASSED [ 50%] tests/v1/core/test_kv_cache_utils.py::test_draft_group_not_annotated_without_spec_decode PASSED [ 66%] tests/v1/core/test_kv_cache_utils.py::test_unidentifiable_draft_with_mamba_warns PASSED [ 83%] tests/v1/core/test_kv_cache_utils.py::test_no_warning_when_draft_group_is_identified PASSED [100%] ``` E2E Kimik3: on E2E I've measured the following improvements with the PR Config: ISL 8192 / OSL 1024 / conc 16, 24 sessions, TP8, DSpark, 64 GiB offload pool, GPU KV capped to 163,840 tokens so the 221 k working set evicts. \"image\" ## Accuracy / serving eval ``` ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52047", "createdAt": "2026-08-12T21:08:09Z", "updatedAt": "2026-08-12T23:52:12Z", "timestamp": "2026-08-12T23:52:12Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm" ], "author": "okorzh-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0352d38df4880687efce", "signalId": "github:vllm-project/vllm:pull_request:49790", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49790", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][NVIDIA] Route DSA models to the SM100 implementation", "text": "## What this does Point the DSA architectures at the SM100-optimized implementation in `vllm/models/deepseek_v32/`, which nothing on `main` currently references. - `GlmMoeDsaForCausalLM` now resolves to `vllm.models.deepseek_v32` instead of the generic `deepseek_v2`, and `DeepseekV32MTPModel` is registered so speculative decoding uses the matching MTP module. - `deepseek_v32/__init__.py` gains a platform gate: SM100 gets the optimized module, everything else (ROCm, XPU, pre-SM100 CUDA, CPU) falls back to the generic `deepseek_v2`. Previously it imported the SM100 module on any CUDA device, which is what broke the H100 CI job on #48597. - DeepSeek V3.2 is unaffected: `DeepseekV32ForCausalLM` still maps to `deepseek_v2`, and the MTP rewrite is gated on `model_type == \"glm_moe_dsa\"`. This is the enabler for the rest of the series — until it lands, the other PRs in the split are runtime no-ops. Part of the #48597 re-split — see that PR for the tracker and merge order. ## Benchmark 8×B300 TP8, `nvidia/GLM-5.2-NVFP4`, MTP=5, `VLLM_USE_V2_MODEL_RUNNER=1`, fp8 KV cache, 8192 input / 1024 output tokens, concurrency 1, 64 prompts. ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve $MODEL --served-model-name glm-5.2 \\ -tp 8 --port 8300 \\ --kv-cache-dtype fp8_e4m3 --max-model-len 16384 \\ --max-num-seqs 256 --max-num-batched-tokens 16384 \\ --no-enable-prefix-caching --gpu-memory-utilization 0.85 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":5}' \\ --kernel-config '{\"ir_op_priority\":{\"rms_norm\":[\"vllm_c\",\"native\"],\"fused_add_rms_norm\":[\"vllm_c\",\"native\"]},\"enable_flashinfer_autotune\":false}' ``` ### Performance vs main `main` @ `48a077e4cf` versus the same commit with all six branches of the split applied, each built and measured back-to-back on one node, 32 prompts at concurrency 1, `--kv-cache-dtype fp8_e4m3` throughout. | weights | build | output tok/s | total tok/s | median TPOT | GSM8K | | --- | --- | --- | --- | --- | --- | | NVFP4 | `main` | 447.4 | 4026.5 | 1.93 ms | 0.940 | | NVFP4 | full series | **541.9** | 4876.7 | **1.56 ms** | 0.948 | | FP8 | `main` | 365.8 | 3291.8 | 2.39 ms | 0.956 | | FP8 | full series | **403.1** | 3627.7 | **2.15 ms** | 0.946 | That is **+21.1%** output throughput on NVFP4 (TPOT -19.2%) and **+10.2%** on FP8 (TPOT -10.0%). The GSM8K spread is roughly one standard error at 500 questions (SE ~= 0.01), so it shows no accuracy regression in either direction rather than a real difference. AI assistance (Claude) was used for the split and the benchmarks; every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49790", "createdAt": "2026-07-25T08:16:16Z", "updatedAt": "2026-08-12T23:40:19Z", "timestamp": "2026-08-12T23:40:19Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "new-model", "speculative-decoding", "needs-rebase", "v1", "deepseek", "nvidia" ], "author": "zhou9402", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e4b5cf542d08db284a68", "signalId": "github:vllm-project/vllm:pull_request:49925", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49925", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Switch to the Rock, Keep Python 3.12 and Ubuntu 22.04", "text": "## Purpose This PR switches to using The Rock 7.14 with wheels provided by The Rock while keeping Python 3.12 and Ubuntu 22.04 as-is. Need to wait for @Rohan138 to bump to torch 2.12, then will need to rerun testing. We could possibly add` /ready` to this and retry as needed when it becomes convenient. ## Test Plan Full CI runs. ## Test Result Currently failing groups that need to be addressed are: ``` MI300: Basic Models Tests (Other) Transformers Nightly Models (Single) MI355: Entrypoints Integration (Pooling) Language Models Tests (Standard) Multi-Modal Models (Extended Generation 3) ``` Keeping track in this BK build: https://buildkite.com/vllm/amd-ci/builds/11580/list ---
Essential Elements of an Effective PR Description Checklist - [X ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X ] The test plan, such as providing test command. - [ X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49925", "createdAt": "2026-07-27T02:32:21Z", "updatedAt": "2026-08-12T23:33:54Z", "timestamp": "2026-08-12T23:33:54Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "rocm", "ci/build" ], "author": "rasmith", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:010011d10001ae42298d", "signalId": "github:vllm-project/vllm:pull_request:52068", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52068", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Offload] Preserve KV event metadata until final residency removal", "text": "## Purpose Track offloaded-block residencies by medium and ownership so removing one residency does not discard metadata still needed by another. Depends on PR #52067, no conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py", "url": "https://github.com/vllm-project/vllm/pull/52068", "createdAt": "2026-08-12T23:26:46Z", "updatedAt": "2026-08-12T23:27:25Z", "timestamp": "2026-08-12T23:27:25Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kv-connector" ], "author": "mkhazraee", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:77dbe65bda64110d9a9f", "signalId": "github:vllm-project/vllm:pull_request:51302", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51302", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] deepseek-vl2: restore original DeepseekV2Config defaults for omitted language_config fields", "text": "## Purpose Loading `deepseek-ai/deepseek-vl2-small` (and `deepseek-ai/deepseek-vl2`) currently crashes during model initialization: ``` TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' ``` at `DeepseekV2Attention.__init__` (`self.kv_lora_rank + self.qk_rope_head_dim`). Working around that surfaces a second failure while loading `embed_tokens`: ``` AssertionError # loaded_weight.shape[output_dim] == self.org_vocab_size ``` Both failures share one root cause. DeepSeek-VL2 checkpoints ship an incomplete `language_config` that relies on the defaults of the *original* DeepSeek-VL2 `DeepseekV2Config` (`vocab_size=102400`, `kv_lora_rank=512`, `qk_nope_head_dim=128`, `qk_rope_head_dim=64`, `v_head_dim=128`). For example, `deepseek-vl2-small`'s `language_config` contains neither `vocab_size` nor `kv_lora_rank`. vLLM does not use the model's remote config; `DeepseekVLV2Config` parses `language_config` with the built-in Transformers `DeepseekV2Config` (`DeepseekVLV2TextConfig`), whose generic defaults differ: - `DeepseekV2Config().vocab_size == 32000` (not 102400) - `DeepseekVLV2TextConfig.kv_lora_rank` is forced to `None` So omitted fields silently resolve to wrong values: 1. `kv_lora_rank=None` makes `ModelConfig.is_deepseek_mla` return `False` (`use_mla=False`); combined with the non-zero `qk_*_head_dim` defaults the decoder falls through to `DeepseekV2Attention`, which computes `kv_lora_rank + qk_rope_head_dim` → `None + int` → `TypeError`. 2. `vocab_size=32000` mismatches the checkpoint's 102400-row `embed_tokens`, tripping the assertion in `VocabParallelEmbedding.weight_loader`. This fix restores the original DeepSeek-VL2 defaults **only for fields the checkpoint does not set**, before building the text config: ``` deepseek_vl2_reference_defaults = { \"vocab_size\": 102400, \"kv_lora_rank\": 512, \"qk_nope_head_dim\": 128, \"qk_rope_head_dim\": 64, \"v_head_dim\": 128, } for key, value in deepseek_vl2_reference_defaults.items(): language_config.setdefault(key, value) ``` ## Test Plan Serve deepseek-ai/deepseek-vl2-small and confirm it no longer crashes ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51302", "createdAt": "2026-08-06T19:26:22Z", "updatedAt": "2026-08-12T23:24:11Z", "timestamp": "2026-08-12T23:24:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "deepseek" ], "author": "shepark", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:850037df10783c479f80", "signalId": "github:vllm-project/vllm:pull_request:48693", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48693", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Make Gemma 4 suppress-token masking CUDA-graph safe", "text": "## Purpose Gemma 4 CUDA graph capture used Python suppress-token IDs and advanced indexed assignment with a Python scalar. Both can introduce host-to-device work during capture. Store validated IDs as nonpersistent model buffers and use `index_fill_` for graph-safe masking in the target, unified, and MTP paths. The centroid MTP proposer also returned sparse token IDs without applying the same suppression policy as full logits. Sparse argmax now masks those IDs and, if every selected candidate is suppressed, returns the same lowest eligible fallback token that full-vocabulary argmax selects. Invalid dtype/range values and suppression of the entire vocabulary fail at model initialization. Addresses #48503. Use `Fixes #48503` after the exact affected-model evaluation below passes. #48509 covered only part of the failure; #48515 closed unmerged when its source repository disappeared. Mandatory duplicate searches found no competing open PR. AI assistance was used. The human submitter must review every changed line and the recorded model result before readiness. ## Validation Current head `79d3a4c7f` is based on upstream `1a659a0c3`. ```text pytest -q tests/model_executor/test_gemma4_suppress_tokens.py 29 passed, 3 skipped # local host has no CUDA pre-commit run --files all applicable hooks passed, including mypy git diff --check passed ``` The suite covers direct dense masking, sparse/full argmax parity for none, partial, and all-selected suppression, invalid types and bounds, scalar zero, empty values, nonpersistent registration, actual LM-head placement, module and meta-device movement, and separate dense and sparse CUDA-graph capture/replay tests. On an earlier H100 revision with the same dense masking mechanism, `Gemma4MTP.compute_logits` captured and replayed twice successfully and the then-current focused suite passed. The newly strengthened sparse/fallback CUDA tests exercise both partial and all-selected suppression, but have not yet run on H100, so that historical result is not claimed as a current-head GPU pass. Still required before readiness: - Run the complete current-head focused suite on H100. - Start and generate under CUDA graphs with `google/gemma-4-12B-it-qat-w4a16-ct` and the affected Gemma 4 MTP assistant. - Compare eager and graph greedy token IDs or run the repository-appropriate deterministic model evaluation because this changes proposal/logit output. - Human review and reviewer-triggered full CI. ## Release note Gemma 4 with MTP speculative decoding no longer performs suppress-token host-to-device copies during CUDA graph capture, and sparse centroid proposals honor the same validated suppression policy as full logits. After affected-model evaluation and mainline merge, this crash/correctness fix is a v0.26 backport candidate.", "url": "https://github.com/vllm-project/vllm/pull/48693", "createdAt": "2026-07-15T06:22:39Z", "updatedAt": "2026-08-12T23:22:17Z", "timestamp": "2026-08-12T23:22:17Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "speculative-decoding", "needs-rebase", "v1", "nvidia" ], "author": "dumko2001", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0843d665e5f419bf01e2", "signalId": "github:vllm-project/vllm:pull_request:52061", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52061", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add native forward-pass metrics emission", "text": "## Purpose Related to #38760. Add an opt-in native vLLM forward-pass metrics (FPM) pipeline for production per-iteration observability. This moves production FPM emission out of an external custom scheduler while keeping the disabled path lightweight. The change: - defines the FPM v1 wire contract and utilities together in `vllm/v1/metrics/forward_pass_metrics.py`; - captures scheduled and queued prefill/decode aggregates from an explicit scheduler interface; - measures `execute_model` directly with pooled CUDA events on the executor output rank; - drains completed events without synchronizing the inference hot path and only waits during the cold transition to idle/shutdown; - joins delayed CUDA timings to their originating scheduler snapshots by iteration ID; - follows vLLM's existing native [KV-event ZMQ emission pattern](https://github.com/vllm-project/vllm/blob/98f86b9c02329200a0390aecfe598e27928cbf40/vllm/distributed/kv_events.py#L290-L367): one publisher per DP rank, a background publishing thread, and `(topic, sequence, msgpack payload)` multipart frames ([framing reference](https://github.com/vllm-project/vllm/blob/98f86b9c02329200a0390aecfe598e27928cbf40/vllm/distributed/kv_events.py#L459-L466)); - keeps FPM delivery intentionally best-effort and non-blocking, with a bounded queue and no replay endpoint, so observability cannot backpressure inference; and - exposes opt-in CLI/config fields for the port, worker ID, and publisher queue bound. When disabled, no CUDA events, publisher, or pending snapshot state are allocated. This does not duplicate #48292. That PR records the wall-clock time spent in `future.result()` in iteration-detail logs. This PR implements a versioned streaming metrics contract, scheduler state aggregation, direct CUDA timing, and external ZMQ emission. It also does not move external self-benchmark state machines into vLLM. ## Difference from the current Dynamo implementation The references below are pinned to [`ai-dynamo/dynamo@427046c`](https://github.com/ai-dynamo/dynamo/tree/427046cf5c0d84c2d879d2c82b1e127665540961) so this comparison remains stable as Dynamo evolves. - **Ownership and activation:** Dynamo currently enables production FPM by replacing vLLM's scheduler with `InstrumentedScheduler`; an already configured custom scheduler prevents that injection ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/args.py#L314-L336)). This PR makes production FPM a native, opt-in vLLM engine facility and uses a narrow scheduler-state interface, so production FPM no longer requires a Dynamo scheduler subclass. - **Iteration timing:** Dynamo records host `time.monotonic()` timestamps around `schedule()` and model-output arrival, then estimates `wall_time` from either consecutive output updates or the schedule-to-update interval ([measurement description](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L52-L68), [implementation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1594-L1611), [calculation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1644-L1674)). This PR records pooled CUDA events immediately around `execute_model`, drains them asynchronously, and joins the result to the originating engine iteration. Thus the engine directly emits measured `execute_model` GPU timeline duration instead of a scheduler-derived approximation. - **Aggregation and emission:** Dynamo currently computes scheduled/queued aggregates in `InstrumentedScheduler` and passes each payload to its own background ZMQ publisher ([aggregation](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1676-L1692), [publisher](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1347-L1439)); Dynamo's parent process then relays that raw stream to the event plane ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/main.py#L445-L488)). This PR moves the aggregation, iteration association, CUDA timing, and raw ZMQ emission into vLLM. The existing Dynamo relay remains the downstream integration point. - **Self-benchmarking:** Dynamo benchmark mode currently requires the same `InstrumentedScheduler` ([configuration](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/args.py#L338-L357)), and its benchmark state machine controls synthetic scheduler outputs and FPM collection inside that subclass ([reference](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/vllm/instrumented_scheduler.py#L1523-L1554)). This PR deliberately leaves self-benchmarking in Dynamo; Dynamo can retain its custom scheduler for benchmark mode while ordinary serving uses native vLLM FPM. - **Wire contract:** Dynamo's current payload is version 1 ([version](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/common/forward_pass_metrics.py#L45), [schema](https://github.com/ai-dynamo/dynamo/blob/427046cf5c0d84c2d879d2c82b1e127665540961/components/src/dynamo/common/forward_pass_metrics.py#L153-L189)). This PR keeps `version=1` and explicitly labels the changed `wall_time` semantics with `timing_scope=execute_model_cuda`; it does not bump the wire version. ## Test Plan - Run the focused FPM and argument parsing unit tests. - Run all staged pre-commit hooks. - Compare FPM disabled/enabled serving performance with an interleaved A/B campaign and a live subscriber. - Compare every non-timing FPM field against the existing Dynamo scheduler implementation under serial, batched, prefix-cache, chunked-prefill, queued-prefill, decode-heavy, and forced-preemption traffic. ## Test Result All staged pre-commit hooks passed: ```bash .venv/bin/pre-commit run ``` This included ruff check/format, mypy, SPDX, forbidden-import, CUDA-API, configuration-default, and repository-specific validation hooks. The focused local pytest command was attempted after installing the documented precompiled editable build: ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto .venv/bin/python -m pytest \\ tests/v1/metrics/test_forward_pass_metrics.py \\ tests/engine/test_arg_utils.py -q ``` Collection remains blocked on this host because its environment contains CPU Torch while the installed precompiled vLLM extension requires `libtorch_cuda.so`. No test assertion was reached; CI should run these tests in a consistent build environment. GPU validation used one H100 NVL with `Qwen/Qwen3-0.6B` and controlled images built from the same vLLM source: - 132/132 overhead benchmark samples completed successfully across serial, concurrent, decode-heavy, prefill-heavy, chunked-prefill, prefix-cache, and open-loop traffic. - The stable worst-case signal was about 4.4 microseconds per iteration (`+0.283%` TPOT and `-0.269%` output throughput) for concurrency-1 short serial traffic. Decode-heavy output throughput changed by `+0.036%`. Longer concurrent and prefix-cache reruns showed no reproducible throughput regression. - A live subscriber received 151,263 FPM messages with zero observed sequence gaps, queue-full/drop logs, publisher errors, or server failures. - Native FPM matched the existing scheduler implementation on 1,249/1,249 deterministic active iterations for every non-timing field, including scheduled/queued request counts, token and KV-token sums, and variances. Forced memory pressure observed queued prefill and preempted decode states on both implementations. - All serving requests succeeded and prompt/completion token totals matched; this change does not alter model output or accuracy. The payload continues to use wire `version=1`. Timing values intentionally differ from the existing scheduler approximation, so consumers must use the explicit `timing_scope=execute_model_cuda` field when interpreting `wall_time`. This implementation and validation were developed with assistance from OpenAI Codex. This draft PR is intentionally open for the submitter's line-by-line human review and must not be marked ready until that review is complete. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR and related issue are described. - [x] The test plan and commands are included. - [x] Test and performance results are included. - [x] No documentation update is required for this opt-in internal interface.
", "url": "https://github.com/vllm-project/vllm/pull/52061", "createdAt": "2026-08-12T22:38:02Z", "updatedAt": "2026-08-12T23:21:45Z", "timestamp": "2026-08-12T23:21:45Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "mrv2" ], "author": "tedzhouhk", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b55431e2529320d262d2", "signalId": "github:vllm-project/vllm:pull_request:52000", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52000", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[V1][CUDA graph] Dispatch uniform-decode batches to a padded FULL graph instead of falling to eager PIECEWISE", "text": "**TL;DR:** with speculative decode, some concurrencies silently fall back to running attention *eagerly* every step instead of from a captured CUDA graph, adding a fixed per-step latency bubble. This makes them dispatch to a captured graph instead. **What happens.** When `dispatch()` picks a CUDA graph for a batch, it looks up the smallest captured graph big enough to hold it. With speculative decode there are two kinds of graphs captured at *different* sizes: - **FULL decode** graphs (fast: whole decode step, incl. attention, is captured) - **PIECEWISE** graphs (attention runs eagerly, not captured) Their sizes interleave, so for some batch sizes the smallest graph that fits happens to be a PIECEWISE one — even though a slightly larger FULL decode graph exists and could serve the batch by padding it with a few dummy rows. PIECEWISE always matches, so the batch takes it and pays the eager-attention cost every decode step. **Example** (`decode_query_len = 3`, default capture ladder): | batch (conc) | tokens | smallest graph that fits | result | |---|---|---|---| | conc-4 | 12 | size-16 **PIECEWISE** | eager attention ❌ | | conc-4 (this PR) | 12 | size-18 **FULL decode** (pad 4→6 reqs) | captured ✅ | Symptom on a long-context agentic run: conc-4 / conc-12 show decode ITL p50 ~75 ms — *slower* than conc-24 doing 6× the work — with the GPU stuck at launch-bound power while reading \"100% busy\". **Fix.** FULL decode graphs are ignored by non-decode batches anyway (they only match uniform decode), so we simply offer them — smallest-first — before the PIECEWISE fallback for every batch size. `dispatch()` then picks the smallest FULL decode graph that fits (padding a few requests) and only falls back to PIECEWISE when none does. - No change to `dispatch()` or `_is_compatible()`. - **Zero extra captured graphs** — just reorders the existing candidate lists. - Non-decode and non-spec configs are unaffected. **One assumption:** the attention backend must tolerate padding a uniform-decode batch with dummy rows (4 real → 6). The ROCm AITER MLA backend already does this (`_uniform_padded_mtp_qo_len` in `rocm_aiter_mla.py`). Other backends with a separate FULL decode routine should confirm their metadata builder handles the padded rows. ## Test Plan ## Test Plan - `python -m py_compile vllm/v1/worker/gpu/cudagraph_utils.py`. - Unit test: build a `CudaGraphManager` with a separate decode routine, `decode_query_len=3`, capture sizes `[8,16,24]`; assert `dispatch(num_reqs=4, num_tokens=12, uniform_token_count=3, num_active_loras=0)` returns a FULL decode descriptor (not PIECEWISE / NONE). - E2E: Kimi-K3 TP8 on MI355X, spec decode (`num_spec=2`), `FULL_AND_PIECEWISE`, long-context agentic sweep; compare conc-4 / conc-12 decode ITL before vs after. ## Test Result - `py_compile` — clean. - Kimi-K3 MI355X: conc-4 / conc-12 previously dropped to eager PIECEWISE (ITL p50 ~75 ms, GPU launch-bound ~370 W); with this change they dispatch to the size-18 / size-42 FULL decode graphs, removing the bubble. - On-box before/after ITL numbers to be pasted here; consider opening as **draft** until measured.", "url": "https://github.com/vllm-project/vllm/pull/52000", "createdAt": "2026-08-12T14:13:19Z", "updatedAt": "2026-08-12T23:18:39Z", "timestamp": "2026-08-12T23:18:39Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "nvidia", "mrv2", "verified" ], "author": "xiaohuguo2023", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:567a2f5bb5cb797db329", "signalId": "github:vllm-project/vllm:pull_request:51040", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][K3] Extend FP8 asm MLA prefill to non-divisor small head counts", "text": "## Context The AITER FP8 MLA **prefill** path (`mla_prefill_ps_asm_fwd` + `mla_reduce_v1`) is gated on `num_heads % 16 == 0`. Kimi-K3 has 96 heads over `kv_lora_rank=512` → **12 heads/rank at TP8**, a non-divisor of 16, so its FP8 prefill falls back to the BF16 FMHA decompress path. That fallback builds a bf16 working set not covered by the FP8 KV-pool accounting and exhausts the activation arena at long context (K3 OOM'd at ~197k tokens, KV pool <4% used). This is the **prefill counterpart to #50578** (asm decode pad 12→16). MLA attention is independent per query head over the shared latent KV, so padding the query heads up to 16 and slicing the result back is exact. Rebased onto current `main` (includes merged #50578). Prefill-only diff: **+48 / −11**, one file (`rocm_aiter_mla.py`). ## Changes **FP8 prefill enablement (this PR):** - Relax the `_fp8_prefill_enabled` gate to also allow `0 < num_heads < 16`, and **gate it on FP8 KV** (`kv_cache_dtype_str == \"fp8\"` in the metadata builder; `is_quantized_kv_cache(kv_cache_dtype)` in the impl). Without this, a bf16-KV serve would reserve PS workspace unnecessarily. - In `_mla_fp8_prefill_attn`: replicate-pad Q/K/V to 16 via `AiterMLAHelper.get_mla_padded_q` (from #50578), run PS asm prefill + `mla_reduce_v1` at 16 heads, slice output back to `num_heads`. - Set `num_head_k = max(16, num_heads)` in `_init_fp8_prefill_ps_buffers` and `_build_fp8_prefill_ps_metadata` so work/reduce maps match padded tensors — also cuts partial tiles (`gcd(16,256)=16` vs `gcd(12,256)=4`), reclaiming **~6 GiB** PS workspace. Decode padding / Gluon routing is **not** in this PR — that is #50578 (merged). ## Dependencies / related - **#50578** (merged) — `AiterMLAHelper.get_mla_padded_q` / asm decode for non-divisor heads. - **ROCm/aiter#4452** (merged `a63ede724`) — 64-bit paged-KV byte offsets for >4 GB addressing. - Composes with **#48712** (gate PS workspace on fp8 KV) — this PR includes the same gate locally. ## vllm serve command **Hardware:** 8× MI355X (gfx950), TP8 **Model:** `moonshotai/Kimi-K3` (mxfp4) ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export AITER_SITUV2_A8W4=1 export SAFETENSORS_FAST_GPU=1 export GPU_ARCHS=gfx950 export HF_HUB_CACHE=/dev/shm/hf-cache HF_HOME=/dev/shm/hf-cache export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 export VLLM_HTTP_TIMEOUT_KEEP_ALIVE=900 export HIPBLASLT_WORKSPACE_SIZE=32768 export CUBLASLT_WORKSPACE_SIZE=32768 MODEL=/path/to/Kimi-K3 # e.g. /shared_nfs/models/Kimi-K3 vllm serve \"$MODEL\" --served-model-name moonshotai/Kimi-K3 \\ --host 0.0.0.0 --port 8888 --tensor-parallel-size 8 --async-scheduling \\ --distributed-executor-backend mp \\ --gpu-memory-utilization 0.95 \\ --max-num-seqs 64 --max-model-len 1048576 --max-num-batched-tokens 4096 \\ --trust-remote-code --load-format auto --moe-backend aiter \\ --kv-cache-dtype fp8 --attention-backend ROCM_AITER_MLA --mm-encoder-tp-mode data \\ --compilation-config '{\"mode\":3,\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --enable-prefix-caching --no-disable-hybrid-kv-cache-manager \\ --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 --enable-auto-tool-choice \\ --enable-prompt-tokens-details --disable-uvicorn-access-log ## Test results ### Long-context prefill (functional) | | Before PR-A | After PR-A | |---|-------------|------------| | Fresh prefill @ util 0.95 | OOM ~197k tokens (BF16 FMHA fallback, KV <4%) | **470k (68 s)**, **590k (28.6 s)** OK | | PS workspace | ~4032 partial tiles (~12 heads) | ~960 partial tiles (~6 GiB saved) | ### Accuracy — GSM8K (lm-eval, full 1319-test split, 5-shot greedy, conc=32, ctx=16384) Patches applied for this run: **#50578 + PR-A + #50618 only** (no cgmem/MoE/kv_b_proj). Server at `gpu-memory-utilization 0.88`. ```bash export HF_HUB_CACHE=/dev/shm/hf-cache HF_HOME=/dev/shm/hf-cache export OPENAI_API_KEY=EMPTY python3 -m lm_eval --model local-chat-completions --apply_chat_template \\ --include_path /path/to/InferenceX/utils/evals \\ --tasks /path/to/InferenceX/utils/evals/gsm8k.yaml \\ --output_path /tmp/eval_out --log_samples \\ --model_args \"model=moonshotai/Kimi-K3,base_url=http://localhost:8888/v1/chat/completions,api_key=EMPTY,eos_string=,max_retries=5,num_concurrent=32,timeout=1800,tokenized_requests=False,max_length=16384\" \\ --gen_kwargs \"max_tokens=12288,temperature=0,top_p=1\" ``` | Run | n | strict-match EM | flexible-extract EM | |-----|---|-----------------|---------------------| | **With PR-A** (MI355X node 193, 2026-08-09) | 1319 | **97.04%** ± 0.47% | **96.97%** ± 0.47% | | Prior stack (cgmem+MoE+PR-A, node 1319, util 0.95) | 1319 | 96.89% ± 0.48% | 96.89% ± 0.48% | | Baseline replicate (stock + ASM deps, node 1319) | 1319 | 96.66% ± 0.49% | 96.66% ± 0.49% | No accuracy regression vs baseline; delta within run noise.", "url": "https://github.com/vllm-project/vllm/pull/51040", "createdAt": "2026-08-04T14:48:33Z", "updatedAt": "2026-08-12T23:12:27Z", "timestamp": "2026-08-12T23:12:27Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "verified", "k3" ], "author": "xiaohuguo2023", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8915dc2f29990b227cf6", "signalId": "github:vllm-project/vllm:pull_request:51590", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51590", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Memory] Measure complete CUDA graph capture footprint for KV budgeting", "text": "## Purpose Fix CUDA graph memory undercounting in `profile_cudagraph_memory()`. The old path profiled only two descriptors per graph mode and extrapolated the rest. It also summed per-mode deltas even though FULL and PIECEWISE share one runtime pool, and it did not budget memory allocated during profiling setup or retained after profiling cleanup. The shortfall was assigned to the KV cache, so `--gpu-memory-utilization` could be exceeded during capture or serving while the KV pool was still mostly empty. This PR: - Profiles **every** capture descriptor in **one end-to-end span** across all decoder modes instead of extrapolating from two samples per mode. - Includes profiling KV-cache / attention-metadata setup and **`cudagraph_profiling_retained_memory`** in the estimate (scratch the real capture reuses). - Skips the profiling capture entirely when `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` (previously ran and discarded the result). - Guards the unaccounted-graph-memory warning so it only fires when graphs will actually be captured (`--enforce-eager` / `cudagraph_mode=NONE` no longer trigger it). - Preserves cleanup and graph-pool restoration when profiling raises. On Kimi-K3 (8× MI355X, TP8), the estimate rose from ~**1.3 GiB/GPU** to **~4.7–16.9 GiB/GPU** depending on config — matching what capture actually needs. This is accounting only; no kernel or graph-capture logic changes. Fixes agentic OOMs where `HSA_STATUS_ERROR_OUT_OF_RESOURCES` occurred at **3–16%** KV usage because graph/setup memory was budgeted to the KV cache. ## Test Plan **Unit tests** (CPU-only; fake device allocator, no GPU required): ```bash pytest tests/v1/cudagraph/test_cudagraph_memory_profiling.py \\ tests/v1/cudagraph/test_cudagraph_memory_worker.py -q ``` Coverage: - `test_cudagraph_memory_profiling.py` — all descriptors and modes; shared-pool span accounting; negative deltas and zero clamping; encoder graphs; profiling-KV setup; retained memory; cleanup on capture failure. - `test_cudagraph_memory_worker.py` — estimate profiled and subtracted from KV budget; `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` skips profiling; no warning when `cudagraph_mode=NONE`. **Serving smoke** (ROCm, optional for reviewers): - Kimi-K3, TP8, FP8 KV, `FULL_AND_PIECEWISE`, `gpu-memory-utilization=0.95` - Confirm log line `Estimated CUDA graph memory: … GiB total` reflects full capture footprint and serve completes without HSA OOM at previously failing batch/seq limits. ## Test Result **Unit tests (this branch):** ``` 13 passed in 5.89s ``` Negative control: 8 behavior-changing tests fail on pristine `upstream/main`. **Serving (Kimi-K3, MI355X TP8):** | config | metric | before (old sampler) | after (this PR) | |---|---|---:|---:| | cm16k s128, c16, util 0.95 | graph memory estimate | ~1.3 GiB/GPU | 4.7–7.1 GiB/GPU | | cm16k s128, c16, util 0.95 | agentic output tok/s | 61 | **227 (+272%)** | | cm16k s128, c16, util 0.95 | TTFT / ITL avg | 56s / 187ms | **2.9s / 50ms** | | cm8k s128, c24, util 0.95 | graph memory estimate | ~1.3 GiB/GPU | **16.90 GiB/GPU** | | cm8k s128, c24, util 0.95 | run health | OOM (transient workspace) | **440 reqs, 0.23% err** | | cm8k s128, c24, util 0.93 | GPU KV cache size | — | **3,660,123 tokens**, clean run | Startup: profiling every descriptor adds ~8s on Kimi-K3 TP8 (21s → 29s for profiling + capture). Warmup reuse across profiling and real capture is intentionally deferred to a follow-up PR.", "url": "https://github.com/vllm-project/vllm/pull/51590", "createdAt": "2026-08-09T23:55:48Z", "updatedAt": "2026-08-12T23:12:21Z", "timestamp": "2026-08-12T23:12:21Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm", "intel-gpu", "nvidia", "verified", "k3" ], "author": "xiaohuguo2023", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b0dd3410d341be095c31", "signalId": "github:vllm-project/vllm:pull_request:50017", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50017", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] [bugfix] Chunked prefill paged decode masked load perf", "text": "## Purpose Observed 10-15% latency performance regression for serving Qwen/Qwen3-30B-A3B-Thinking-2507; trace revealed that `kernel_paged_attention_2d` was the culprit, taking 1.33x as long on average on v0.25.0 vs v0.24.0. #47305 fixed correctness but introduced performance drop because of universally applied masking. Edited masking so that it is only enacted during last block, when token id can be greater than sequence length. ## Test Plan ### Latency Ensure that fixes solve benchmark latency issue. `VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 --dtype float16 -tp 1 --no-enable-prefix-caching --disable-uvicorn-access-log` `vllm bench serve --model Qwen/Qwen3-30B-A3B-Thinking-2507 --percentile-metrics tpot,ttft,itl,e2el --dataset-name random --ignore-eos --temperature 0 --max-concurrency 1 --num-prompts 10 --random-input-len 1024 --random-output-len 1024` ### Accuracy Ensure accuracy remains using gsm8k benchmark. lm_eval --model local-completions --model_args model=Qwen/Qwen3-30B-A3B-Thinking-2507,base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=256,max_retries=10,max_gen_toks=2048 --batch_size auto --tasks gsm8k --num_fewshot 5 --limit 1000 ## Test Result ### Latency | Metric | v0.24.0 baseline | v0.25.0 (before fix) | v0.25.0 + fix | |---|---|---|---| | Output tok/s | 91.33 | 79.76 | **97.08** | | Mean TPOT (ms) | 10.69 | 12.25 | **10.23** | | Median E2EL (ms) | 11006 | 12613 | **10546** | ### Accuracy gsm8k, 5-shot, `--limit 1000`, same serve config: | Filter | v0.24.0 baseline | v0.25.0 + fix | |---|---|---| | flexible-extract | 0.901 ± 0.0094 | 0.913 ± 0.0089 | | strict-match | 0.920 ± 0.0086 | 0.917 ± 0.0087 | Deltas (+1.2pt flexible, −0.3pt strict) are within stderr. --- No other PRs address this performance regression. --- This PR was prepared with AI assistance (Claude). All changes were reviewed and the test/eval commands were run by Andy.", "url": "https://github.com/vllm-project/vllm/pull/50017", "createdAt": "2026-07-27T16:49:53Z", "updatedAt": "2026-08-12T23:04:02Z", "timestamp": "2026-08-12T23:04:02Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "rocm", "ready", "v1" ], "author": "afriedri", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:7d87aae2a37c3cb6a192", "signalId": "github:vllm-project/vllm:pull_request:45559", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45559", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Kernel] Extend skinny gemm N=5 to N=8 cases on GFX12 (RDNA4) using SWMMAC optimization", "text": "## Purpose This PR adds an optimized ROCm GFX12 (RDNA4) SWMMAC-based skinny GEMM path. The new path targets serving workloads with low max concurrency, matching the benchmarked decode GEMM range of N=5-8. The implementation uses sparse wave matrix multiply-accumulate instructions on GFX12 (RDNA4) to improve throughput for these skinny GEMM cases, while keeping the existing fallback path for unsupported cases. Skinny GEMMs can be important for ROCm/GFX12 (RDNA4) serving performance, which is often used in local or small-scale serving setups. In these deployments, the server is expected to handle single users, or a small number of users with a relatively lower request rate, so the active decode batch can remain small instead of being filled to a large batch size. This optimization follows the same general direction described in the Hugging Face technical blog, [*Creating custom kernels for the AMD MI300*](https://huggingface.co/blog/mi300kernels). The blog discusses custom kernels for AMD MI300X vLLM serving and describes a skinny GEMM optimization for inputs with 8 or fewer rows. The key idea is to map each logical skinny A row into two adjacent SWMMAC sparse rows. The first sparse row selects one set of 16-bit pairs from `A[n, :]`, and the second sparse row selects the complementary pairs from the same logical row. Both sparse rows multiply with the same normal `B[32 x 16]` fragment, producing two partial accumulator rows. The epilogue then adds the adjacent partial rows back together to recover the final logical output row. This interleaved sparse-row layout lets the kernel use the SWMMAC instruction shape more effectively for N=5-8 instead of leaving the skinny rows underfilled. A related optimization direction is also being explored in IREE: [*[AMDGPU][Codegen] Optimizing skinny GEMM workloads with sparse matrix instructions*](https://github.com/iree-org/iree/issues/22863). The IREE issue describes using sparse matrix instructions to optimize skinny GEMM workloads on AMD CDNA GPUs. Although the IREE discussion focuses on IREE codegen and CDNA-style sparse MFMA examples, the motivation is closely related: skinny GEMM workloads can waste matrix instruction capacity due to padding or under-filled dimensions, and sparse matrix instructions can recover more effective throughput. Note: this PR is not a direct port of either implementation. It adapts the same idea to the vLLM ROCm/GFX12 (RDNA4) path using SWMMAC with tuned optimization strategies for the measured small-N decode GEMM cases. **This PR:** * Adds a GFX12 (RDNA4) SWMMAC skinny GEMM kernel path * Targets decode GEMM shapes with N=5-8 * Preserves the existing fallback path for unsupported cases * Adds correctness coverage for the new SWMMAC path across: * N = 5, 6, 7, 8 * FP16 and BF16, matching the existing skinny GEMM dtype coverage * Multiple M/K sizes * Aligned (divisible by 16) and non-aligned M dimensions * Large-shape cases Note: N=4 was also evaluated, but it did not show a benefit in this implementation, so this PR keeps the optimized path scoped to N=5-8. ## Test Plan ### Unit tests ```bash pytest -v tests/kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_swmmac_gemm_kernel ``` ### GSM8K end-to-end accuracy The GSM8K run was performed with the vLLM server launched using --max-num-seqs 8, and lm_eval used num_concurrent=8. This keeps the active decode batch within the targeted skinny GEMM range and exercises the SWMMAC N=5-8 paths. **Model:** `Qwen/Qwen3-4B` Server command: ```bash vllm serve Qwen/Qwen3-4B \\ --host 0.0.0.0 \\ --port 8000 \\ --dtype bfloat16 \\ --max-num-seqs 8 \\ --gpu-memory-utilization 0.90 ``` Benchmark command: ```bash lm_eval \\ --model local-completions \\ --model_args \"model=${MODEL},base_url=http://0.0.0.0:8000/v1/completions,num_concurrent=8,max_retries=10,max_gen_toks=1024,max_length=8192,timeout=60000\" \\ --batch_size auto \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path ./results_qwen3_4b_gsm8k \\ --log_samples ``` ### Performance Benchmark - **Hardware:** GPU: AMD Radeon AI PRO R9700, gfx1201, GFX12 (RDNA4), **x1** | CPU: AMD Ryzen 9 9950X 16-Core Processor, **x32** | RAM: 123Gi - **Model:** Qwen/Qwen3-4B - **Benchmark setup:** - **Concurrency:** 5, 6, 7, 8 - **Input/output token shapes:** 512/128, 1024/256, 2048/128, 2048/512, 4096/512, 8192/512, 1024/2048, 512/4096, 512/8192 Baseline commit (https://github.com/vllm-project/vllm/commit/6bdabbad5bce747865fd3a249658518a4269cc22): ```text commit 6bdabbad5bce747865fd3a249658518a4269cc22 Author: Jee Jee Li <[pandaleefree@gmail.com](mailto:pandaleefree@gmail.com)> Date: Sun May 31 13:16:12 2026 +0800 [CI/Build] Enable Step3p7ForConditionalGeneration testing (#43956) Signed-off-by: Jee Jee Li <[jeejeelee@inferact.ai](mailto:jeejeelee@inferact.ai)> ``` Server script: ```bash #!/bin/bash set -euo pipefail rm -rf ~/.cache/vllm vllm serve Qwen/Qwen3-4B \\ --dtype auto \\ --max-model-len 16384 \\ --gpu-memory-utilization 0.9 ``` Benchmark script: ```bash #!/bin/bash set -euo pipefail BASE_URL=${BASE_URL:-http://localhost:8000} RESULT_DIR=${RESULT_DIR:-./bench-results} RESULT_PREFIX=${RESULT_PREFIX:-qwen3-4b} CONCURRENCIES=${CONCURRENCIES:-\"5 6 7 8\"} NUM_WARMUPS=${NUM_WARMUPS:-16} for workload in \\ \"512 128\" \\ \"1024 256\" \\ \"2048 512\" \\ \"4096 512\" \\ \"8192 512\" \\ \"2048 128\" \\ \"1024 2048\" \\ \"512 4096\" \\ \"512 8192\"; do set -- $workload INPUT_LEN=$1 OUTPUT_LEN=$2 for C in ${CONCURRENCIES}; do NUM_PROMPTS=$((C * 10)) vllm bench serve \\ --backend openai-chat \\ --base-url \"${BASE_URL}\" \\ --endpoint /v1/chat/completions \\ --model Qwen/Qwen3-4B \\ --dataset-name random \\ --input-len \"${INPUT_LEN}\" \\ --output-len \"${OUTPUT_LEN}\" \\ --num-prompts \"${NUM_PROMPTS}\" \\ --request-rate inf \\ --max-concurrency \"${C}\" \\ --num-warmups \"${NUM_WARMUPS}\" \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,99 \\ --save-result \\ --result-dir \"${RESULT_DIR}\" \\ --result-filename \"${RESULT_PREFIX}-${INPUT_LEN}-${OUTPUT_LEN}-C${C}.json\" \\ --metadata model=qwen3-4b concurrency=\"${C}\" workload=random_${INPUT_LEN}_${OUTPUT_LEN} num_prompts=\"${NUM_PROMPTS}\" done done ``` ## Test Results ### Unit tests 216 passed, 16 warnings in 52.55s ### GSM8K end-to-end accuracy | Version | flexible-extract exact_match | strict-match exact_match | Request progress | |---|---|---|---| | **Baseline** | 0.8613 ± 0.0095 | 0.8650 ± 0.0094 | 1319 requests in 11:13, 1.96 it/s | | **SWMMAC** | 0.8605 ± 0.0095 | 0.8666 ± 0.0094 | 1319 requests in 09:48, 2.24 it/s | *The GSM8K accuracy is comparable between the baseline and SWMMAC runs. The SWMMAC run also completed faster in this setup.* ### Performance Benchmark Results The SWMMAC path improves serving throughput across all 36 measured cases. Averaged by workload across concurrency 5-8, request throughput improves by about 10-23%, output token throughput improves by about 7-23%, and mean E2EL decreases by about 6-19%. TTFT stays neutral. #### Performance summary Average percentage change across concurrency 5-8 for each workload: | Metric | 512/128 | 1024/256 | 2048/128 | 2048/512 | 4096/512 | 8192/512 | 1024/2048 | 512/4096 | 512/8192 | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Request throughput | +23.0% | +22.2% | +18.5% | +18.8% | +12.2% | +9.8% | +20.8% | +16.0% | +20.8% | | Output token throughput | +23.2% | +22.5% | +18.5% | +18.5% | +12.0% | +7.2% | +21.2% | +20.2% | +17.0% | | Mean E2EL | -19.0% | -18.2% | -15.5% | -15.8% | -10.8% | -6.5% | -17.5% | -17.0% | -14.5% | | Mean TTFT | -6.0% | -5.8% | -7.5% | -1.8% | +2.8% | +0.0% | -0.2% | -0.5% | -0.2% | #### Detailed performance results | Concurrency | Input | Output | Baseline Req/s | Baseline TTFT ms | Baseline Out tok/s | Baseline E2EL ms | SWMMAC Req/s | SWMMAC TTFT ms | SWMMAC Out tok/s | SWMMAC E2EL ms | Req/s Δ | Req/s Δ% | TTFT Δ ms | TTFT Δ% | Out tok/s Δ | Out tok/s Δ% | E2EL Δ ms | E2EL Δ% | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 5 | 512 | 128 | 1.83 | 91.64 | 233.69 | 2737.24 | 2.24 | 90.12 | 286.76 | 2229.83 | 0.41 | 22% | -1.52 | -2% | 53.07 | 23% | -507.41 | -19% | | 6 | 512 | 128 | 2.15 | 101.55 | 274.57 | 2795.15 | 2.65 | 91.26 | 338.93 | 2264.20 | 0.50 | 23% | -10.29 | -10% | 64.36 | 23% | -530.95 | -19% | | 7 | 512 | 128 | 2.52 | 106.90 | 322.28 | 2778.32 | 3.11 | 100.77 | 397.88 | 2249.62 | 0.59 | 23% | -6.13 | -6% | 75.60 | 23% | -528.70 | -19% | | 8 | 512 | 128 | 2.91 | 104.41 | 372.54 | 2746.35 | 3.60 | 98.61 | 460.84 | 2219.05 | 0.69 | 24% | -5.80 | -6% | 88.30 | 24% | -527.30 | -19% | | 5 | 1024 | 256 | 0.89 | 107.05 | 228.34 | 5604.23 | 1.09 | 110.63 | 278.86 | 4588.01 | 0.20 | 22% | 3.58 | 3% | 50.52 | 22% | -1016.22 | -18% | | 6 | 1024 | 256 | 1.04 | 122.90 | 267.14 | 5747.64 | 1.28 | 115.96 | 327.74 | 4684.68 | 0.24 | 23% | -6.94 | -6% | 60.60 | 23% | -1062.96 | -18% | | 7 | 1024 | 256 | 1.21 | 155.45 | 309.23 | 5791.30 | 1.48 | 146.18 | 379.74 | 4715.45 | 0.27 | 22% | -9.27 | -6% | 70.51 | 23% | -1075.85 | -19% | | 8 | 1024 | 256 | 1.38 | 168.67 | 354.14 | 5779.56 | 1.69 | 145.70 | 433.32 | 4723.14 | 0.31 | 22% | -22.97 | -14% | 79.18 | 22% | -1056.42 | -18% | | 5 | 2048 | 128 | 1.54 | 256.74 | 197.00 | 3244.90 | 1.86 | 221.03 | 238.20 | 2683.05 | 0.32 | 21% | -35.71 | -14% | 41.20 | 21% | -561.85 | -17% | | 6 | 2048 | 128 | 1.78 | 262.92 | 227.79 | 3366.40 | 2.11 | 235.44 | 270.09 | 2838.57 | 0.33 | 19% | -27.48 | -10% | 42.30 | 19% | -527.83 | -16% | | 7 | 2048 | 128 | 2.04 | 287.25 | 260.95 | 3427.79 | 2.41 | 286.41 | 308.32 | 2900.50 | 0.37 | 18% | -0.84 | 0% | 47.37 | 18% | -527.29 | -15% | | 8 | 2048 | 128 | 1.53 | 1062.40 | 195.54 | 5230.33 | 1.77 | 993.84 | 226.15 | 4521.79 | 0.24 | 16% | -68.56 | -6% | 30.61 | 16% | -708.54 | -14% | | 5 | 2048 | 512 | 0.41 | 256.67 | 211.48 | 12101.57 | 0.50 | 253.09 | 253.98 | 10075.56 | 0.09 | 22% | -3.58 | -1% | 42.50 | 20% | -2026.01 | -17% | | 6 | 2048 | 512 | 0.48 | 255.56 | 244.88 | 12540.07 | 0.57 | 235.94 | 294.06 | 10442.02 | 0.09 | 19% | -19.62 | -8% | 49.18 | 20% | -2098.05 | -17% | | 7 | 2048 | 512 | 0.49 | 1018.78 | 252.18 | 14206.53 | 0.58 | 1019.99 | 295.39 | 12127.83 | 0.09 | 18% | 1.21 | 0% | 43.21 | 17% | -2078.70 | -15% | | 8 | 2048 | 512 | 0.55 | 1040.09 | 279.59 | 14643.40 | 0.64 | 1058.69 | 325.90 | 12562.18 | 0.09 | 16% | 18.60 | 2% | 46.31 | 17% | -2081.22 | -14% | | 5 | 4096 | 512 | 0.30 | 1698.20 | 152.78 | 16748.79 | 0.34 | 1751.38 | 173.28 | 14766.71 | 0.04 | 13% | 53.18 | 3% | 20.50 | 13% | -1982.08 | -12% | | 6 | 4096 | 512 | 0.33 | 1769.82 | 171.31 | 17923.55 | 0.38 | 1687.34 | 193.27 | 15886.16 | 0.05 | 15% | -82.48 | -5% | 21.96 | 13% | -2037.39 | -11% | | 7 | 4096 | 512 | 0.37 | 1648.96 | 189.37 | 18914.07 | 0.41 | 1868.01 | 209.70 | 17080.13 | 0.04 | 11% | 219.05 | 13% | 20.33 | 11% | -1833.94 | -10% | | 8 | 4096 | 512 | 0.40 | 1828.76 | 203.68 | 20095.59 | 0.44 | 1821.36 | 226.98 | 18032.25 | 0.04 | 10% | -7.40 | 0% | 23.30 | 11% | -2063.34 | -10% | | 5 | 8192 | 512 | 0.19 | 3829.88 | 99.69 | 25664.15 | 0.21 | 3663.23 | 107.66 | 23763.46 | 0.02 | 11% | -166.65 | -4% | 7.97 | 8% | -1900.69 | -7% | | 6 | 8192 | 512 | 0.21 | 3859.11 | 107.91 | 28448.65 | 0.23 | 3844.42 | 116.08 | 26443.94 | 0.02 | 10% | -14.69 | 0% | 8.17 | 8% | -2004.71 | -7% | | 7 | 8192 | 512 | 0.22 | 4007.26 | 114.38 | 31307.53 | 0.24 | 3968.81 | 122.24 | 29293.29 | 0.02 | 9% | -38.45 | -1% | 7.86 | 7% | -2014.24 | -6% | | 8 | 8192 | 512 | 0.23 | 3929.31 | 119.34 | 34289.29 | 0.25 | 4131.67 | 126.76 | 32281.71 | 0.02 | 9% | 202.36 | 5% | 7.42 | 6% | -2007.58 | -6% | | 5 | 1024 | 2048 | 0.11 | 272.15 | 217.61 | 47052.99 | 0.13 | 257.10 | 264.88 | 38655.63 | 0.02 | 18% | -15.05 | -6% | 47.27 | 22% | -8397.36 | -18% | | 6 | 1024 | 2048 | 0.12 | 319.32 | 252.43 | 48674.07 | 0.15 | 336.41 | 306.06 | 40145.27 | 0.03 | 25% | 17.09 | 5% | 53.63 | 21% | -8528.80 | -18% | | 7 | 1024 | 2048 | 0.14 | 323.50 | 291.21 | 49222.81 | 0.17 | 345.86 | 352.14 | 40706.31 | 0.03 | 21% | 22.36 | 7% | 60.93 | 21% | -8516.50 | -17% | | 8 | 1024 | 2048 | 0.16 | 371.08 | 329.52 | 49714.14 | 0.19 | 345.11 | 398.09 | 41150.64 | 0.03 | 19% | -25.97 | -7% | 68.57 | 21% | -8563.50 | -17% | | 5 | 512 | 4096 | 0.05 | 182.87 | 212.34 | 96449.01 | 0.06 | 183.89 | 256.32 | 79898.13 | 0.01 | 20% | 1.02 | 1% | 43.98 | 21% | -16550.88 | -17% | | 6 | 512 | 4096 | 0.06 | 211.20 | 245.59 | 100068.20 | 0.07 | 208.61 | 295.82 | 83077.61 | 0.01 | 17% | -2.59 | -1% | 50.23 | 20% | -16990.61 | -17% | | 7 | 512 | 4096 | 0.07 | 235.87 | 282.21 | 101596.10 | 0.08 | 234.92 | 338.94 | 84590.28 | 0.01 | 14% | -0.95 | 0% | 56.73 | 20% | -17005.85 | -17% | | 8 | 512 | 4096 | 0.08 | 253.05 | 318.55 | 102863.60 | 0.09 | 248.26 | 381.88 | 85803.16 | 0.01 | 13% | -4.79 | -2% | 63.33 | 20% | -17060.44 | -17% | | 5 | 512 | 8192 | 0.02 | 185.16 | 190.11 | 215451.10 | 0.03 | 183.53 | 223.90 | 182940.48 | 0.01 | 50% | -1.63 | -1% | 33.79 | 18% | -32510.62 | -15% | | 6 | 512 | 8192 | 0.03 | 212.97 | 216.41 | 227118.40 | 0.03 | 212.17 | 253.98 | 193528.35 | 0.00 | 0% | -0.80 | 0% | 37.57 | 17% | -33590.02 | -15% | | 7 | 512 | 8192 | 0.03 | 236.14 | 244.39 | 234637.00 | 0.03 | 235.02 | 285.46 | 200879.94 | 0.00 | 0% | -1.12 | 0% | 41.07 | 17% | -33757.10 | -14% | | 8 | 512 | 8192 | 0.03 | 256.54 | 271.29 | 241564.50 | 0.04 | 255.74 | 315.74 | 207557.14 | 0.01 | 33% | -0.80 | 0% | 44.45 | 16% | -34007.34 | -14% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/45559", "createdAt": "2026-06-14T03:23:39Z", "updatedAt": "2026-08-12T23:01:35Z", "timestamp": "2026-08-12T23:01:35Z", "metrics": { "reactions": 3, "comments": 3 }, "labels": [ "rocm" ], "author": "keneoneth", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:543b76041c84073f9185", "signalId": "github:vllm-project/vllm:pull_request:52054", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52054", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][V1] Chunk prompt-logprobs logits to bound the activation peak", "text": "## Purpose `_get_prompt_logprobs_dict` materializes the full `[num_scheduled_tokens, vocab_size]` logits tensor in one shot and then upcasts it to float32 via `log_softmax`. Both tensors are live simultaneously, so the transient peak is roughly: ``` num_scheduled_tokens * vocab_size * (2 bytes bf16 + 4 bytes fp32) ``` At 8192 scheduled tokens and a 152k vocabulary that is **~7.5 GiB**. That peak is never observed by memory profiling. `profile_run` calls `_dummy_sampler_run`, which constructs its `SamplingMetadata` with: https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu_model_runner.py#L6362 ```python max_num_logprobs=None, ``` so the logprobs path is not exercised and the float32 score tensor is never allocated during profiling. KV cache capacity is sized against a peak that excludes prompt logprobs, and the allocation has to come out of whatever is left at serving time. This is exactly the mechanism @robertgshaw2-redhat described in #5907: > When determining the KV cache size, we calculate peak memory running a long prefill *without logprobs*. If a prompt requests many logprobs, however, this is an additional source of memory usage which is not considered during warmup and can cause OOM because we have nothing in scheduler to prevent this. ## Approach Rather than restructure profiling, this bounds the allocation directly — the \"chunked logits processing\" approach proposed by @tjohnson31415 in #5907 and endorsed in that thread as \"the ideal solution\". `compute_logits` and the score computation now run over row chunks of `VLLM_PROMPT_LOGPROBS_CHUNK_SIZE` (default 1024). Each chunk's results are copied into the existing per-request CPU tensors before the next chunk is materialized, so the peak becomes independent of `max_num_batched_tokens`. Behaviour is unchanged: same logits, same scores, same destination rows, same `logprobs_mode` handling. Only the number of rows materialized at once differs. The default of 1024 leaves the common case (short prompts, small `max_num_batched_tokens`) at a single chunk, so there is no change in behaviour or performance for most deployments. ## Why this is not duplicating an existing PR Checks run: ``` gh issue view 5907 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"prompt_logprobs memory\" gh pr list --repo vllm-project/vllm --state open --search \"prompt logprobs profile_run\" gh pr list --repo vllm-project/vllm --state open --search \"prompt_logprobs OOM\" gh pr list --repo vllm-project/vllm --state merged --search \"5907 in:body\" ``` - **#5907** — the tracking issue for this exact defect. Closed as **stale**, never fixed. - **#5355** (\"[Bugfix] Take the VRAM usage of prompt_logprobs into account\") — the referenced fix attempt. **Closed, unmerged.** - No merged PR addresses it. Verified the behaviour is still present on `main` at `23f360edaa`. - The four open `prompt_logprobs` PRs are unrelated concerns: **#51846** (`prompt_logprobs=0` admission guards / chat echo), **#49622** (`kv_sharing_fast_prefill` incompatibility validation), **#42245** (non-determinism with prefix caching), **#51953** (DiffusionGemma tiled logits projection — a different model's sampler path). ## Test commands run and results New test file `tests/v1/worker/test_gpu_prompt_logprobs_chunking.py`, modelled on the existing `tests/v1/worker/test_gpu_rejection_sampler_chunking.py` convention: ``` python -m pytest tests/v1/worker/test_gpu_prompt_logprobs_chunking.py -v 9 passed ``` Coverage — chunked output must match a single-chunk reference exactly, across `chunk_size` ∈ {1, 3, 4, 1024} (degenerate, short-final-chunk, evenly-dividing, single-chunk) and `logprobs_mode` ∈ {`raw_logprobs`, `raw_logits`}, plus an assertion that no single `compute_logits` call exceeds the configured chunk size and that every row is covered exactly once. **Verified by negation:** injecting the classic chunking bug (destination slice ignoring the chunk offset, `slice(start_idx, start_idx + (end - pos))`) makes 6 of the 9 tests fail with a 100%-mismatched tensor. The 3 that still pass are the single-chunk cases, where the bug is unreachable. So the tests genuinely pin the row mapping rather than passing vacuously. The tests are CPU-runnable. They disable `PIN_MEMORY` via an autouse fixture because `async_tensor_h2d` pins the source buffer, which requires a CUDA allocator; the chunking logic under test is unaffected. ## AI assistance This change was developed with AI assistance. The problem analysis, the upstream-vs-fork verification, the patch and the tests were produced with an AI coding agent; the duplicate-work checks above were run as part of that process. A human submitter *has* reviewed every changed line. The chunking approach is a port of a fix that has been running in production in a downstream fork, where it resolved this OOM.", "url": "https://github.com/vllm-project/vllm/pull/52054", "createdAt": "2026-08-12T21:45:27Z", "updatedAt": "2026-08-12T22:59:02Z", "timestamp": "2026-08-12T22:59:02Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "mrv1-only" ], "author": "malaiwah", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:292dba3ef42e5064138b", "signalId": "github:vllm-project/vllm:pull_request:51575", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51575", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MRV2] Respect dynamic K=0 from scheduler in AutoRegressiveSpeculator", "text": "## Problem When Dynamic Speculative Decoding (DSD) selects K=0 (no speculation) at higher batch sizes, the `AutoRegressiveSpeculator` ignores the dynamic K and always runs the full K=max pipeline — wasting GPU compute on decode steps whose draft tokens are never used. The scheduler correctly computes dynamic K and places it in `SchedulerOutput.num_spec_tokens_to_schedule` (existing field on main), but the model runner and speculator never read it. Fixes #51510. ## Solution Propagate `num_spec_tokens_to_schedule` from `SchedulerOutput` through `ExecuteModelState` into `sample_tokens()`, and pass it to `speculator.propose()`. When K=0, the speculator runs prefill (to keep draft KV cache in sync with the target) but skips all decode steps, returning an empty-width draft token tensor. ## Changes (6 files, +30/-2) | File | Change | |------|--------| | `model_runner.py` | Add `num_spec_tokens_to_schedule` field to `ExecuteModelState` (default `None`). Populate from `scheduler_output` in `execute_model()`. Read in `sample_tokens()` and pass to `propose()`. | | `spec_decode/autoregressive/speculator.py` | Accept `num_speculative_tokens` param in `propose()`. When `==0`, early-exit after prefill, return `draft_tokens[:num_reqs, :0]`. | | `spec_decode/speculator.py` | Signature compatibility — base `BaseSpeculator.propose()` accepts the new param. | | `spec_decode/multi_module_mtp/speculator.py` | Signature compatibility. | | `spec_decode/dflash/speculator.py` | Signature compatibility. | | `test_gpu_model_runner_v2_eplb.py` | Add `num_spec_tokens_to_schedule=None` to test fixture. | ## Stale draft token fix When the speculator returns fewer tokens than configured (K=0 → width 0), use `[:active_width]` slice on `draft_tokens` passed to `DraftTokensHandler`. Without this, stale columns from the persistent buffer create phantom draft slots. ## Benchmark 2×RTX 5090, ThinkingCap-Qwen3.6-27B-FP8, MRV2 (FULL_AND_PIECEWISE CG), TP=2, DSD `[[1,2,2],[3,16,0]]` (K=0 active at batch ≥3): | Sessions | Before fix | After fix | Δ | No-spec baseline | |----------|-----------|-----------|---|-----------------| | 1* | 107 t/s | 108 t/s | +1% | 107 t/s | | **4** | **193 t/s** | **211 t/s** | **+9%** | **209 t/s** | | 8 | 364 t/s | 399 t/s | +10% | 393 t/s | \\*At 1 session K=2 is active (no K=0 selected), so no difference expected. Before the fix, each K=0 step wasted ~7ms on two unnecessary draft model decode passes (measured via instrumented per-step timing). After the fix, K=0 throughput matches the no-spec baseline (within ±2%). ## Notes - `num_speculative_tokens=None` (the default) preserves prior behavior — non-DSD deployments are unaffected. - Only `AutoRegressiveSpeculator` acts on the parameter; other speculator types accept it in their signature for compatibility but ignore the value. - The `draft_tokens` `[:active_width]` slice is safe for K=0 (`:0` is a valid empty slice). ## AI Assistance Disclosure AI tools were used in the development of this PR. All changes were reviewed and tested by the human submitter (Greg Weyer).", "url": "https://github.com/vllm-project/vllm/pull/51575", "createdAt": "2026-08-09T18:49:09Z", "updatedAt": "2026-08-12T22:55:27Z", "timestamp": "2026-08-12T22:55:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "needs-rebase", "mrv2" ], "author": "Suppressor72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0d555bc946ffdb4f24e4", "signalId": "github:vllm-project/vllm:pull_request:51742", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51742", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode] Fix KV cache group explosion for hybrid models", "text": "## Purpose When profiling Qwen/Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash, we found the KV cache group_size become 1. This increases build metadata overhead. We found the target model Qwen/Qwen3.6-35B-A3B layer buckets are [30 mamba, 10 full]. The DFlash drafter z-lab/Qwen3.6-35B-A3B-DFlash adds [5 sliding, 1 full], the added 1 full attention layer that can't merge with the target's 10 full attention layers. The current logic used `min` as the group size [here](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/core/kv_cache_utils.py#L1245-L1246). `group_size = min([30, 10, 5, 1]) = 1` makes it to split into 46 groups of 1 layer each. So this PR update `_get_kv_cache_groups_uniform_page_size` to exclude buckets with 1 layer from the `min` computation. Result: layers [30 mamba, 10 full, 5 sliding, 1 full] become 10 groups instead of 46 groups. e2e output throughput improves 102% at batch size 1 on H200. ``` Group 0: MambaSpec, layers=5 Group 1: MambaSpec, layers=5 Group 2: MambaSpec, layers=5 Group 3: MambaSpec, layers=5 Group 4: MambaSpec, layers=5 Group 5: MambaSpec, layers=5 Group 6: FullAttentionSpec, layers=5 Group 7: FullAttentionSpec, layers=5 Group 8: SlidingWindowSpec, layers=5 Group 9: FullAttentionSpec, layers=1 ``` Note: This PR does not cause regression for other models. ``` ┌─────────────────────────────────────────┬───────────────────────────┬────────────────────────────┐ │ Model │ Before │ After │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B │ 4 groups (10 layers each) │ 4 groups (unchanged) │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B + DFlash │ 46 groups (1 layer each) │ 10 groups │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ Qwen3.6-35B-A3B + Eagle │ 41 groups (1 layer each) │ 5 groups │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ gemma-4-26B-A4B-it (5:1 sw/full) │ 6 groups │ 6 groups (unchanged) │ ├─────────────────────────────────────────┼───────────────────────────┼────────────────────────────┤ │ gpt-oss-20b (12 sw + 13 full) │ 2 groups │ 2 groups (unchanged) │ └─────────────────────────────────────────┴───────────────────────────┴────────────────────────────┘ ``` ## Test Plan ``` pytest -s -v tests/v1/core/test_kv_cache_utils.py ``` ## Test Result Unit tests passed. ## Profiling Profiled Qwen/Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash on H200. Main: \"Screenshot PR: \"Screenshot Main: `build_attn_metadata` takes around 31ms, because [attn_metadata_builder.build](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/worker/gpu/attn_utils.py#L583) are called 46 times. PR: `build_attn_metadata` takes around 6ms, because [attn_metadata_builder.build](https://github.com/vllm-project/vllm/blob/v0.27.0/vllm/v1/worker/gpu/attn_utils.py#L583) are called 10 times. ## Benchmark * **DFlash** ``` vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"z-lab/Qwen3.6-35B-A3B-DFlash\",\"method\":\"dflash\",\"num_speculative_tokens\":8}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 97.79 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.61 Output token throughput (tok/s): 184.07 Peak output token throughput (tok/s): 60.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 351.88 ---------------Time to First Token---------------- Mean TTFT (ms): 138.20 Median TTFT (ms): 132.91 P99 TTFT (ms): 723.12 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.99 Median TPOT (ms): 4.71 P99 TPOT (ms): 16.71 ---------------Inter-token Latency---------------- Mean ITL (ms): 17.07 Median ITL (ms): 17.06 P99 ITL (ms): 17.78 ---------------Speculative Decoding--------------- Acceptance rate (%): 30.62 Acceptance length: 3.45 Drafts: 5243 Draft tokens: 41944 Accepted tokens: 12844 Per-position acceptance (%): Position 0: 66.83 Position 1: 49.11 Position 2: 36.62 Position 3: 27.90 Position 4: 22.09 Position 5: 17.17 Position 6: 13.98 Position 7: 11.27 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 208.46 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 4.61 Output token throughput (tok/s): 1381.58 Peak output token throughput (tok/s): 636.00 Peak concurrent requests: 24.00 Total token throughput (tok/s): 2473.16 ---------------Time to First Token---------------- Mean TTFT (ms): 205.39 Median TTFT (ms): 192.83 P99 TTFT (ms): 367.00 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.87 Median TPOT (ms): 10.59 P99 TPOT (ms): 36.67 ---------------Inter-token Latency---------------- Mean ITL (ms): 37.74 Median ITL (ms): 22.07 P99 ITL (ms): 155.20 ---------------Speculative Decoding--------------- Acceptance rate (%): 31.23 Acceptance length: 3.50 Drafts: 82664 Draft tokens: 661312 Accepted tokens: 206538 Per-position acceptance (%): Position 0: 70.86 Position 1: 49.80 Position 2: 36.20 Position 3: 27.68 Position 4: 21.82 Position 5: 17.57 Position 6: 14.30 Position 7: 11.62 ================================================== ``` PR: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 48.40 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 1.24 Output token throughput (tok/s): 371.88 Peak output token throughput (tok/s): 122.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 710.91 ---------------Time to First Token---------------- Mean TTFT (ms): 95.85 Median TTFT (ms): 117.33 P99 TTFT (ms): 126.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.38 Median TPOT (ms): 2.19 P99 TPOT (ms): 7.99 ---------------Inter-token Latency---------------- Mean ITL (ms): 8.23 Median ITL (ms): 8.26 P99 ITL (ms): 9.22 ---------------Speculative Decoding--------------- Acceptance rate (%): 31.16 Acceptance length: 3.49 Drafts: 5181 Draft tokens: 41448 Accepted tokens: 12916 Per-position acceptance (%): Position 0: 68.06 Position 1: 49.53 Position 2: 36.98 Position 3: 28.37 Position 4: 22.33 Position 5: 17.76 Position 6: 14.32 Position 7: 11.95 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 175.64 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 5.47 Output token throughput (tok/s): 1639.68 Peak output token throughput (tok/s): 652.00 Peak concurrent requests: 26.00 Total token throughput (tok/s): 2935.18 ---------------Time to First Token---------------- Mean TTFT (ms): 154.56 Median TTFT (ms): 153.48 P99 TTFT (ms): 299.94 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.23 Median TPOT (ms): 8.79 P99 TPOT (ms): 30.92 ---------------Inter-token Latency---------------- Mean ITL (ms): 31.33 Median ITL (ms): 21.56 P99 ITL (ms): 128.12 ---------------Speculative Decoding--------------- Acceptance rate (%): 30.26 Acceptance length: 3.42 Drafts: 84594 Draft tokens: 676752 Accepted tokens: 204774 Per-position acceptance (%): Position 0: 69.47 Position 1: 48.27 Position 2: 34.92 Position 3: 26.63 Position 4: 21.04 Position 5: 16.87 Position 6: 13.70 Position 7: 11.16 ================================================== ``` e2e output throughput improves 102% at batch size 1, improves 18% at batch size 16. * **Eagle** ``` vllm serve Qwen/Qwen3.5-35B-A3B \\ --tensor-parallel-size 1 \\ --max-num-seqs 16 \\ --speculative-config '{\"model\":\"jiapingW/Qwen3.5-35B-A3B-Eagle3-Specforge\",\"method\":\"eagle3\",\"num_speculative_tokens\":4}' \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model Qwen/Qwen3.5-35B-A3B \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 146.45 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 0.41 Output token throughput (tok/s): 122.91 Peak output token throughput (tok/s): 48.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 234.96 ---------------Time to First Token---------------- Mean TTFT (ms): 115.48 Median TTFT (ms): 137.76 P99 TTFT (ms): 150.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.78 Median TPOT (ms): 7.51 P99 TPOT (ms): 14.46 ---------------Inter-token Latency---------------- Mean ITL (ms): 21.12 Median ITL (ms): 21.16 P99 ITL (ms): 21.72 ---------------Speculative Decoding--------------- Acceptance rate (%): 43.24 Acceptance length: 2.73 Drafts: 6605 Draft tokens: 26420 Accepted tokens: 11424 Per-position acceptance (%): Position 0: 67.51 Position 1: 46.98 Position 2: 33.72 Position 3: 24.75 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 305.64 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 3.14 Output token throughput (tok/s): 942.27 Peak output token throughput (tok/s): 672.00 Peak concurrent requests: 23.00 Total token throughput (tok/s): 1686.76 ---------------Time to First Token---------------- Mean TTFT (ms): 201.53 Median TTFT (ms): 202.58 P99 TTFT (ms): 345.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.27 Median TPOT (ms): 15.30 P99 TPOT (ms): 29.23 ---------------Inter-token Latency---------------- Mean ITL (ms): 34.26 Median ITL (ms): 24.06 P99 ITL (ms): 157.68 ---------------Speculative Decoding--------------- Acceptance rate (%): 27.84 Acceptance length: 2.11 Drafts: 136316 Draft tokens: 545264 Accepted tokens: 151791 Per-position acceptance (%): Position 0: 50.02 Position 1: 29.58 Position 2: 18.92 Position 3: 12.84 ================================================== ``` PR: 1) concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 54.89 Total input tokens: 16410 Total generated tokens: 18000 Request throughput (req/s): 1.09 Output token throughput (tok/s): 327.94 Peak output token throughput (tok/s): 130.00 Peak concurrent requests: 3.00 Total token throughput (tok/s): 626.91 ---------------Time to First Token---------------- Mean TTFT (ms): 88.68 Median TTFT (ms): 109.96 P99 TTFT (ms): 123.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.76 Median TPOT (ms): 2.75 P99 TPOT (ms): 4.20 ---------------Inter-token Latency---------------- Mean ITL (ms): 7.63 Median ITL (ms): 7.65 P99 ITL (ms): 8.03 ---------------Speculative Decoding--------------- Acceptance rate (%): 44.37 Acceptance length: 2.77 Drafts: 6495 Draft tokens: 25980 Accepted tokens: 11528 Per-position acceptance (%): Position 0: 69.75 Position 1: 48.21 Position 2: 34.26 Position 3: 25.28 ================================================== ``` 2) concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 188.81 Total input tokens: 227546 Total generated tokens: 288000 Request throughput (req/s): 5.08 Output token throughput (tok/s): 1525.38 Peak output token throughput (tok/s): 763.00 Peak concurrent requests: 25.00 Total token throughput (tok/s): 2730.56 ---------------Time to First Token---------------- Mean TTFT (ms): 139.44 Median TTFT (ms): 142.08 P99 TTFT (ms): 295.77 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.02 Median TPOT (ms): 9.90 P99 TPOT (ms): 18.75 ---------------Inter-token Latency---------------- Mean ITL (ms): 26.48 Median ITL (ms): 18.78 P99 ITL (ms): 119.41 ---------------Speculative Decoding--------------- Acceptance rate (%): 41.36 Acceptance length: 2.65 Drafts: 108609 Draft tokens: 434436 Accepted tokens: 179686 Per-position acceptance (%): Position 0: 66.34 Position 1: 45.16 Position 2: 31.29 Position 3: 22.65 ================================================== ``` e2e output throughput improves 166% at batch size 1, improves 61% at batch size 16. ## Accuracy Testing * **DFlash** ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.6-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3472|± |0.0131| | | |strict-match | 5|exact_match|↑ |0.3268|± |0.0129| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3594|± |0.0132| | | |strict-match | 5|exact_match|↑ |0.3389|± |0.0130| ``` * **Eagle** ``` python3 -m lm_eval --model local-completions \\ --model_args model=Qwen/Qwen3.5-35B-A3B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.0121|± | 0.003| | | |strict-match | 5|exact_match|↑ |0.0000|± | 0.000| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8605|± |0.0095| | | |strict-match | 5|exact_match|↑ |0.8491|± |0.0099| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51742", "createdAt": "2026-08-11T00:16:11Z", "updatedAt": "2026-08-12T22:39:41Z", "timestamp": "2026-08-12T22:39:41Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "speculative-decoding" ], "author": "xyang16", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e831cdb95b66e296948b", "signalId": "github:vllm-project/vllm:pull_request:51932", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Move context_lens compute into prefill path for Minimax", "text": "## Purpose Move `compute_num_computed_tokens()` from the top of the method into the `if num_prefills > 0` branch because it's only used in prefill. Since context_lens tensor is not used in decode path, this change avoids the tensor computed and discarded in decode path. Similar as #51913 ## Profiling Main: \"Screenshot PR: \"Screenshot Main: `compute_num_computed_tokens` called in MiniMaxM3IndexerTritonMetadataBuilder.build() in decode path, takes extra 74µs. PR: `compute_num_computed_tokens` not called. ## Benchmark ``` vllm serve MiniMaxAI/MiniMax-M3 \\ --tensor-parallel-size 8 \\ --max-num-seqs 16 \\ --block-size 128 \\ --no-enable-prefix-caching ``` ``` vllm bench serve \\ --model MiniMaxAI/MiniMax-M3 \\ --dataset-name sharegpt \\ --dataset-path /tmp/ShareGPT_V3_unfiltered_cleaned_split.json \\ --sharegpt-output-len 300 \\ --num-prompts ${num_prompts} \\ --max-concurrency ${concurrency} \\ --num-warmups 200 \\ --ignore-eos ``` Main: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 119.30 Total input tokens: 15421 Total generated tokens: 18000 Request throughput (req/s): 0.50 Output token throughput (tok/s): 150.89 Peak output token throughput (tok/s): 158.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 280.15 ---------------Time to First Token---------------- Mean TTFT (ms): 85.10 Median TTFT (ms): 114.17 P99 TTFT (ms): 133.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.36 Median TPOT (ms): 6.36 P99 TPOT (ms): 6.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 6.36 Median ITL (ms): 6.36 P99 ITL (ms): 6.72 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 300.81 Total input tokens: 216233 Total generated tokens: 288000 Request throughput (req/s): 3.19 Output token throughput (tok/s): 957.42 Peak output token throughput (tok/s): 1072.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 1676.25 ---------------Time to First Token---------------- Mean TTFT (ms): 303.93 Median TTFT (ms): 293.91 P99 TTFT (ms): 401.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.75 Median TPOT (ms): 15.72 P99 TPOT (ms): 16.54 ---------------Inter-token Latency---------------- Mean ITL (ms): 15.75 Median ITL (ms): 15.39 P99 ITL (ms): 16.67 ================================================== ``` PR: * concurrency 1 ``` ============ Serving Benchmark Result ============ Successful requests: 60 Failed requests: 0 Maximum request concurrency: 1 Benchmark duration (s): 119.22 Total input tokens: 15421 Total generated tokens: 18000 Request throughput (req/s): 0.50 Output token throughput (tok/s): 150.98 Peak output token throughput (tok/s): 158.00 Peak concurrent requests: 2.00 Total token throughput (tok/s): 280.32 ---------------Time to First Token---------------- Mean TTFT (ms): 84.08 Median TTFT (ms): 113.42 P99 TTFT (ms): 130.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.36 Median TPOT (ms): 6.36 P99 TPOT (ms): 6.38 ---------------Inter-token Latency---------------- Mean ITL (ms): 6.36 Median ITL (ms): 6.36 P99 ITL (ms): 6.74 ================================================== ``` * concurrency 16 ``` ============ Serving Benchmark Result ============ Successful requests: 960 Failed requests: 0 Maximum request concurrency: 16 Benchmark duration (s): 296.84 Total input tokens: 216233 Total generated tokens: 288000 Request throughput (req/s): 3.23 Output token throughput (tok/s): 970.22 Peak output token throughput (tok/s): 1057.00 Peak concurrent requests: 32.00 Total token throughput (tok/s): 1698.67 ---------------Time to First Token---------------- Mean TTFT (ms): 247.74 Median TTFT (ms): 260.92 P99 TTFT (ms): 285.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.71 Median TPOT (ms): 15.72 P99 TPOT (ms): 16.26 ---------------Inter-token Latency---------------- Mean ITL (ms): 15.71 Median ITL (ms): 15.43 P99 ITL (ms): 16.74 ================================================== ``` ## Accuracy Testing ``` python3 -m lm_eval --model local-completions \\ --model_args model=MiniMaxAI/MiniMax-M3,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16 \\ --tasks gsm8k ``` Main: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9121|± |0.0078| | | |strict-match | 5|exact_match|↑ |0.9113|± |0.0078| ``` PR: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9121|± |0.0078| | | |strict-match | 5|exact_match|↑ |0.9121|± |0.0078| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51932", "createdAt": "2026-08-12T06:33:53Z", "updatedAt": "2026-08-12T22:38:43Z", "timestamp": "2026-08-12T22:38:43Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [], "author": "xyang16", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4db6639f8f84327a3ae0", "signalId": "github:vllm-project/vllm:pull_request:50514", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50514", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Feat/spec decode under pipeline parallel", "text": "## Purpose EAGLE3-style speculative decoding (`eagle3` / `dflash` / `dspark`) is rejected outright when pipeline parallelism is enabled: ``` ValueError: with pipeline parallel is not supported. ``` The drafter runs on the last PP rank, but it consumes auxiliary hidden states tapped from target layers that may live on earlier stages, and those tensors were dropped at the stage boundary. This PR forwards them and lifts the guard. Lifting the guard alone is not sufficient. The code behind it had never executed, so it hid a stack of latent defects in the shared PP path — all of them reachable only once spec decode and PP run together, which means they were unobservable rather than latent-but-live: 1. **Draft tokens never reached the non-last ranks.** Only the last rank runs the drafter, so `req_states.draft_tokens` is written nowhere else — yet the *first* rank owns `embed_tokens` and builds the embeddings for the whole pipeline. It was embedding the scheduler's `PLACEHOLDER_TOKEN_ID` (`-1`) into the draft slots, so the last rank verified real proposals against logits computed from placeholders. Silent corruption. Fixed by broadcasting the proposals from the last rank on the existing deferred slot, which has identical production and consumption timing. 2. **`compute_need_sampled_mask` ended the broadcast early.** The scheduler advances `num_computed_tokens` by the full scheduled width up front and rolls the rejected part back in `update_from_output`, which under PP lands *after* the next batch is scheduled. Reading the inflated count marked requests as finishing up to `num_speculative_tokens` early, after which the last rank stopped broadcasting and the other ranks' `last_sampled_tokens` froze, repeating a stale token. 3. **The sampled-token broadcast width could mismatch.** `receive()` always allocates `max_sample_len` columns while `broadcast()` sent whatever width the sampler produced, and the non-spec path returns width 1. NCCL does not diagnose the count mismatch: the root completes and the peer hangs until the watchdog fires, so any PP + spec-decode run deadlocked on its first prefill. Fixed by padding the send side; `post_update` stops at `num_sampled`, so the pad columns are never read. ### Design Aux-state forwarding is added generically on `EagleModelMixin`. A stage packs its own taps plus those inherited from upstream into the `IntermediateTensors` payload, and sizes its recv buffers from the same rule — a tap `a` is upstream iff `a <= start_layer` — so send and recv counts agree with no negotiation. Models opt in with `supports_aux_hidden_states_over_pp`. The previous blanket error is retained for those that do not, now raised with the offending model named. Kimi-K3 opts in; no other model does, so **EAGLE3 and dflash under PP remain gated and untested.** Two smaller changes ride along, both no-ops for configurations that worked before (spec decode under PP > 1 was rejected outright): the DSpark drafter gets a real vocab embedding on the last stage, where the target's is a `PPMissingLayer`; and the draft parallel config stops inheriting the target's PP size, since the drafter is instantiated only on the last stage and is never itself pipelined. ### Scope: capped at `pipeline_parallel_size <= 2` `GPUModelRunner.load_model` raises `NotImplementedError` above 2. This reflects validation coverage, not a known defect. The forwarding rule is size-agnostic by construction and the accounting is unit-tested up to pp=8. But pp>2 is the first topology with a **middle** stage, which must both adopt upstream taps and contribute its own to the same payload, and no such run has happened on hardware. The conservative default is deliberate because this feature does not fail loudly: a drafter fed mis-ordered or missing taps still emits syntactically valid proposals that simply get rejected more often, so the only symptom is a depressed acceptance rate. Lifting the cap should require an acceptance-rate comparison at pp>2, not just a successful boot. Happy to drop the cap if maintainers would rather have the untested-but-general path available. ### Note on commit 1 The first commit is an independent bugfix — `MambaHybridModelState.postprocess_state` feeds an int32 `idx_mapping` to `index_fill_`, which accepts only int64: ``` IndexError: index_fill_(): Expected dtype int64 for index. ``` **This is reachable on main today with no speculative decoding involved.** Its only caller is the non-last-PP-rank path in `GPUModelRunner.execute_model`, so any Mamba-hybrid model (Jamba, Falcon-H1, Nemotron-H, Qwen3-Next, MiniMax, Kimi Linear) crashes under `pipeline_parallel_size >= 2` with a chunked-prefill batch. It is included here because Kimi-K3 is a hybrid model and hits it under PP. Glad to split it into its own PR if preferred. ## Test Plan **Unit.** A CPU test for the aux accounting invariant, no distributed init required: ```bash pytest tests/v1/worker/test_eagle3_aux_hidden_states_pp.py ``` 12 cases over pp ∈ {1,2,3,4,6,8}, exercising the real `EagleModelMixin` and real `get_pp_indices`. Covers tap ordering and the boundary-tap double-count hazard (a tap landing exactly on `start_layer` must be counted as upstream by exactly one stage). The parametrization intentionally runs past the pp<=2 cap, since the accounting is general and this is what a future enablement would build on. **End-to-end.** Kimi-K3 (93 layers, DSpark drafter, `num_speculative_tokens=7`) at TP8 × PP2 on 2× 8×B200 with EFA. Two checks: 1. *Correctness*: greedy decode compared token-for-token against the identical topology with speculative decoding disabled. Speculative decoding is supposed to be output-neutral, so any divergence is a bug. 2. *Acceptance rate*: the sensitive detector. Corruption of aux taps crossing the PP boundary degrades acceptance before it produces visibly wrong text. Throughput measured on 128 fixed GSM8K prompts, same seed, concurrency 1 → 32. ## Test Result Unit: 12 passed. `ruff check` and `ruff format --check` clean. **Correctness**: greedy output is token-for-token identical to the same topology without speculative decoding. **Acceptance**: 2.32 of 7 draft tokens per step. **Throughput** (output tok/s, GSM8K, 128 prompts): | Concurrency | No spec | Spec (PP2) | Speedup | |---:|---:|---:|---:| | 1 | 77.30 | 174.52 | 2.26× | | 4 | 255.77 | 461.53 | 1.80× | | 8 | 423.86 | 714.01 | 1.68× | | 32 | 1138.69 | 1584.07 | 1.39× |", "url": "https://github.com/vllm-project/vllm/pull/50514", "createdAt": "2026-07-31T03:29:39Z", "updatedAt": "2026-08-12T22:36:16Z", "timestamp": "2026-08-12T22:36:16Z", "metrics": { "reactions": 2, "comments": 23 }, "labels": [ "speculative-decoding", "ready", "needs-rebase", "ci/build", "llama", "qwen", "deepseek", "cpu", "nvidia", "mrv2", "kimi", "k3" ], "author": "yongqinwang-cmd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9d21113eca07254dde4b", "signalId": "github:vllm-project/vllm:pull_request:51952", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51952", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "NIXL: Use int32 array for indices to avoid intermediate conversion", "text": "## Purpose As documented NIXL accepts KV indices as `int32` array, while vLLM creates and passes `int64` array. This results in extra allocation and conversion on NIXL side, which is redundant. Removing this conversion + adding zero-copy on NIXL side improves performance https://github.com/ai-dynamo/nixl/pull/1718 ## Test Result \"pareto_compare_nixl_zci_vs_concurrency_100538\"", "url": "https://github.com/vllm-project/vllm/pull/51952", "createdAt": "2026-08-12T08:15:50Z", "updatedAt": "2026-08-12T22:33:28Z", "timestamp": "2026-08-12T22:33:28Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "kv-connector" ], "author": "iyastreb", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a2ee73725c34f0b9a462", "signalId": "github:vllm-project/vllm:pull_request:52056", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52056", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build][AMD] Update triton_kernels pin for RDNA MXFP4 matmul_ogs config", "text": "Advances the ROCm triton_kernels pin to 74e4569a70, which backports upstream Triton PR #10810 (\"[AMD][kernels] Improve MXFP4 matmul_ogs config on RDNA\"). Avoids register spilling in the MXFP4 MoE GEMMs on RDNA. ## Purpose Advances the ROCm `triton_kernels` pin in `cmake/external_projects/triton_kernels.cmake` from `0f380657` to `74e4569a70`, picking up the RDNA MXFP4 `matmul_ogs` configuration change. - Upstream Triton PR: https://github.com/triton-lang/triton/pull/10810 - ROCm backport (the commit this pin now points at): https://github.com/ROCm/triton/pull/951 **Why it matters.** On RDNA the current pin selects `block_m = 128` tiles for the MXFP4 MoE GEMMs. The fp32 accumulator alone is `BLOCK_M × BLOCK_N` registers, which at that tile size consumes half of the 256-VGPR wave32 budget before any operand is loaded. Register demand exceeds the architectural cap, the compiler spills to scratch memory, and the spills land inside the K-loop where the cost is paid on every iteration. WMMA instruction counts are unchanged — the arithmetic is identical and the extra time is pure spill overhead. The backported configuration sets `block_m` explicitly for MXFP4 on RDNA, so the spilling tiles are never built. This is a pin bump only. The kernel selection logic is ROCm's backport of upstream work; no vLLM code is changed beyond the pinned SHA. ## Test Plan `gpt-oss-20b` (MXFP4 MoE) on RDNA, TP=1. ```bash vllm bench throughput \\ --model openai/gpt-oss-20b --max_model_len 4096 --trust-remote-code \\ --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \\ --num_prompts 1000 --tensor_parallel_size 1 --gpu-memory-utilization 0.8 ``` Register and spill counts read from `.vgpr_count` / `.vgpr_spill_count` in the compiled `.amdgcn` products, selecting the autotune-chosen variant by `num_warps`. ## Test Result Confirmed the performance gain on RDNA that the linked PRs describe. Throughput improved and no remaining kernel spills observed.", "url": "https://github.com/vllm-project/vllm/pull/52056", "createdAt": "2026-08-12T21:52:07Z", "updatedAt": "2026-08-12T22:33:06Z", "timestamp": "2026-08-12T22:33:06Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "rocm", "ready", "ci/build" ], "author": "umarkovi-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4f4bb09d796e7431eb6e", "signalId": "github:vllm-project/vllm:pull_request:49819", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49819", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add Cohere2MoE Eagle3 auxiliary hidden states", "text": "## Purpose Enable Cohere2MoE target models to serve with compatible Eagle3 and DFlash draft models by exposing the auxiliary hidden states required by the speculator. I ~am working on training~ have released a DFlash speculator for North-Mini-Code-1.0 and realized that this was missing! Hopefully it helps others, ~especially when I upload the checkpoints.~ :) ### Update: Public Checkpoints As promised, the checkpoints are now public and there’s a turnkey runtime so this change can be exercised without a custom build. **Weights (auxiliary draft/speculator checkpoints, Apache-2.0):** - DFlash: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dflash - DSpark: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dspark **Runtime:** a thin overlay on official vLLM v0.27.1 carrying exactly this PR (#49819) plus PR #50937 (skip empty expert bias — required because North-Mini-Code w4a16 has an all-zero per-expert bias, causing vllm `0.27.1` to throw `AttributeError: 'RoutedExperts' object has no attribute 'w2_bias'`). Nothing is compiled from source; it’s two small pure-Python module overlays. - Repo: https://github.com/sdougbrown/north-mini-code-draft-runtime - Published multi-arch image: `ghcr.io/sdougbrown/north-mini-code runtime:v0.27.1-49819-50937` **Validation:** both DFlash (K3) and DSpark (K4) serve and run speculative decoding on NVIDIA GB10. ### Running the Draft Models You can use the pre-built docker runtime I put togther if you like, or apply the patches from https://github.com/sdougbrown/north-mini-code-draft-runtime/tree/main/patches to a fresh `0.27.1` wheel and then serve: ``` vllm serve sdougbrown/North-Mini-Code-1.0-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' ``` The easy docker path is: ``` docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ sdougbrown/North-Mini-Code-1.0-dspark \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --max-model-len 32768 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` Note that by default this fetches the weights from huggingface. If that's not desired, you can get fancier. Here's an example I would use on my local machine assuming `~/Code` is the parent folder for the runtime repo, and `~/Models` is where I've downloaded the weights: ``` # 1) Point the verifier at a local copy of the target and render a bundle: cd ~/Code/north-mini-code-draft-runtime VERIFIER_MODEL=\"$HOME/Models/North-Mini-Code-1.0-w4a16\" \\ RUNTIME_WORK=\"$HOME/Models/north-deploy\" \\ ./scripts/prepare-deployment-model.sh dspark # -> $HOME/Models/north-deploy/deploy/dspark # (config.json now has verifier.name_or_path -> local path; weights are symlinks) # 2) Docker: mount Models at the SAME path and pass the bundle path: docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ -v \"$HOME/Models:$HOME/Models\" \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ \"$HOME/Models/north-deploy/deploy/dspark\" \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` ## Implementation - implement `SupportsEagle3` for `Cohere2MoeForCausalLM` - capture embedding and post-layer boundaries using global layer indices - preserve Cohere2's parallel-residual behavior without adding the residual twice - preserve baseline model outputs when auxiliary states are disabled - preserve DFlash's `target_layer_ids -> auxiliary boundary + 1` semantics Eagle3, DFlash, and DSpark with pipeline parallelism are already rejected by the V1 GPU model runner. This change does not add auxiliary-state propagation between pipeline stages. ## Duplicate-work check PR #42078 added Cohere Eagle support, but it did not expose auxiliary hidden states from the Cohere2MoE target. Searches of open PRs found no overlapping implementation. ## Tests ```bash .venv/bin/pre-commit run --files \\ vllm/model_executor/models/cohere2_moe.py \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py .venv/bin/python -m pytest \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py -q ``` All pre-commit hooks passed. Pytest result: `15 passed`. ## Model evaluation The original `a1cad4f67` implementation was successfully exercised with: - AMD ROCm 7.2.4, TP=2, INT4/AutoGPTQ - NVIDIA GB10 CUDA 13, TP=1, W4A16 with MARLIN NVFP4 MoE - NVIDIA GB10 CUDA 13, TP=1, FP8 with TRITON FP8 MoE All three configurations completed target-model requests while exposing five finite auxiliary hidden states. The ROCm TP=2 run also verified byte-identical per-layer values across both ranks. These runs validated the target-side auxiliary-state path rather than end-to-end draft-token acceptance. ## AI assistance AI assistance was used to port and test this change. I manually reviewed every changed line and the test results.", "url": "https://github.com/vllm-project/vllm/pull/49819", "createdAt": "2026-07-25T17:06:03Z", "updatedAt": "2026-08-12T22:32:57Z", "timestamp": "2026-08-12T22:32:57Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "speculative-decoding", "v1" ], "author": "sdougbrown", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8244a5f6bd3b2d76513c", "signalId": "github:vllm-project/vllm:pull_request:49225", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49225", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV-offload][FS]: Batching for read/write threads", "text": "PR #49152 introduced batched C versions of load_block / store_block. The batching was request-level, i.e. each thread in the thread_pool receives all the keys for a request and loads/stores them one after another. If there are 16 requests, each with say 100 keys to load, and we have 16 threads. Each thread will work on a separate request. This is less desirable as read jobs for different requests are interleaved on the disk, increasing any single request's total read time. This PR: Batch load/stores with the intent of parallelizing per-request loads/stores. i.e. Care about FCFS at the request level. In this PR, a single request is split into as many batches as there are threads and each thread gets an even distribution of work. ### Benchmarks Benchmark the following configurations, offload block size | n_read_threads / n_write_threads | comment -- | -- | -- 1024 | 16 | less keys to load per request (tens of keys) 1024 | 8 16 (default) | 16 | lots of keys to load per request (thousands of keys) 16 (default) | 8 #### Setup ``` MODEL=\"openai/gpt-oss-120b\" TP_SIZE=2 CPU_BYTES=25769803776 KV_TRANSFER_CONFIG=$(cat < offload block size | rw threads | main Total tokens / s | PR Total tokens / s | improvement x -- | -- | -- | -- | -- 1024 | 16 | 57490.4 | 59455.2 | 1.03x 1024 | 8 | 57923.7 | 60093.6 | 1.03x 16 | 16 | 52987.7 | 55501.7 | 1.04x 16 | 8 | 54061 | 56137 | 1.03x ### Eval ``` vllm serve openai/gpt-oss-120b --tensor-parallel-size=2 --kv-transfer-config '{ \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"spec_name\": \"TieringOffloadingSpec\", \"cpu_bytes_to_use\": 150930000000, \"eviction_policy\": \"lru\", \"secondary_tiers\": [{ \"type\": \"fs\", \"root_dir\": \"/mnt/nvme-storage/\", \"n_read_threads\": 16, \"n_write_threads\": 16 }] ,\"enable_cross_layers_blocks\" : \"True\" } }' --enable-prefix-caching --no-disable-hybrid-kv-cache-manager --port 8000 ``` ``` TARGET_URL=\"http://127.0.0.1:8000\" MODEL=\"openai/gpt-oss-120b\" LM_EVAL_NUM_CONCURRENT=1000 LM_EVAL_TASKS=\"gsm8k\" lm_eval \\ --model local-completions \\ --model_args \"base_url=${TARGET_URL}/v1/completions,model=${MODEL},tokenized_requests=False,num_concurrent=${LM_EVAL_NUM_CONCURRENT},trust_remote_code=True\" \\ --tasks ${LM_EVAL_TASKS} \\ --seed 42 \\ --num_fewshot 25 \\ --gen_kwargs temperature=0.0 ``` ### Eval results #### Run1 : cold disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6353|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4200|± |0.0136| ``` #### Run2: warm disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6323|± |0.0133| | | |strict-match | 25|exact_match|↑ |0.4481|± |0.0137| ``` #### Run3: warm disk ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 25|exact_match|↑ |0.6164|± |0.0134| | | |strict-match | 25|exact_match|↑ |0.4253|± |0.0136| ```", "url": "https://github.com/vllm-project/vllm/pull/49225", "createdAt": "2026-07-20T16:29:10Z", "updatedAt": "2026-08-12T22:30:14Z", "timestamp": "2026-08-12T22:30:14Z", "metrics": { "reactions": 2, "comments": 20 }, "labels": [ "ready", "v1" ], "author": "varun-sundar-rabindranath", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:30e50b6de8a1dda916ed", "signalId": "github:vllm-project/vllm:pull_request:48915", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48915", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses", "text": "## Purpose Speculative decoding guesses several tokens ahead, then checks them. Some guesses get accepted, some don't. How many get accepted tells you how well spec decode is working. Today you can only see this as a server-wide average on the `/metrics` page. You can't tell how any single request did. Tools like AIPerf want that per-request number, and right now they'd have to wrap the engine to get it. This PR puts the numbers right in the response. It's off by default; you turn it on at server start with `--per-request-spec-decode-stats summary` (or `detailed`). When it's off, the response is unchanged. The stats go on each choice: ```json { \"choices\": [ { \"index\": 0, \"message\": {\"role\": \"assistant\", \"content\": \"...\"}, \"finish_reason\": \"stop\", \"speculative_decoding_stats\": { \"mean_acceptance_length\": 2.5, \"draft_acceptance_rate\": 0.5, \"acceptance_histogram\": {\"0\": 8, \"2\": 6, \"3\": 6}, \"num_spec_steps\": 20, \"num_accepted_draft_tokens\": 30, \"num_draft_tokens\": 60, \"num_spec_tokens\": 3 } } ] } ``` `summary` gives the fields above. `detailed` also adds `per_step_accepted` and `per_step_drafted` arrays, one entry per step. This builds on #47322, which does the same job with a different shape. That PR attaches one stats object to the response as a whole, taken from the first sequence, and leads with per-step arrays instead of a histogram. Putting the stats on each choice matters when you ask for `n>1`: every choice reports its own acceptance, instead of a single blob that only reflects one of them. ## Relationship to prior work This builds directly on @reed-meyerson's #47322 — it's an adaptation and continuation of their work, and thanks to them for the original design and groundwork. See the Purpose section above for how the shape differs (per-choice + histogram vs one response-level blob + per-step arrays). The closest other relative is #43310, so to be clear on why this isn't a duplicate of it: #43310 also returns per-request spec-decode counts in the response, but on a different surface for a different user. It targets vLLM's native `/inference/v1/generate` endpoint, attaches raw counts at the request level, and is aimed at RL rollout tools (verl). This PR targets the OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`), puts stats per choice so `n>1` works, and adds derived numbers like mean acceptance length and the histogram. The two don't overlap in endpoint, placement, or consumer. A few other efforts are nearby but doing something else: - RFC #48202 (with impls #48204 and #48692) is about *changing* how many tokens get drafted per request — adaptive, variable-length drafting. This PR doesn't change any behavior. It just reports what happened. And `detailed` already records how many tokens were drafted each step, so it keeps working if drafting becomes variable. - #44487 adds a per-request acceptance histogram to the `/metrics` Prometheus page. That's server-side scraping, not fields in the response body. ## Test Plan Unit: ``` .venv/bin/python -m pytest tests/v1/spec_decode/test_request_acceptance.py -v .venv/bin/python -m pytest tests/v1/core/test_scheduler.py -k spec_decod -v ``` E2E (H100, n-gram spec decode, no draft weights required): ``` vllm serve Qwen/Qwen2.5-0.5B-Instruct --port 8000 \\ --speculative-config '{\"method\":\"ngram\",\"num_speculative_tokens\":3,\"prompt_lookup_max\":3,\"prompt_lookup_min\":1}' \\ --per-request-spec-decode-stats detailed --max-model-len 4096 ``` - Non-streaming chat: `choices[0].speculative_decoding_stats` populated. - `n=2` completion: each `choices[i]` carries independent stats. - Streaming: stats present only on the terminal chunk's choice. - `--per-request-spec-decode-stats summary`: `per_step_*` arrays absent; default `none`: field is `null` on every choice. - Reconciliation: summed per-request fields vs the aggregate `/metrics` spec-decode counters on a fresh server. ## Test Result Unit: `11 passed` (accumulator math, summary/detailed `to_dict`, msgspec round-trip, `CompletionOutput` field). Scheduler integration (`tests/v1/core/test_scheduler.py -k spec_decod`): passed — the per-request accumulator is asserted alongside the aggregate stats in `test_schedule_spec_decoding_stats`, plus the `detailed` per-step recording, invalid-draft subtraction, and off-by-default cases. E2E on NVIDIA H100 80GB (CUDA 13.2): all checks passed — per-choice emission, independent `n=2` stats, streaming terminal-chunk placement, `detailed`→`summary` gating, and default-`none` omitting the field. Representative `detailed` choice (n-gram, `num_speculative_tokens=3`): ```json {\"mean_acceptance_length\":1.2325581395348837,\"draft_acceptance_rate\":0.07751937984496124,\"acceptance_histogram\":{\"0\":39,\"1\":1,\"3\":3},\"num_spec_steps\":43,\"num_accepted_draft_tokens\":10,\"num_draft_tokens\":129,\"num_spec_tokens\":3,\"per_step_accepted\":[3,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0],\"per_step_drafted\":[3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3]} ``` Self-consistent: histogram counts sum to `num_spec_steps` (43); `sum(j·count) = num_accepted_draft_tokens` (10); `mean_acceptance_length = 1 + 10/43`; `num_draft_tokens = 43×3`. Prometheus reconciliation (fresh server, summed across all choices of all requests vs the aggregate counters): | Per-request field (summed) | `/metrics` counter | Result | | --- | --- | --- | | `num_spec_steps` | `vllm:spec_decode_num_drafts_total` | 134 = 134 | | `num_draft_tokens` | `vllm:spec_decode_num_draft_tokens_total` | 402 = 402 | | `num_accepted_draft_tokens` | `vllm:spec_decode_num_accepted_tokens_total` | 126 = 126 | ## Model evaluation Purely observational — no change to sampling, scheduling, or generated tokens. Output is byte-identical across `none`/`summary`/`detailed`. No accuracy eval applicable. ## AI assistance AI assistance (Claude) was used for implementation and test authoring. All changed lines were reviewed by the submitting human, and the tests above were run and verified.", "url": "https://github.com/vllm-project/vllm/pull/48915", "createdAt": "2026-07-17T05:17:50Z", "updatedAt": "2026-08-12T22:29:13Z", "timestamp": "2026-08-12T22:29:13Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "documentation", "frontend", "speculative-decoding", "ready", "needs-rebase", "v1", "verified" ], "author": "matthewkotila", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ac352e73df2f7d9a941c", "signalId": "github:vllm-project/vllm:pull_request:48282", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48282", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Emit added/done lifecycle events for zero-delta streaming items", "text": "Fixes #48274 ## What's going on There's a known bug in `emit_previous_item_done_events` where items that complete with zero deltas get silently dropped from the SSE stream. The code already acknowledges it: ```python if not state.sent_output_item_added and not state.is_first_function_call_delta: # Suppress done events for items had no delta and thus had no # added/in-progress lifecycle events. This is a bug. # TODO: Ensure added/in-progress events are emitted for zero-delta items. return [] ``` This means any client tracking output items by lifecycle events just never sees these items at all. ## The fix I added a small helper `_emit_zero_delta_added()` that synthesizes the missing `response.output_item.added` event when an item completes without any prior deltas. It covers all five item types: function calls, code interpreter, MCP calls, reasoning, and text messages. The helper sets up the correct `StreamingState` flags so the downstream `emit_*_done_events` functions work as expected. The existing `emit_previous_item_done_events` now calls this helper instead of bailing out early. If the helper can't identify the item type (unknown channel, no recipient), it still returns early to avoid emitting orphaned done events. Also removed the `xfail` marker from `test_zero_delta_items_should_preserve_streaming_lifecycle` since the test passes now. ## Testing The xfail test that was already written for this bug now runs green. It covers all the zero-delta item types (text, function call, code interpreter, MCP, reasoning) and validates the full added->done lifecycle for each. ## AI disclosure This PR was developed with assistance from Cursor (AI coding tool). I reviewed and validated all changes.", "url": "https://github.com/vllm-project/vllm/pull/48282", "createdAt": "2026-07-10T21:45:25Z", "updatedAt": "2026-08-12T22:27:36Z", "timestamp": "2026-08-12T22:27:36Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "frontend" ], "author": "mimran-khan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d963a0e203dd2f3d2c39", "signalId": "github:vllm-project/vllm:pull_request:50605", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50605", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm]: Bump torch 2.13, triton 3.8, torchaudio, torchvision", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50605", "createdAt": "2026-07-31T17:52:47Z", "updatedAt": "2026-08-12T22:23:49Z", "timestamp": "2026-08-12T22:23:49Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "ready", "needs-rebase", "ci/build" ], "author": "Rohan138", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d279d131b8de31d1f18a", "signalId": "github:vllm-project/vllm:pull_request:50082", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add Kimi K3 MoE support to benchmark_moe.py", "text": "## Purpose `benchmarks/kernels/benchmark_moe.py` cannot tune the fused MoE Triton kernel for Kimi K3. `get_model_params()` does not recognize `KimiK3ForConditionalGeneration`, so it falls through to the Mixtral default branch and raises `AttributeError` (Kimi's config has no `num_local_experts`). Kimi K3 is multimodal and stores its MoE parameters in a nested `KimiLinearConfig` `text_config`, and it exposes the routing top-k as `num_experts_per_token` rather than the more common `num_experts_per_tok` used by the existing handlers. This adds a dedicated branch, following the same pattern as the Qwen3.5 handler added in #37068. It also covers the text-only `KimiLinearForCausalLM`, for which `get_text_config()` returns the config itself. ## Test Plan Parse the released Kimi K3 config and confirm the extracted MoE parameters: ```python from vllm.transformers_utils.config import get_config from benchmark_moe import get_model_params cfg = get_config(\"\", trust_remote_code=True) print(get_model_params(cfg)) ``` ## Test Result ``` architectures = ['KimiK3ForConditionalGeneration'] (text_config: KimiLinearConfig) get_model_params -> (E=896, topk=16, moe_intermediate_size=3072, hidden_size=7168) ``` The values match the model config. With `--tp-size 8 --enable-expert-parallel` this yields `E_local=112` and shard `N=6144`. Before this change the script raised `AttributeError` before tuning could start.", "url": "https://github.com/vllm-project/vllm/pull/50082", "createdAt": "2026-07-28T04:27:24Z", "updatedAt": "2026-08-12T22:10:00Z", "timestamp": "2026-08-12T22:10:00Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "performance", "ready", "kimi", "k3" ], "author": "vanshbhatia-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:62c1bf01057494bfcb39", "signalId": "github:vllm-project/vllm:pull_request:50487", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50487", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Spec Decode] Tap the pre-norm AttnRes mixture as the Kimi K3 DFlash aux state", "text": "## Purpose The DFlash drafter consumes auxiliary hidden states captured at a fixed set of target layers. K3 captures the post-mixture stream, which is not what the drafter was trained against: the AttnRes residual mixture is applied before the layer norm, and the current capture site reads the value after it. Tapping the pre-norm mixture instead recovers the stream the drafter expects. This selects the correct weights to mix against depending on where the tapped layer sits — the next layer's own AttnRes weights when one follows, the model's output-side aggregation for the last layer on the final pipeline stage, and a fall back to the running prefix for the last layer of a non-final stage, where the consumer lives on the next rank and the output-side weights do not exist there. Gated behind `VLLM_KIMI_K3_AUX_ATTN_RES_STREAM` (default off) while it settles, since it changes what the model feeds the speculator. The one-shot log of which layers are tapped and which capture mode is active is emitted from `_set_aux_hidden_state_layers`, the setup-time hook the speculator calls once, rather than from inside `forward`. Setting an `nn.Module` attribute inside a compiled forward pass risks a graph break or recompile under `torch.compile`; doing it at setup time avoids that and is also the natural place to log the tapped layer tuple. I checked for duplicate/overlapping open PRs (`gh pr list --search` on \"kimi k3 aux hidden state\" and \"attn_res eagle3\"); no open PR touches the Kimi K3 auxiliary hidden state capture site. I used AI assistance (Cursor) to draft, test, and validate this change, and I reviewed every changed line before submitting. ## Test Plan Unit: ``` pytest tests/models/kimi_k3/test_aux_attn_res_stream.py \\ tests/models/kimi_k3/test_eagle3.py \\ tests/models/kimi_k3/test_attn_res.py -v ``` End to end: Kimi K3 on 8×B300 at TP8, DFlash drafter at `num_speculative_tokens 16`, one concurrent user, single variable (the capture mode) on an otherwise identical container. Acceptance is `1 + accepted/steps` from raw counter deltas. ## Test Result **Unit.** All pass: ``` tests/models/kimi_k3/test_aux_attn_res_stream.py ....... [7 passed] tests/models/kimi_k3/test_eagle3.py .... [4 passed] tests/models/kimi_k3/test_attn_res.py ......sss..... [12 passed, 3 skipped] ``` The 3 skips are the multi-GPU (`nvidia-1/4/8`) parametrizations of `test_attn_res`, not runnable on the single-GPU box used to verify this PR. `test_aux_attn_res_stream.py` is new and covers the selection logic added here: the consumer-layer branch, the last-layer/final-rank branch that uses the model's output-side aggregation, the last-layer/non-final-rank fallback to the running prefix, and that the feature is a no-op reproducing the original `prefix_sum + hidden_states` when disabled. One pre-existing test needed a fixture fix rather than a code change: `test_eagle3.py::test_kimi_k3_uses_shared_eagle3_layer_configuration`'s shared model stub predates this PR and didn't set `use_attn_res`, which the new `_set_aux_hidden_state_layers` override now reads unconditionally. Fixed the stub to set it, matching what the other two tests in that file already do at their own call sites. **End to end.** Mean accept length 4.7167 and 4.3948 with the capture on, against 2.2913 and 2.6579 with it off. Paired across 32 matched rows: +1.903 accept length, treatment winning 31 of 32, p = 1.5e-8. An independent implementation of the same idea reports +1.536 at a different operating point (2.165 prefix-only against 4.454), which is the same effect measured twice. **How to read these numbers.** Some rows come from different containers rather than uniformly from the same one, and the gain was measured with prefix caching off; a follow-up with caching on read +0.083, inside the within-arm spread. The flag is off by default for that reason — the mechanism is right and the off-cache effect is large and well established, but it should be re-measured on a cached lane before anyone depends on it there. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/50487", "createdAt": "2026-07-30T22:35:57Z", "updatedAt": "2026-08-12T22:00:51Z", "timestamp": "2026-08-12T22:00:51Z", "metrics": { "reactions": 3, "comments": 24 }, "labels": [ "ready", "verified", "kimi", "k3" ], "author": "rchalamala", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3b04a13cebe457dcfe3f", "signalId": "github:vllm-project/vllm:pull_request:50303", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50303", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Stop the Granite reasoning marker leaking into streamed reasoning", "text": "## Summary `GraniteReasoningParser` leaks the tail of `Here is my thought process:` into the streamed `reasoning` field when a single delta closes that marker and opens the reasoning at the same time. `_get_delta_message_with_no_response_bounds` reconstructs \"the reasoning before this delta\" as: ```python previous_text = reasoning[: -len(delta_text)] ``` That assumes the whole delta lies inside the parsed reasoning. It does not when the delta straddles the end of the marker. BPE merges the marker's `:` with whatever punctuation follows it into one token, so for `Here is my thought process:(thinking)` the tokenizer emits `':('` as a single delta while `reasoning` is only `'('`. The slice underflows to `\"\"`, `prev_idx` and `delta_idx` both go to `-1`, and the function falls through to `return DeltaMessage(reasoning=delta_text)`, emitting the colon as reasoning. ## Reproduction Using the tokenizer the existing test file already uses (`facebook/opt-125m`): ``` text : Here is my thought process:(thinking)Here is my response:(answer) deltas : ['Here', ' is', ' my', ' thought', ' process', ':(', 'thinking', ')', 'Here', ...] expected : reasoning='(thinking)' content='(answer)' streaming: reasoning=':(thinking)' content='(answer)' ``` Non-streaming `extract_reasoning` is already correct on the same text, so the two paths disagree. This is not exotic. Any punctuation directly after the marker triggers it, because that is exactly where BPE merges. Over 70 marker/joiner combinations, `main` disagrees with non-streaming extraction on 16 of them, identically under three tokenizers: | tokenizer | before | after | |---|---|---| | facebook/opt-125m | 16/70 mismatched | 0/70 | | gpt2 | 16/70 | 0/70 | | bert-base-uncased | 16/70 | 0/70 | The joiners that trigger it are `,` `]` `\"` `'` `)` `(` `*` `-` `_` `/` `:` `;` `!` `?` `#` `0`. The ones that do not are whitespace, `.` and letters, which is why the existing fixtures (all of which use a letter or a newline after the marker) never caught it. ## Fix Clamp to what `reasoning` actually holds, and emit only that part of the delta: ```python delta_len = min(len(delta_text), len(reasoning)) previous_text = reasoning[: len(reasoning) - delta_len] delta_text = reasoning[len(reasoning) - delta_len :] ``` The explicit index math also removes the `-0` case: `reasoning[:-len(delta_text)]` returns `\"\"` rather than the whole reasoning when the delta is empty, which is reachable because the streaming loop only skips an empty delta while there are no token ids and no previous tokens. That case happens to produce the same `None` delta message either way, so it is a latent slice bug rather than an observable one, and this fix removes it in passing. ## Scope Only `_get_delta_message_with_no_response_bounds`. Two neighbours were checked and left alone: `_get_delta_message_with_no_reasoning_bounds` returns accumulated `current_text` by design, and `_get_delta_message_with_both_bounds` computes its offsets from `current_text`, so neither has this assumption. One limitation stays, deliberately, because I could not reach it with a real tokenizer: `delta_idx = delta_text.rfind(\"Here\")` needs the whole word `Here` inside one delta, so an arbitrary sub-word split of the *response* marker (`'H'`, `'er'`, `'e'`) can still leak a partial marker into reasoning. All three tokenizers above emit `Here` whole in every position tested, so this is not reachable today, and fixing it means reworking the buffering logic rather than a slice. Flagging it rather than quietly leaving it. ## Not a duplicate - `gh pr list --repo vllm-project/vllm --state open --search \"granite reasoning\"` returns nothing touching `vllm/reasoning/granite_reasoning_parser.py`. #44713 adds Granite parsers to the **Rust** frontend, a different implementation. - No open or closed issue describes this; the only Granite reasoning issue, #14202, is the closed request that added the parser. - Last change to this file was #45988 in June, an unrelated logger removal. ## Tests Added to the existing `tests/reasoning/test_granite_reasoning_parser.py` rather than a new file: - `PUNCTUATION_AFTER_START_SEQ`, wired into `TEST_CASES` for both streaming and non-streaming, so the two paths are pinned to agree. - `STREAMING_14`, the focused unit subcase for the straddling delta (`previous_text=\"Here is my thought process\"`, `delta_text=\":(\"`). Fail-before / pass-after, with the fix reverted and the tests present: ``` $ python -m pytest tests/reasoning/test_granite_reasoning_parser.py -q FAILED ...::test_reasoning[punctuation_after_start_seq_streaming] FAILED ...::test_streaming_subcases[Delta ends the start reasoning sequence and starts the reasoning] 2 failed, 28 passed $ # with the fix 30 passed ``` The non-streaming half of the new case passes both before and after, which is the point: it pins the expectation to what `extract_reasoning` already returns rather than to something invented. Whole directory, with a clean-tree control so the pre-existing failures are not mistaken for mine: ``` $ python -m pytest tests/reasoning/ -q # clean tree 158 failed, 249 passed $ python -m pytest tests/reasoning/ -q # this branch 158 failed, 252 passed ``` Same 158 either way. They are all `test_qwen3_reasoning_parser.py` cases that need model files my machine cannot fetch, and the delta is exactly the 3 tests added here. Five files (`cohere_command`, `kimi_k2`, `kimi_k3`, `mistral`, `nemotron_v3`) are excluded from both runs because they fail to import without `mistral_common` / `partial_json_parser` / `xgrammar` locally. Lint: `ruff 0.14.0` (the version pinned in `.pre-commit-config.yaml`) `check` and `format --diff` are both clean on the two changed files. ## Model evaluation Not applicable, and I want to be explicit rather than silently skip the requirement. This changes string slicing inside a reasoning parser that runs on already-decoded text in the API layer. It does not touch model weights, sampling, kernels or the scheduler, and cannot change which tokens are generated: identical output text produces identical results before and after, only the split between the `reasoning` and `content` fields of the streamed response changes. The behaviour is fully determined on CPU, which is what the tests above exercise. I do not have a GPU available to run `tests/evals/`, and an eval would not exercise this path anyway. AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/50303", "createdAt": "2026-07-29T18:01:52Z", "updatedAt": "2026-08-12T21:55:23Z", "timestamp": "2026-08-12T21:55:23Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "tool-calling" ], "author": "vineethsaivs", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d8a7655d997c7ad2d521", "signalId": "github:vllm-project/vllm:issue:51874", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51874", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Polish DSpark adaptive timings", "text": "### 🚀 The feature, motivation and pitch Following #47808, the engine profiles the forward pass and proposer timings for use in estimating the batch runtime. The measurement facility could use a refactor. It is valuable to expose these metrics to the frontend, via /metrics endpoint and also compatibility with iteration-level logging so we can easily observe the fine-grained engine GPU latency for different phases (drafting, verification, sampling, other). See https://github.com/vllm-project/vllm/pull/50919 for a starting point. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51874", "createdAt": "2026-08-11T20:39:19Z", "updatedAt": "2026-08-12T21:54:14Z", "timestamp": "2026-08-12T21:54:14Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request" ], "author": "benchislett", "state": "open", "assignees": [ "benchislett" ], "change": "updated" } }, { "id": "event:ee269877e8385b98f181", "signalId": "github:vllm-project/vllm:issue:52057", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52057", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Improve Adaptive DSpark Online Profiling", "text": "### 🚀 The feature, motivation and pitch The initial version does profiling at startup to estimate the verification step time by replaying the recorded CUDA graphs. This is known to have some drift, especially at higher batch sizes. Here are my results from initial study: \"Image\" We want to extend this to fully online profiling. However, there are some challenges that make this tricky: - Online profiling may lead to \"dead spots\": one-time latency spikes make a certain batch size appear unattractive, leading to the adaptive verifier to never select that batch size, which leads to negative feedback and stale data limiting our range of profiled values. Generally there is an \"exploration vs exploitation\" problem here, as well as a requirement for some way to detect and eliminate outliers (such as those caused by JIT compilation). - Variable sequence lengths make it hard to measure a single reliable estimate of execution time for a given number of tokens. In practice we will want some way to account for this: either tracking measured points in 2D (num_tokens, max_seq_len or mean_seq_len), or estimating the overhead added per-token of KV reads We should do a more comprehensive study on the drift across models, batch sizes, and sequence lengths. Then, we should design a fully-online solution that addresses the shortcomings, and evaluate the performance gain. We should also evaluate the \"oracle\" speedup with perfect timing prediction (retrospectively) to assess how much performance can be gained", "url": "https://github.com/vllm-project/vllm/issues/52057", "createdAt": "2026-08-12T21:53:44Z", "updatedAt": "2026-08-12T21:53:44Z", "timestamp": "2026-08-12T21:53:44Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "benchislett", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4f297786b5a175718170", "signalId": "github:vllm-project/vllm:pull_request:52048", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52048", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add DeepseekV4ForCausalLM to benchmark_moe.py model param dispatch", "text": "## Purpose Fixes #52042. `benchmark_moe.py --tune` crashes for DeepSeek-V4 models (e.g. `deepseek-ai/DeepSeek-V4-Flash-0731`): ```text AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` `get_model_params()` dispatches on the architecture string. The DeepSeek-family branch reads `n_routed_experts` / `num_experts_per_tok` / `moe_intermediate_size`, but `DeepseekV4ForCausalLM` is missing from its tuple, so V4 falls through to the Mixtral default, which reads `num_local_experts` — not present on DeepSeek configs. This adds `DeepseekV4ForCausalLM` (the registered arch name, see `vllm/model_executor/models/registry.py`) to that tuple. vLLM's own V4 model reads exactly those three fields when building the MoE gate and experts (`vllm/models/deepseek_v4/nvidia/model.py`), so the branch returns the correct shapes rather than plausible-but-wrong ones. `DeepSeekV4MTPModel` / `DSparkDraftModel` are intentionally left out: the dispatch carries no MTP/draft architecture for any family, and this benchmark tunes main-model MoE shapes. **Relation to existing PRs:** #49088 generalizes this dispatch to alias-based field probing and would make this change redundant, but it has been inactive since 2026-07-19; this is a minimal targeted unblock for the filed issue, and I am happy to close it if #49088 lands first. #40546 only improves the fallback error message. #50082 adds Kimi K3 via the same one-architecture pattern. ## Test Plan The change only affects config-field dispatch, so no GPU is required. Called `get_model_params()` with a `DeepseekV4Config` carrying V4-Flash MoE fields (`n_routed_experts=256`, `num_experts_per_tok=8`, `moe_intermediate_size=2048`, `hidden_size=7168`), plus a `DeepseekV3ForCausalLM` control to confirm the existing branch is unaffected. ## Test Result - Before: `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'` — the exact error from #52042 - After: returns `(256, 8, 2048, 7168)` - V3 control unchanged: `(160, 6, 1536, 5120)` - `ruff check` and `ruff format --check` clean on the file Model evaluation: not applicable — this is a benchmarking-script fix with no effect on model output, accuracy, or serving. --- This change was developed with AI assistance; I reviewed and tested every line.", "url": "https://github.com/vllm-project/vllm/pull/52048", "createdAt": "2026-08-12T21:09:52Z", "updatedAt": "2026-08-12T21:49:06Z", "timestamp": "2026-08-12T21:49:06Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "performance", "deepseek" ], "author": "rajathpi", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f84ed2a008c0db042fcf", "signalId": "github:vllm-project/vllm:pull_request:50796", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50796", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Build] Restore DeepGEMM SM120 support while preserving SITU", "text": "## Purpose Addresses the current-main regression reported in #47436 for block-scaled FP8 checkpoints on SM120. The Kimi K3 integration changed the vendored DeepGEMM revision from the `nv-dev` commit used by vLLM v0.26.0 to `f5a76426`, which provides SITU support but does not contain the SM120 scale-factor layout dispatch. As a result, current nightly builds select `DeepGemmFp8BlockScaledMMKernel` and fail while post-processing weights: ```text RuntimeError: Assertion error (.../deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation ``` This updates both DeepGEMM pins to `2fd67329`. Its direct parent is the previously validated `5f33a180` `nv_dev+situ` merge commit, which combines the SM120 support from `nv-dev` with the SITU activation support required by Kimi K3. The additional commit carries the CUDA 12.9 `cuda_fp8.h` include fix from #51003. ### Duplicate-work check No open PR addressed this regression when this PR was opened. Draft #51382 was opened later as a materially different alternative: it minimizes the DeepGEMM dependency delta by cherry-picking SITU onto the earlier SM120 revision, while this PR uses the later `nv_dev+situ` merge and its CUDA 12.9 follow-up. The other related PRs are also not duplicates: - #47988 and #48588 improve CUTLASS/Triton fallback behavior; they do not restore the DeepGEMM SM120 path. - #41834 is a broader DeepSeek V4 SM12x enablement branch and does not currently modify either DeepGEMM pin file. - #47304 previously moved the pin to `nv-dev` for SM120 support. This change restores that support after the later SITU pin update, while preserving SITU. ## Test Plan Build a CUDA 13 production image for SM120 from vLLM main with the updated pin, then serve the real `DeepSeek-V4-Flash-0731` checkpoint on six RTX 5090 GPUs: ```text model: DeepSeek-V4-Flash-0731 vLLM: 0.26.1rc1.dev251+g0033211c0.d20260803 GPU: 6x NVIDIA RTX 5090 (SM120) parallelism: TP=2, PP=3 checkpoint size: 155.43 GiB, 48 safetensors shards ``` ```bash vllm serve /model \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --max-model-len 524288 \\ --tensor-parallel-size 2 \\ --pipeline-parallel-size 3 \\ --gpu-memory-utilization 0.97 \\ --max-num-seqs 1 \\ --enable-prefix-caching \\ --disable-log-stats \\ --enable-auto-tool-choice \\ --tool-call-parser deepseek_v4 ``` Additional checks: ```bash bash -n tools/install_deepgemm.sh git diff --check ``` After startup, send repeated OpenAI-compatible chat-completion requests, followed by a request with a `get_weather` function schema. Scan the serving log for `Unknown SF transformation`, `illegal memory access`, engine failures, and CUDA errors. ## Test Result The original pre-rebase candidate (`5f33a180`) built successfully with CUDA architecture `12.0f`; the build log checked out that merge revision: ```text 5f33a18 Merge branch 'situ-activation' into nv_dev+situ DeepGEMM CUDA architectures: 12.0f ``` The model loaded all real checkpoint shards and selected the intended kernels: ```text Loading safetensors checkpoint shards: 100% Completed | 48/48 DeepGEMM E8M0 enabled on current platform. Using 'DEEPGEMM_MXFP4' Mxfp4 MoE backend. Using DeepGemmFP4Experts Application startup complete. ``` Runtime results after one-time per-shape JIT warmup: | Check | Result | | --- | --- | | Weight loading | 48/48 shards loaded; no SF-layout assertion | | Chat completion | Correct Chinese responses returned | | Warmed decode | 84.1-100.6 output tok/s across repeated 83/90-token responses | | Tool calling | `finish_reason=tool_calls`, `get_weather({\"city\":\"北京\"})` | | Error scan | No `Unknown SF transformation`, illegal memory access, engine failure, or CUDA error | The first request for a new shape was slower because current main performs one-time TileLang/DeepGEMM JIT compilation; repeated requests used the cached kernels. The performance figures above are warmed single-request measurements, not aggregate throughput. After rebasing, static validation confirmed that `2fd67329` is a direct child of the hardware-tested `5f33a180` and differs only by the CUDA FP8 header include from #51003. `bash -n tools/install_deepgemm.sh`, pin-consistency validation, and `git diff --check` pass. An exact-pin SM120 rebuild and long-context run have not yet been completed. No model-quality change is expected from this pin update. As a runtime sanity check, generated Chinese responses were coherent, and the tool-call name and JSON arguments matched the supplied schema. --- AI assistance was used for root-cause analysis, patch preparation, validation scripting, and PR drafting with GPT-5.6-sol (low). I reviewed the complete two-file diff and validated the patched image end-to-end on the hardware described above. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, including the related issue. - [x] The test plan and commands are provided. - [x] Before/after behavior and end-to-end results are provided. - [x] Duplicate work was checked.
", "url": "https://github.com/vllm-project/vllm/pull/50796", "createdAt": "2026-08-03T03:11:01Z", "updatedAt": "2026-08-12T21:47:33Z", "timestamp": "2026-08-12T21:47:33Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "ZacharyZcR", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:6b4960e1808ad18e92a9", "signalId": "github:vllm-project/vllm:issue:52053", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52053", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Evaluate STS for online DSpark confidence alignment", "text": "### 🚀 The feature, motivation and pitch The confidence scores from a DSpark head are not always perfectly calibrated. In particular, it seems like the deeper positions have higher variance and some bias in their expectation. The DSpark paper suggests [Sequential Temperature Scaling (STS)](https://arxiv.org/html/2607.05147v1) as an online solution to calibrating these estimates. I did not see much benefit when adding this feature, but we should experiment more thoroughly and assess the potential benefit of improved confidence estimates. A comprehensive study is warranted, which includes: - A detailed assessment of confidence head bias across domains (high entropy, low entropy), prediction depth (first draft token vs n-th draft token), and draft/target model choices (DSV4 Flash 0731, DSV4 Pro, and a community-trained DSpark such as for Kimi K3) - A draft implementation of online-STS - - A study of its effect on calibration bias and variance: can we reliably rescale confidence online? Does it drift over time as dataset changes? Can a sudden change in dataset degrade performance? How should we best tune the recency-bias-coefficient? - - A study of the downstream effect on performance: is additional confidence calibration useful? How much increase in AL / throughput do we get from \"oracle\"-level perfect confidence predictions? These are my preliminary results on DSV4-Flash-0731: \"Image\" \"Image\"", "url": "https://github.com/vllm-project/vllm/issues/52053", "createdAt": "2026-08-12T21:45:26Z", "updatedAt": "2026-08-12T21:45:34Z", "timestamp": "2026-08-12T21:45:34Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "kimi", "k3" ], "author": "benchislett", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:80a35af636ec3db681b7", "signalId": "github:vllm-project/vllm:pull_request:52052", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52052", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Use silu_and_mul_with_clamp's torch._C op", "text": "This is limited to alpha=1.0 and beta=0.0 as a safety consideration due to MiniMax previously avoiding this kernel in other cases. During DeepSeekV4, day 0 support, forward_cuda was disabled in favor of forward_native for silu_and_mul_with_clamp. This can be reverted for similar accuracy and around a 6% speedup in cases without speculative decode. Example Command: vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --attention_backend ROCM_AITER_UNIFIED_ATTN \\ --compilation_config '{\"mode\":3,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}' \\ --gpu_memory_utilization 0.9 \\ --host 0.0.0.0 \\ --max_model_len 4096 \\ --max_num_seqs 512 \\ --no_enable_prefix_caching \\ --port 39817 \\ --tensor_parallel_size 4 \\ --no-enable-log-requests \\ --dtype auto \\ --kv-cache-dtype fp8 \\ --max-num-batched-tokens 8192 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --reasoning-parser deepseek_v4 \\ --tool-call-parser deepseek_v4 \\ --enable-auto-tool-choice Accuracy for gsm8k with num_fewshot=8 remained in the 0.94 range. Performance (tok/s before->after) for ISL=512, OSL=1024, temperature=0: concurrency=16: 1762.42->1883.16 (6.85%) concurrency=128: 9152.36-> 9602.00 (5%)", "url": "https://github.com/vllm-project/vllm/pull/52052", "createdAt": "2026-08-12T21:34:09Z", "updatedAt": "2026-08-12T21:34:44Z", "timestamp": "2026-08-12T21:34:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "tpopp", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:76e7da36e9bb25681714", "signalId": "github:vllm-project/vllm:issue:52051", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52051", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: gpt-oss-120b MXFP4 + --enable-expert-parallel crashes at startup: modular MoE finalize allocates padded (3072) output for an unpadded (2880) expert result", "text": "**Before submitting a new issue...** - [x] I have searched for existing and past issues, at four breadths: the symptom (padded/unpadded shape mismatch in MoE finalize), the subsystem (`gpt-oss` MXFP4 expert-parallel MoE), the faulting file/class (`modular_kernel.py`, `FusedMoEPrepareAndFinalize`), and the exact symbols (`has_unpadded_output`, `hidden_dim_unpadded`). The closest existing item is PR #44452, which restructures this same call site but never mentions MXFP4 padding or DP+EP; #42118 is the same file/class with a different defect (unswizzled mxfp8 scales). --- ### Your current environment
The output of python collect_env.py ```text `collect_env.py` output is not available: this was found by an automated CI triage agent from a batch job log, and the container is not retained after the run, so the script could not be executed retroactively. The relevant values were taken verbatim from the engine's own startup log instead: vLLM version : 0.26.1rc1.dev602+g65b7662d3 PyTorch : 2.x + CUDA 13.0 wheel (from the vLLM nightly container built on this commit; exact torch build string not printed in the log) Python : 3.12 CUDA : 13.0 NCCL : 2.30.7 in the plain nightly container; 2.29.7 in the container that additionally installs FlashInfer nightlies (pynccl prints \"vLLM is using nccl==2.30.7\" and \"...==2.29.7\" respectively). The crash is byte-identical at both NCCL versions. FlashInfer : 0.6.18.dev20260811 (flashinfer-python / -cubin / -jit-cache+cu130 nightlies) GPU : 8 x NVIDIA B200 (sm100), single node Model : openai/gpt-oss-120b, MXFP4 checkpoint (hidden_size = 2880) ``` Selected engine config echoed by vLLM at startup: ```text dtype=torch.bfloat16, max_seq_len=4096, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=8, quantization=gpt_oss_mxfp4, enforce_eager=False, load_format=runai_streamer ``` Two log lines matter for backend selection: ```text INFO [mxfp4.py:528] Using 'FLASHINFER_TRTLLM_MXFP4_BF16' Mxfp4 MoE backend. INFO [mxfp4.py:1819] Using MoEPrepareAndFinalizeNaiveDPEPModular INFO [mxfp4.py:1820] Using TrtLlmMxfp4ExpertsModular ``` The naive DP/EP all-to-all path is selected under these plain default flags: ```text INFO [cuda_communicator.py:203] Using AgRsAll2AllManager all2all manager. ``` Note on DeepEP: one of the two containers additionally logs `DeepEP v2 requires NCCL >= 2.30.4 but found 2.29.7`, but **that is not what puts this configuration on the naive path.** The other container ships NCCL 2.30.7, prints no DeepEP-unavailable message at all, and still selects `MoEPrepareAndFinalizeNaiveDPEPModular` and still crashes identically. The naive DP/EP `prepare_finalize` is simply what `--enable-expert-parallel` resolves to here under default flags, so raising NCCL does not avoid the bug.
### 🐛 Describe the bug `gpt-oss-120b` (MXFP4) fails deterministically during V1 startup memory profiling when expert parallelism is enabled on Blackwell, before the server ever becomes ready. The same build serves the model fine without `--enable-expert-parallel`. #### Reproduce On a single 8xB200 node, with the `FLASHINFER_TRTLLM_MXFP4_BF16` MoE backend selected automatically (no MoE-backend or activation override): ```console vllm serve openai/gpt-oss-120b \\ --tensor-parallel-size 1 \\ --data-parallel-size 8 \\ --enable-expert-parallel \\ --max-model-len 4096 ``` The failing ingredients are: `hidden_size` that is **not** a multiple of 256 (gpt-oss-120b is 2880), the `FLASHINFER_TRTLLM_MXFP4_BF16` backend, and a modular `prepare_finalize` that copies into a caller-provided buffer — here `MoEPrepareAndFinalizeNaiveDPEPModular`, which is what `--enable-expert-parallel` resolves to here under default flags (it is reached at NCCL 2.30.7 as well as 2.29.7, so it is not a consequence of DeepEP being unavailable). Removing `--enable-expert-parallel` avoids it; so does forcing `--quantization-config.moe.activation mxfp8`, but that changes activation precision (see \"Why the mxfp8 workaround is not a fix\" below). #### Actual behavior Every DP rank aborts in `determine_available_memory` → `profile_run` → `_dummy_run` → `gpt_oss.py forward` → `torch.ops.vllm.moe_forward`: ``` RuntimeError: The size of tensor a (3072) must match the size of tensor b (2880) at non-singleton dimension 1 File \".../vllm/model_executor/layers/fused_moe/prepare_finalize/naive_dp_ep.py\", line 207, in finalize output.copy_(get_ep_group().combine(...)) ``` 3072 is `round_up(2880, 256)`; 2880 is the model's true `hidden_size`. #### Expected behavior The engine starts and serves, as it does with EP disabled. #### Root cause The modular fused-MoE kernel allocates the finalize destination at the **padded** hidden dim, but this backend produces an **unpadded** expert output: 1. `moe_runner._maybe_pad_hidden_states` pads hidden states 2880 → 3072, because `GptOssMxfp4MoEMethod.skip_forward_padding` is `True` only for `FLASHINFER_TRTLLM_MXFP4_MXFP8` — so `..._MXFP4_BF16` gets padding. 2. `FusedMoEKernelModularImpl.apply` allocates `output = torch.empty_like(hidden_states)` → width **3072**. 3. `GptOssMxfp4MoEMethod.has_unpadded_output` is `True` for `..._MXFP4_BF16`, so `TrtLlmMxfp4ExpertsModular.workspace_shapes` returns `(M, hidden_dim_unpadded)` → `fused_out` is **2880** wide. `AgRsAll2AllManager.combine` reduce-scatters on dim 0 only, so the trailing dim stays 2880. 4. `finalize` is therefore asked to copy 2880 into 3072 → the `RuntimeError`. `FLASHINFER_TRTLLM_MXFP4_BF16` appears to be the only backend that sets padding **on** and unpadded output **on** at the same time, which is why nothing else trips this. `..._MXFP4_MXFP8` sets both `skip_forward_padding` and `has_unpadded_output`, so it never mismatches. The monolithic prepare/finalize sibling is immune because it *returns* its own tensor rather than copying into a caller buffer. Notably, `has_unpadded_output` / `hidden_dim_unpadded` do not appear anywhere in `modular_kernel.py` or under `fused_moe/prepare_finalize/`: the runner learned the padded/unpadded distinction, but the modular kernel's output allocation did not. There is even an existing `# TODO(bnell): move to MK/expert_class?` directly above the `has_unpadded_output` property in `mxfp4.py`. #### The copy is not the real defect — the allocation is Making only the `finalize` copy width-tolerant fixes the `RuntimeError` and then exposes a second, deeper failure in the compiled path: ``` File \".../torch/_inductor/output_code.py\", line 763, in __call__ return self.current_callable(inputs) File \".../inductor_cache/fi/cfi3kg....py\", line 994, in call assert_size_stride(buf8, (s72, 2880), (2880, 1), 'torch.ops.vllm.moe_forward.default') AssertionError: expected size 8192==8192, stride 3072==2880 at dim=0; expected size 3072==2880, stride 1==1 at dim=1 Error in op: torch.ops.vllm.moe_forward.default This error most often comes from a incorrect fake (aka meta) kernel for a custom op. ``` `_finalize` ends with `return output`, so the 3072-wide buffer **is** the custom op's return value, while `moe_runner._moe_forward_fake` declares the op returns `hidden_dim_unpadded` (2880). `_maybe_reduce_final_output` does strip the padding, but it runs *outside* the op, so the op boundary itself must already be 2880. The buffer must be allocated at the unpadded width, not merely copied into tolerantly. #### Suggested fix Recover the output width from the expert output once it is known, in `FusedMoEKernelModularImpl.apply` (before `self._finalize(...)`): ```python if fused_out.shape[-1] != output.shape[-1]: output = torch.empty( (*output.shape[:-1], fused_out.shape[-1]), dtype=output.dtype, device=output.device, ) ``` This appears safe: `output` reaches the experts only as `output_alias`, and `_fused_experts` ignores that alias unless `output_alias.shape == fused_out.shape` — which is false in exactly the mismatching case, so nothing has been written into the discarded buffer. It also fixes every `prepare_finalize` implementation at once rather than one `finalize()`. I have validated this change by patching it into the library in a nightly container and re-running the failing 8xB200 EP configuration: startup memory profiling completes and the server serves normally, while the non-EP configuration stays green. Happy to open a PR if maintainers prefer this shape of fix. PR #44452 (\"Delegate finalize output-buffer allocation to prepare_finalize\") restructures this exact site and recovers the width from `fused_expert_output.shape[-1]`, so it would likely fix this incidentally — but it is a broad refactor, currently open and needing a rebase, and does not mention MXFP4/padding/DP+EP. A targeted fix may be worth landing independently. #### Why the mxfp8 workaround is not a fix Setting `--quantization-config.moe.activation mxfp8` switches to `FLASHINFER_TRTLLM_MXFP4_MXFP8`, which sets `skip_forward_padding = True` and so never hits the mismatch. That masks the bug and changes activation precision; MXFP4 with BF16 activations under DP+EP remains broken. --- _This issue was drafted with assistance from the `opus` AI model._", "url": "https://github.com/vllm-project/vllm/issues/52051", "createdAt": "2026-08-12T21:33:42Z", "updatedAt": "2026-08-12T21:33:59Z", "timestamp": "2026-08-12T21:33:59Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "quantization" ], "author": "janbernloehr", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8c85a665703903b61b0b", "signalId": "github:vllm-project/vllm:pull_request:52009", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52009", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI Bug] Fix ci moe test", "text": "## Purpose Fixes https://buildkite.com/vllm/ci/builds/83443#019ff2a1-641e-4d2b-bca9-9eda8a060573 There are two errors here: 1. vLLM side, we name it triton test, but actually running the flashinfer path, this PR fixes the issue 2. the root cause of flashinfer is a bug upstream with TRT-LLM BF16 MoE, we may wait for their fix, not related to this PR ## Test Covered in CI", "url": "https://github.com/vllm-project/vllm/pull/52009", "createdAt": "2026-08-12T15:13:37Z", "updatedAt": "2026-08-12T21:28:54Z", "timestamp": "2026-08-12T21:28:54Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "yewentao256", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:ad9e9c69becefd2ce467", "signalId": "github:vllm-project/vllm:issue:52049", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52049", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Gemma 4 31B MTP - Slower T/S at high context", "text": "### Your current environment ``` Collecting environment information... ============================== System Info ============================== OS : Ubuntu 26.04 LTS (x86_64) GCC version : (Ubuntu 15.2.0-16ubuntu1) 15.2.0 Clang version : Could not collect CMake version : version 4.2.3 Libc version : glibc-2.43 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu132 Is debug build : False CUDA used to build PyTorch : 13.2 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Aug 7 2026, 02:25:39) [Clang 22.1.3 ] (64-bit runtime) Python platform : Linux-7.0.0-29-generic-x86_64-with-glibc2.43 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.3.73 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 GPU 3: NVIDIA GeForce RTX 3090 Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD Ryzen 9 7950X 16-Core Processor CPU family: 25 Model: 97 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 Frequency boost: enabled CPU(s) scaling MHz: 62% CPU max MHz: 5883.1968 CPU min MHz: 425.2920 BogoMIPS: 8999.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze Virtualization: AMD-V L1d cache: 512 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 16 MiB (16 instances) L3 cache: 64 MiB (2 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.3.5 [pip3] nvidia-cublas==13.4.0.1 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.2.75 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.2.78 [pip3] nvidia-cuda-runtime==13.2.75 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.2.0.46 [pip3] nvidia-cufile==1.17.1.22 [pip3] nvidia-curand==10.4.2.55 [pip3] nvidia-cusolver==12.2.0.1 [pip3] nvidia-cusparse==12.7.10.1 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.2 [pip3] nvidia-cutlass-dsl-libs-base==4.6.2 [pip3] nvidia-cutlass-dsl-libs-core==4.6.2 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.2 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.2 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.2.75 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu132 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cpu [pip3] torchcodec==0.15.0+cu132 [pip3] torchvision==0.28.0+cu132 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.26.1rc1.dev640+g1ab2801dd (git sha: 1ab2801dd) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PXB PXB PXB 0-31 0 N/A GPU1 PXB X PXB PXB 0-31 0 N/A GPU2 PXB PXB X PIX 0-31 0 N/A GPU3 PXB PXB PIX X 0-31 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== LD_LIBRARY_PATH=/usr/local/cuda-13/lib64: PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_westfox35 ``` ### 🐛 Describe the bug I did a quick search but couldn't find anything. Not saying there isnt anything though. I've tried this on vllm, vllm nightly and ``` uv pip install \"git+https://github.com/vllm-project/vllm.git@1ab2801ddebe31b75dd6022c69113b610bbdc950\" --torch-backend=auto ``` Initially MTP is fast (ish). When at higher contexts (say 50k+) MTP on VLLM will be very slow. When disabling MTP, it's 3x faster or higher in gen T/S. Llama.cpp and ik_llama.cpp do not suffer from this fate. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52049", "createdAt": "2026-08-12T21:24:05Z", "updatedAt": "2026-08-12T21:28:12Z", "timestamp": "2026-08-12T21:28:12Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "frenzybiscuit", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:194dee0e23c39c0bd008", "signalId": "github:vllm-project/vllm:issue:51868", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51868", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable varlen for Nemotron-H (Mamba)", "text": "### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, it is not clear if our various Mamba decode kernels are compatible with varlen execution. We should audit the flashinfer and/or triton mamba decode kernels and check if we can run decode with a ragged batch. If it's feasible, we should wire it up, validate correctness, evaluate performance, and merge support. DEPENDS ON https://github.com/vllm-project/vllm/pull/47808 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51868", "createdAt": "2026-08-11T20:06:53Z", "updatedAt": "2026-08-12T21:18:52Z", "timestamp": "2026-08-12T21:18:52Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request", "nvidia" ], "author": "benchislett", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1a7712ac2de2b253665d", "signalId": "github:vllm-project/vllm:issue:51869", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51869", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature][DSpark]: Enable varlen for Qwen 3.x (GDN)", "text": "### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches where requests have different query lengths per request. Currently, it is not clear if our various GDN decode kernels are compatible with varlen execution. We should audit the flashinfer and/or triton GDN decode kernels and check if we can run decode with a ragged batch. If it's feasible, we should wire it up, validate correctness, evaluate performance, and merge support. DEPENDS ON https://github.com/vllm-project/vllm/pull/47808 ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51869", "createdAt": "2026-08-11T20:07:42Z", "updatedAt": "2026-08-12T21:18:44Z", "timestamp": "2026-08-12T21:18:44Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "feature request", "qwen" ], "author": "benchislett", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4d4a30eb9c052a69f74c", "signalId": "github:vllm-project/vllm:pull_request:46595", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46595", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MooncakeStore] track resumed requests via scheduler's resumed_req_ids", "text": "## Summary The MooncakeStore connector tracks block tables per request so it knows whether a step's `new_block_ids` should be **appended** (normal decode) or **replace** the existing table (resumed from preemption). It was deciding this from a connector-local `_preempted_req_ids` set, which drifts from what the scheduler actually does. The scheduler already tells us the answer: `CachedRequestData.resumed_req_ids` is exactly the set of requests whose `new_block_ids` is a full replacement this step (everything else is a delta append). This PR drops the local set and reads `resumed_req_ids` directly, so the connector's append-vs-replace choice always matches how the scheduler built the block IDs. ## Symptom On a heavily warm-cache run (hybrid KV cache, multiple groups), the store send-thread throws `list index out of range` / `Store chunk out of range` from `KVCacheStoreSendingThread`. The store path indexes `block_id = block_ids[start // block_size]` while walking the logical token/hash chunks implied by `token_len`, but `token_len` is the absolute prefix length (e.g. 786432) whereas `block_ids` only covers the suffix actually held (~8K tokens), so the per-group scan walks off the end of the block table. Root cause: a request that was preempted and re-admitted stayed in `_preempted_req_ids` (nothing cleared it on that path), so on its next decode step the connector took the \"resumed\" branch and replaced `allocated_block_ids` with just the delta while `token_len` still reflected the full computed length — exactly the `token_len ≫ len(block_ids)` mismatch above. ## Fix Worker thread errors now log via `logger.exception`, so the queue-drain loop reports the full traceback and the offending `req_id` instead of just the exception message. ## Test ``` .venv/bin/python -m pytest tests/v1/kv_connector/unit/test_mooncake_store_scheduler.py -q # 14 passed ``` Added two regression tests: a running (non-resumed) request appends its delta, and a resumed request replaces a stale tracker's table — both asserting `token_len` stays covered by the block table so the store-path index can't run off the end. Before/after on the same workload at comparable cache warmth: the buggy build produced ~900 send-thread errors across a run; with this fix, zero (clean exit), while still exercising the external-store load path. ## Notes - Not a duplicate: no open PR touches the Mooncake store connector's preemption/resume tracking (searched open PRs for \"mooncake preempt resumed\" and \"resumed_req_ids connector\"). - AI assistance was used for this change.", "url": "https://github.com/vllm-project/vllm/pull/46595", "createdAt": "2026-06-24T08:31:41Z", "updatedAt": "2026-08-12T21:11:35Z", "timestamp": "2026-08-12T21:11:35Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "bug", "ready", "v1", "kv-connector" ], "author": "ivanium", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:bb5434cbac3d3f0570cd", "signalId": "github:vllm-project/vllm:pull_request:45111", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:45111", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Re-enable cross-layer KV cache layout for MLA via stride-aware kernels", "text": "## Purpose #37090 disabled the cross-layer (block-major) KV cache layout for all MLA backends after #37032 (GLM-4.7-Flash garbage output with KV offloading), attributing the bug to \"MLA kernels requiring contiguous per-layer KV cache views\". The actual cause is narrower: a few kernels computed page addresses from `block_size * entry_size` instead of reading the cache tensor's block-dim stride. The MLA write path (`concat_and_cache_mla`), the prefill gather kernels (`cp_gather_cache`, `gather_and_maybe_dequant_cache`), DeepGEMM `fp8_paged_mqa_logits`, and several decode kernels are already stride-aware. This PR fixes the three kernels that genuinely assumed packed pages and re-enables the cross-layer layout per backend (opt-in), keeping the safe identity default on `MLACommonBackend` for backends not yet verified (ROCm AITER, tokenspeed, XPU). ### Kernel fixes - `vllm/v1/attention/ops/triton_decode_attention.py`: both stage-1 decode kernels addressed the cache as `(page_number * PAGE_SIZE + offset) * stride(-3)`, baking in `stride(block) == PAGE_SIZE * stride(token)`. They now take the page-dim stride separately. This is the kernel behind the original #37032 report (A100 falls back to TRITON_MLA). - `csrc/libtorch_stable/attention/mla/sm100_cutlass_mla_kernel.cu`: `stride_C` hardcoded `page_size * (D_latent + D_rope)`; now built from `kv_c_and_k_pe_cache.stride(0)/stride(1)` (identical values for contiguous caches). - `csrc/libtorch_stable/cache_kernels.cu` `indexer_k_quant_and_cache`: block base was `block_idx * cache_block_size * kv_cache.size(2)`; now `block_idx * kv_cache.stride(0)`. Writing through a strided view previously corrupted the target layer and bled into neighbouring layers' segments. This also unblocks the DeepSeek V3.2/V4 indexer KV cache group under packed/cross-layer layouts. ### Per-backend opt-in `get_kv_cache_stride_order(include_num_layers_dimension=True)` returns `(1, 0, 2, 3)` on backends whose decode kernels verifiably honor the cache's block-dim stride: TritonMLA and CutlassMLA (fixed above), FlashAttnMLA (FA3 reads `k_batch_stride = kcache.stride(0)`), FlashMLA (dense decode reads `kcache.stride(0)`), and FlashInferMLA (verified bit-exact). `MLACommonBackend` keeps the identity permutation as the safe default, so unverified backends remain opted out and can opt in individually once verified. ### Tests `tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py` (new) runs each kernel on a contiguous cache vs a per-layer view carved from a cross-layer buffer (inflated `stride(0)`, non-zero storage offset, neighbour layers filled with garbage) and asserts bit-exact equality: `concat_and_cache_mla` write (incl. zero bleed), FlashMLA dense decode (Hopper-gated), FlashMLA dense fp8 decode (Hopper-gated), FA3 decode (Hopper-gated), FlashInfer MLA dense decode (bf16 + fp8), FlashMLA fp8 sparse decode, and `indexer_k_quant_and_cache` (incl. zero bleed). Similar strided-view tests are added for the triton decode kernels (all three address paths: MLA grouped, GQA grouped, MHA normal) and CUTLASS sm100 MLA decode. The MLA stride-order unit tests are updated for the opt-in design. ## Why this is not duplicating an existing PR Searched open PRs for cross-layer/stride-order work: #44577 packs DSv4 KV caches into contiguous per-block allocations but only touches allocation/connector/runner plumbing, no kernels — this PR is complementary (the kernel stride fixes here are what make such packed per-block layouts safe for MLA decode/write kernels). #41093 adds cross-layer support on the Mooncake connector side only. The KV-layout refactor series (#44458 draft, #44455, #42374) standardizes layout plumbing and overlaps some files but does not address the packed-page stride bugs or the MLA cross-layer opt-ins. #34742 is the stride-order default refactor referenced in #37090 review and is orthogonal. ## Test commands and results ``` pytest tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py -v # GB200 (sm100), extensions built from source: 5 passed, 3 skipped (Hopper-gated: FA3, FlashMLA dense bf16/fp8) pytest tests/kernels/attention/test_triton_decode_attention.py -v # 118 passed (incl. new cross-layer strided-view tests; no regression in existing paged/fp8 paths) pytest tests/kernels/attention/test_cutlass_mla_decode.py::test_cutlass_mla_decode_cross_layer_view -v # 1 passed on GB200 (fails against the pre-fix kernel with max diff 0.7, confirming the bug) pytest tests/v1/kv_connector/unit/test_kv_cache_layout.py -v # 7 passed ``` The three Hopper-gated tests were verified to skip cleanly here; they exercise on sm90 CI. Before the fixes, the strided-view tests reproduce the #37032 failure mode: triton/CUTLASS decode read wrong blocks for block_id > 0, and the indexer write corrupts neighbouring layers. ## AI assistance disclosure This PR was developed with AI assistance (Claude Code). The root-cause analysis, kernel fixes, and tests were reviewed line-by-line and the test suite was run on GB200 hardware by the submitter; an independent automated review (Codex) of the final diff reported no findings. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/45111", "createdAt": "2026-06-10T06:54:30Z", "updatedAt": "2026-08-12T21:11:24Z", "timestamp": "2026-08-12T21:11:24Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "ready", "v1", "kv-connector", "nvidia" ], "author": "ivanium", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:2cab2577169d6fe14c38", "signalId": "github:vllm-project/vllm:pull_request:48857", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48857", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][Triton] Reassociate softmax scale onto Q in unified attention", "text": "## Purpose Reassociate the softmax scale so ``score_scale * Q`` is loop-invariant, letting the compiler hoist the multiply out of the tile loop instead of scaling the BLOCK_M x TILE_SIZE S matrix on every iteration. Skip the reassociation for the fp8 query descale path, where folding the tiny score_scale into fp8 Q would requantize the product and destroy precision. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48857", "createdAt": "2026-07-16T16:18:46Z", "updatedAt": "2026-08-12T21:05:33Z", "timestamp": "2026-08-12T21:05:33Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "v1" ], "author": "quinnlp", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e6d5eef4cbbd3e51db74", "signalId": "github:vllm-project/vllm:issue:52042", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52042", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' / vllm 0.27.1 / DeepSeek-V4-Flash-0731", "text": "### Your current environment
The output of python collect_env.py ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (aarch64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.13.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.8.0-1059-nvidia-64k-aarch64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA GH200 144G HBM3e GPU 1: NVIDIA GH200 144G HBM3e Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : False ============================== CPU Info ============================== Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 144 On-line CPU(s) list: 0-143 Vendor ID: ARM Model name: Neoverse-V2 Model: 0 Thread(s) per core: 1 Core(s) per cluster: 72 Socket(s): - Cluster(s): 2 Stepping: r0p0 Frequency boost: disabled CPU max MHz: 3474.0000 CPU min MHz: 81.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti L1d cache: 9 MiB (144 instances) L1i cache: 9 MiB (144 instances) L2 cache: 144 MiB (144 instances) L3 cache: 228 MiB (2 instances) NUMA node(s): 18 NUMA node0 CPU(s): 0-71 NUMA node1 CPU(s): 72-143 NUMA node2 CPU(s): NUMA node3 CPU(s): NUMA node4 CPU(s): NUMA node5 CPU(s): NUMA node6 CPU(s): NUMA node7 CPU(s): NUMA node8 CPU(s): NUMA node9 CPU(s): NUMA node10 CPU(s): NUMA node11 CPU(s): NUMA node12 CPU(s): NUMA node13 CPU(s): NUMA node14 CPU(s): NUMA node15 CPU(s): NUMA node16 CPU(s): NUMA node17 CPU(s): Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.16.post3 [pip3] nccl4py==0.3.1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cccl==13.3.3.4.1 [pip3] nvidia-cuda-crt==13.3.73 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.3.73 [pip3] nvidia-cuda-nvdisasm==13.3.73 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cudnn-frontend==1.27.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-cutlass-dsl==4.6.0 [pip3] nvidia-cutlass-dsl-libs-base==4.6.0 [pip3] nvidia-cutlass-dsl-libs-core==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu12==4.6.0 [pip3] nvidia-cutlass-dsl-libs-cu13==4.6.0 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.30.7 [pip3] nvidia-nvjitlink==13.3.33 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.3.73 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.8.10.post20260721 [pip3] torch==2.13.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchcodec==0.15.0+cu130 [pip3] torchvision==0.28.0+cu130 [pip3] transformers==5.15.0 [pip3] triton==3.7.1 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.27.1 vLLM Build Flags: CUDA Archs: 8.0 8.7 8.9 9.0 10.0 11.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NODE 0-71 0 2 GPU1 NV18 X SYS 72-143 1 10 NIC0 NODE SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: roceP2p1s0f0 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=void VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/5006 NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=8.0 8.7 8.9 9.0 10.0 11.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_IMAGE_TAG=vllm/vllm-openai:v0.27.1 VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 VLLM_ENABLE_CUDA_COMPATIBILITY=0 VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/aarch64-linux-gnu VLLM_BUILD_COMMIT=6e448d0ea9bf3d88d898b65449ca6dc2aec170ac PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug `/vllm-workspace/benchmarks/kernels/benchmark_moe.py` used with DeepSeek-V4-Flash-0731 fails with `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'` To reproduce ; start v0.27.1 Docker image: ``` docker run --gpus all --ipc=host \\ --entrypoint /bin/bash \\ -it vllm/vllm-openai:v0.27.1-aarch64 ``` Run: ``` pip install ray cd /vllm-workspace/benchmarks/kernels python3 benchmark_moe.py -tp 2 --tune --trust-remote-code --model \"deepseek-ai/DeepSeek-V4-Flash-0731\" ``` Result: ``` Namespace(model='deepseek-ai/DeepSeek-V4-Flash-0731', tp_size=2, enable_expert_parallel=False, dtype='auto', use_deep_gemm=False, seed=0, batch_size=None, tune=True, trust_remote_code=True, model_prefix=None) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. INFO 08-12 20:16:45 [config.py:774] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM. Traceback (most recent call last): File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 1074, in main(args) File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 884, in main E, topk, intermediate_size, hidden_size = get_model_params(config) ^^^^^^^^^^^^^^^^^^^^^^^^ File \"/vllm-workspace/benchmarks/kernels/benchmark_moe.py\", line 822, in get_model_params E = config.num_local_experts ^^^^^^^^^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py\", line 482, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py\", line 312, in __getattribute__ return super().__getattribute__(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52042", "createdAt": "2026-08-12T20:25:18Z", "updatedAt": "2026-08-12T20:53:51Z", "timestamp": "2026-08-12T20:53:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug" ], "author": "tomschelsen", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9a33524d6383433d8aed", "signalId": "github:vllm-project/vllm:pull_request:52035", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52035", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Update DeepGEMM pin to deepseek-ai nv_dev tip", "text": "## Purpose Update both DeepGEMM pins (`cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh`, which are documented to stay in sync) from the `vllm-project/DeepGEMM` fork at `e21c821f` to upstream `deepseek-ai/DeepGEMM` at `8b1392b978f5a03c828dd1711090d7fb50958b8a`, the current tip of the `nv_dev` branch. The fork pin was kept because the plain `nv_dev` branch previously lacked SiTU support (see the removed TODO comment in the cmake file). The `nv_dev` tip now carries SiTU for FP8xFP4 MegaMoE (`situ_beta` / `situ_linear_beta` in `csrc/apis/mega.hpp`, `deep_gemm/mega/__init__.py`, `csrc/jit_kernels/impls/sm100_fp8_fp4_mega_moe.hpp`, `deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh`) alongside the SM120/SM121 kernels, so the fork indirection is no longer needed. ### Duplicate-work check Required searches were run (`gh pr list --search deepgemm`, `--search \"DeepGEMM pin\"`). Related open PRs, and why this is not a duplicate: - #51959 pins `deepseek-ai/DeepGEMM` at `a6b593d`, an older `nv_dev` commit. Verified that `a6b593d` has **no** SiTU support (no `situ` in `deep_gemm/mega/__init__.py` at that revision), so it would regress SiTU-dependent models. This PR pins the `nv_dev` tip which includes SiTU. - #51382 and #50796 keep the `vllm-project` fork and pin handcrafted `nv_dev+situ` merge/cherry-pick revisions. This PR achieves the same SM120 + SiTU combination using upstream's own `nv_dev` branch, dropping the fork dependency entirely. ## Test plan and results ```bash bash -n tools/install_deepgemm.sh # passed pre-commit run --files cmake/external_projects/deepgemm.cmake tools/install_deepgemm.sh # all applicable hooks passed; shellcheck hook # could not run (shellcheck not installed locally) git diff --check # clean ``` Additionally verified against the pinned commit that the vendoring layout consumed by `cmake/external_projects/deepgemm.cmake` is unchanged: submodules `third-party/cutlass` + `third-party/fmt`, `csrc/python_api.cpp`, and `deep_gemm/{__init__.py,utils/,testing/,legacy/,mega/,include/}` all present at `8b1392b`. A full CUDA build of the new pin and model evals were **not** run locally; requesting CI coverage. ## Disclosure This change was prepared with AI assistance (Kimi Code); the diff was reviewed line-by-line by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/52035", "createdAt": "2026-08-12T18:47:27Z", "updatedAt": "2026-08-12T20:49:04Z", "timestamp": "2026-08-12T20:49:04Z", "metrics": { "reactions": 4, "comments": 2 }, "labels": [ "ready", "ci/build", "deepseek" ], "author": "zyongye", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f589db2b6a9a244ba76f", "signalId": "github:vllm-project/vllm:issue:52034", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52034", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Mistral-Small-3.1 FP8 (Pixtral) returns NaN on image inputs with compilation enabled; works with --enforce-eager", "text": "### Your current environment
Environment - vLLM: v0.24.0 - Hardware: 1x NVIDIA H100 80GB (tensor-parallel-size 1) - Model: `RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic` (compressed-tensors FP8-dynamic; only the language-model `Linear` layers are quantized — `vision_tower`, `multi_modal_projector`, and `lm_head` are in the `ignore` list and stay bf16) - Served as `PixtralForConditionalGeneration` (via `--config-format mistral`)
### 🐛 Describe the bug With this FP8 Mistral-Small-3.1 (Pixtral) checkpoint, **text/chat/tool/structured requests work fine, but any request that includes an image returns `NaN`**. The engine does not crash; the API fails to serialize the response: ``` openai.BadRequestError: Error code: 400 - {'error': {'message': 'Out of range float values are not JSON compliant: nan', ...}} ``` Server-side, the request completes (HTTP 200 path) but the generated logits/logprobs contain `NaN`, so `json.dumps` in `create_chat_completion` raises `ValueError: Out of range float values are not JSON compliant: nan`. **Key finding: the NaN only occurs with compilation enabled. Adding `--enforce-eager` makes image requests produce valid output.** So this appears to be a torch.compile / cudagraph issue on the FP8 multimodal path rather than pure FP8 numerical instability (the quantized decoder is fine for text; it only goes NaN once image embeddings are in the sequence, and only when compiled). Text-only requests are unaffected in both modes. #### Reproduce Fails (image → NaN): ```bash vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic \\ --tokenizer-mode mistral --config-format mistral --max-model-len 32768 # then send any chat request containing an image_url -> response contains NaN -> 400 ``` Works (same request succeeds): ```bash vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic \\ --tokenizer-mode mistral --config-format mistral --max-model-len 32768 \\ --enforce-eager ``` #### Notes - The bf16 base model (`mistralai/Mistral-Small-3.1-24B-Instruct-2503`) does **not** exhibit this — only the FP8 checkpoint, and only with compilation on. - Loading this FP8 checkpoint via `--config-format hf` (`Mistral3ForConditionalGeneration`) is not a straightforward comparison point: with `--tokenizer-mode mistral` the HF `PixtralProcessor` and Mistral tokenizer disagree on the dummy `[IMG]` token at startup, and with `--tokenizer-mode auto` the Mistral tokenizer backend is selected and then `vllm/tokenizers/registry.py` calls `.is_fast` on it (`AttributeError: CachedMistralCommonBackend has no attribute is_fast`). Those look like separate issues; this report is specifically about the compiled FP8 multimodal NaN. ### Workaround `--enforce-eager` (disables torch.compile + cudagraphs), at a decode-throughput cost.", "url": "https://github.com/vllm-project/vllm/issues/52034", "createdAt": "2026-08-12T18:45:12Z", "updatedAt": "2026-08-12T20:44:23Z", "timestamp": "2026-08-12T20:44:23Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "quantization" ], "author": "wjhrdy", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:b9900dcb90ba6280caac", "signalId": "github:vllm-project/vllm:pull_request:51274", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51274", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Kimi-K3] Add opt-in gfx942 MXFP4-to-int4 conversion", "text": "## Summary Add an explicit `int4_per_group_32` MoE override for Kimi-K3 on gfx942. When selected, the MXFP4 expert weights are requantized once at load time and served through AITER's replacement BF16 x packed-int4 FlyDSL path. The conversion is lossy and never happens by architecture alone. The default MXFP4 path is unchanged. ## Why gfx942 does not support the scaled FP4 conversion instructions used by the native MXFP4 path. This opt-in path pays the conversion cost once during model loading and then uses the replacement packed-int4 kernels from [ROCm/aiter#4646](https://github.com/ROCm/aiter/pull/4646). ## Changes - Register `int4_per_group_32` as an online MoE weight quantization key. - Require the explicit `--quantization-config.moe.weight int4_per_group_32` override on gfx942. - Require the replacement A16W4/int4 AITER entry point and fail closed unless it supports SiTUv2. - Convert eight experts at a time and release each source tensor before converting the next one, bounding transient memory during load. - Log a warning that the selected conversion is lossy. ## Validation Unit tests: ```text tests/quantization/test_quantization_config_args.py: 15 passed tests/models/kimi_k3/test_gfx942_int4.py: 5 passed ``` Strict AITER comparison against the torch reference passed for the production EP shape: ```text model_dim=3584 intermediate_dim=3072 local_experts=112 topk=15 tokens=1,4,7,16,32 activation=SiTUv2 beta=4.0 linear_beta=25.0 ``` Target-only integration on 8x MI325X with TP8, expert parallelism, Triton MLA, and full plus piecewise graph capture: - Graph capture completed successfully. - Three deterministic chat sanity checks returned correct answers. - Full five-shot GSM8K scored 0.960576 strict and flexible on all 1,319 questions, with 0.00536 standard error. - Three 1K input / 128 output serving sweeps at concurrency 1 and 16 completed 510/510 requests without server errors. The previous long-context results used the implementation superseded by ROCm/aiter#4646, so long-context validation against the replacement AITER release remains pending. ## Dependency and scope This remains a draft until ROCm/aiter#4646 is merged and available in an AITER release used by vLLM. The conversion is an explicit opt-in for deployments that accept a lossy one-time weight conversion. It does not change the default checkpoint path. ## Test plan - [x] Quantization key and opt-in selection unit tests. - [x] Strict production-shape comparison against the torch reference. - [x] Full 1,319-question GSM8K. - [x] Target-only graph capture and serving sanity. - [ ] Long-context and DSpark validation against the AITER release containing ROCm/aiter#4646. - [ ] ROCm CI on the rebased draft.", "url": "https://github.com/vllm-project/vllm/pull/51274", "createdAt": "2026-08-06T15:09:53Z", "updatedAt": "2026-08-12T20:42:42Z", "timestamp": "2026-08-12T20:42:42Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "rocm", "quantization", "kimi", "k3" ], "author": "maeehart", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b14d05b5212a2f0c309d", "signalId": "github:vllm-project/vllm:pull_request:50382", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50382", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DCP] Default query replication for GLM sparse attention", "text": "## Summary - add `--dcp-q-replicate` and `--no-dcp-q-replicate` as explicit CLI options; - preserve `VLLM_DCP_Q_REPLICATE` as the highest-precedence override; - enable query replication automatically for `glm_moe_dsa` models when decode DCP is enabled and PCP is not enabled; - keep query replication opt-in for other MLA model families. The automatic default removes GLM's decode query all-gather at the cost of replicating the query projection weights across DCP ranks. Explicit CLI or environment selections still override the model default. ## Performance and memory GLM-5.2-NVFP4, TP=4, DCP=4, 32K input tokens, 128 output tokens, batch 8, two warmups, five measured iterations: - QREP off: 20.1777 s - QREP on: 19.7426 s - delta: -0.4351 s (-2.16%) With QREP enabled, a fresh DCP backend comparison measured: | DCP backend | Average latency | | --- | ---: | | `ag_rs` | 19.7636 s | | `a2a` | 18.0213 s | For the matched 32K benchmark configuration, model-loading memory increased from 110.37 GiB/rank to 114.94 GiB/rank with QREP, approximately +4.57 GiB/rank. ## Model evaluation GLM-5.2-NVFP4, TP=4, DCP=4, GSM8K 5-shot, 300 questions: - automatic GLM QREP default: 94.0% accuracy, 0% invalid outputs; - explicit QREP off: 95.0% accuracy, 0% invalid outputs. The QREP-off evaluation used `max_num_seqs=32` because FlashInfer's sparse-MLA workspace overflowed when all prompts were scheduled concurrently. Both runs evaluated the same 300 questions with the same decoding settings. The one-point difference is not significant at this sample size. ## AI assistance AI assistance was used for implementation, tests, benchmarking, profiling analysis, and drafting this description. The human submitter has reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50382", "createdAt": "2026-07-30T04:48:07Z", "updatedAt": "2026-08-12T20:35:00Z", "timestamp": "2026-08-12T20:35:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "deepseek" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e84b1d48cea7d86a62e9", "signalId": "github:vllm-project/vllm:pull_request:51159", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Defer `tilelang` import through its import `from vllm.tilelang_utils import tilelang` and relaxed `has_tilelang`", "text": "## Purpose Fixes https://github.com/vllm-project/vllm/issues/51151 This PR introduces `vllm.tilelang_utils` and prevents direct `tilelang` imports. Avoid importing TileLang during ROCm module import, because importing TileLang can load **wrongful/bugged** TVM and HIP stub symbols into the global process scope before AITER loads its JIT modules. This changes `_tilelang_jit` so ROCm applies `tilelang.jit` lazily on first kernel call, while non-ROCm keeps eager JIT decoration. This PR is AI-assisted. Proper fixes in followups need to be: * a PR in `tilelang` or TVM to fix the wrongful symbol load. * a PR in vLLM so that `_has_module` does not import modules https://github.com/vllm-project/vllm/issues/51162 ## Test Plan - `python tools/pre_commit/check_forbidden_imports.py --test-regex` - `pytest tests/kernels/test_mhc_tilelang_jit.py -v` Added new test to verify that CUDA applies `tilelang.jit` eagerly, ROCm defers `tilelang.jit` until first call, and the ROCm wrapper caches the compiled kernel. ```python VLLM_ROCM_USE_AITER=1 HIP_VISIBLE_DEVICES=1 pytest tests/quantization/test_online.py::test_online_quantization[True-fp8_per_tensor-None-Fp8PerTensorOnlineLinearMethod-Fp8PerTensorOnlineMoEMethod] -s -vvvvv ``` that is failing on `main` following https://github.com/vllm-project/vllm/pull/50879, and which surfaces the issue https://github.com/vllm-project/vllm/issues/51151 ## Test Result `test_mhc_tilelang_jit.py` pass on MI350. `tests/quantization/test_online.py` as well on MI300 (previous failing case) `python tools/pre_commit/check_forbidden_imports.py --test-regex` passes as well.", "url": "https://github.com/vllm-project/vllm/pull/51159", "createdAt": "2026-08-05T12:25:31Z", "updatedAt": "2026-08-12T21:55:59Z", "timestamp": "2026-08-12T21:55:59Z", "metrics": { "reactions": 2, "comments": 20 }, "labels": [ "documentation", "rocm" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c51da5d5d4862c57689b", "signalId": "github:vllm-project/vllm:pull_request:51084", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51084", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Profiler] Add Proton CUDA graph attribution", "text": "## Purpose Extend the Proton backend merged in #48789 with opt-in CUDA graph replay attribution. Proton must observe graph creation using the same Data object that later receives replay events. Setting `proton_graph_attribution` to true therefore creates the user-configured Proton session immediately before vLLM enters CUDA graph capture, keeps that session active throughout capture, deactivates it with flushing after capture, and reactivates the same session when profiling starts. - Add `proton_graph_attribution`, disabled by default. - Require `proton_data=\"tree\"` when graph attribution is enabled. - Preserve the configured context, backend, mode, hook, and output format during capture. - Reuse one Proton session for graph capture and replay profiling. - Use Triton phases to discard graph-capture activity before request profiling. - Keep the session alive across repeated start/stop cycles; each stop advances the phase, flushes, deactivates, writes that phase, and clears it. - Avoid Proton initialization during CUDA graph capture unless graph attribution is explicitly enabled and graphs actually need capture. - Reject PC sampling with graph attribution because that mode synchronizes the CUDA context. - Require Triton 3.7 or newer for Proton profiling. No option is rewritten for CUDA graph capture, and no dedicated capture-only Proton session is created. Without `proton_graph_attribution`, Proton retains its normal endpoint behavior: each `/start_profile` creates an independent profiling session, and each `/stop_profile` finalizes it and writes the profile immediately. ## Scope This PR only adds opt-in CUDA graph capture and replay attribution for the V2 model runner. It does not add distributed rollback, HTTP error mapping, numeric trace metadata, or ROCm support. Those concerns remain separate follow-ups. ## CUDA graph attribution example Start vLLM with the V2 model runner and opt in to graph attribution. Tree data is required because Proton stores the capture/replay relationship in its tree representation. ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve facebook/opt-125m \\ --profiler-config '{ \"profiler\": \"proton\", \"proton_profiler_dir\": \"./proton_profile\", \"proton_data\": \"tree\", \"proton_output_format\": \"hatchet\", \"proton_hook\": \"triton\", \"proton_graph_attribution\": true }' ``` Profile a request through the existing endpoints: ```bash curl -X POST http://localhost:8000/start_profile curl http://localhost:8000/v1/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"facebook/opt-125m\", \"prompt\": \"CUDA graph replay\", \"max_tokens\": 32 }' curl -X POST http://localhost:8000/stop_profile ``` The stop call flushes the completed phase and immediately writes a rank-qualified Hatchet file under `./proton_profile`. In that file, kernels executed through CUDA graph replay are attributed to the scopes recorded when vLLM captured the graph. The same server can profile another interval without losing graph attribution: ```bash curl -X POST http://localhost:8000/start_profile # Send another inference request. curl -X POST http://localhost:8000/stop_profile ls ./proton_profile/*.hatchet proton-viewer -m time/ns \\ ./proton_profile/proton_dp0_pp0_tp0_dcp0_ep0_rank0_pid1234__run0.hatchet ``` `proton-viewer` displays the attributed tree, including the `` scopes that connect replayed kernels to CUDA graph capture. Each start/stop cycle writes a distinct profile. The long-lived Proton session retains only the graph metadata needed for replay attribution; activity from graph capture and completed profiling phases is cleared. ## Impact Model outputs and serving semantics are unchanged. Users who enable `proton_graph_attribution` can attribute kernels executed through vLLM CUDA graph replay while preserving repeated `/start_profile` and `/stop_profile` semantics. ## Test results - Focused worker profiler suite: 75 passed. - All applicable changed-file pre-commit hooks passed, including Ruff, markdownlint, repository checks, and mypy. - Real NVIDIA H20 FULL/FULL_AND_PIECEWISE CUDA graph smoke test with `facebook/opt-125m` and `proton_graph_attribution=true`: - Two consecutive `start_profile` / graph-replay generation / `stop_profile` cycles returned HTTP 200. - Each stop immediately wrote a distinct Hatchet profile. - Both profiles contained `` attribution entries. - Earlier configuration coverage confirmed that graph attribution works with both shadow and python context sources; graph-attributed output is intentionally restricted to TreeData because Triton's phase export API does not expose TraceData. AI assistance disclosure: OpenAI Codex assisted with implementation, code review, local test orchestration, splitting the original change, and PR preparation. The human submitter remains responsible for reviewing and understanding every changed line before merge.", "url": "https://github.com/vllm-project/vllm/pull/51084", "createdAt": "2026-08-04T22:09:08Z", "updatedAt": "2026-08-12T20:27:29Z", "timestamp": "2026-08-12T20:27:29Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "performance", "nvidia" ], "author": "Luosuu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dd6abca23407270ffe02", "signalId": "github:vllm-project/vllm:pull_request:41834", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:41834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes", "text": "## Summary This PR enables DeepSeek V4 Flash on SM120/SM121 Blackwell client hardware by carrying the SM12x fallback and tuning stack needed for the current vLLM V1 path. It targets RTX PRO 6000 Blackwell Workstation Edition, RTX 5090-class SM120, and GB10 / DGX Spark SM121 users who cannot use SM100-only TMEM / `tcgen05` kernels. The branch is **reconciled on top of the merged #43477** and provides the **stock-deps path**: DeepSeek V4 on SM120/121 that builds and serves on *released* FlashInfer / DeepGEMM wheels, complementing #43477's route that needs the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches. It is kept **synced onto current `upstream/main`**. **Latest validated head: tag `sm120-pr-41834-stable-preview-20260809` (`aa0d513027`), synced onto `upstream/main` as of 2026-08-09 (`f18e10a7e1`)** — see [Update 2026-08-09](#update-2026-08-09) below. **The default model runner is now V2**; `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Model / speculative-decode status.** `deepseek-ai/DeepSeek-V4-Flash-0731` is the checkpoint this branch is validated on. It **removed the MTP heads** and folded the DSpark draft into the main checkpoint, so **DSpark (`method: \"dspark\"`, `num_speculative_tokens: 5`) is the speculative path**; MTP is supported only for older checkpoints that still carry those weights. Running without speculation is also fully supported and validated. ## Change footprint — model kernels vs. core-vLLM touch points 187 files, ~+29.1k / −1.3k against `upstream/main`, of which ~10.7k added lines are tests. The branch splits cleanly into model/kernel code and a small set of core-vLLM integration points: - **DeepSeek-V4 model + SM12x kernels — the enablement itself.** Everything under `vllm/models/deepseek_v4/**` plus the SM12x sparse-MLA decode / indexer / DeepGEMM kernels that live in shared dirs (`v1/attention/backends/mla/sparse_mla_kernels.py`, `model_executor/layers/sparse_attn_indexer.py`, `v1/attention/backends/mla/{indexer,sparse_swa}.py`, `utils/deep_gemm.py`, `kernels/mhc/tilelang.py`), the new DSv4 reasoning parser / tokenizer, and device tuning JSONs. - **C128A metadata device→host sync removed** (`models/deepseek_v4/sparse_mla.py`, perf) — `_c128a_effective_topk_width` takes the max position from the CPU-side `CommonAttentionMetadata.max_seq_len` instead of a per-step `int(positions.max().item())` device sync, dropping a launch-stream stall on every C128A metadata step. Decode is identical (`max_seq_len-1 == positions.max()`); only chunked prefill sees a safe, slightly-wider 128-aligned top-k. - **Core-vLLM integration — the hooks below.** Almost all are gated by model architecture / quant config / an env flag and are inert for other models. | Subsystem | Files | What it does | | --- | --- | --- | | KV-cache core | `single_type_kv_cache_manager.py`, `kv_cache_coordinator.py`, `kv_cache_manager.py`, `sched/scheduler.py` (+1) | prefix-cache correctness for DSv4 sparse-MLA + speculative decode: an MLA cache-manager with prompt-block protection, a hybrid-coordinator `cache_blocks` tail-block-reuse rewrite | | Speculative decode | `v1/spec_decode/{dspark,dspark_sampling,llm_base_proposer,dflash}.py`, `config/speculative.py` | DSpark self-drafting proposer + sampling; DSv4 probabilistic draft sampling and per-step draft-layer routing in the shared proposer base; DSpark/MTP method detection and validation | | MoE quantization | `fused_moe.py`, `oracle/mxfp4.py`, `routed_experts.py`, `experts/flashinfer_cutlass_moe.py`, `quantization/mxfp4.py`, `oracle/nvfp4.py` | MXFP4 / NVFP4 backend selection; the one-line NVFP4 fix (FLASHINFER_CUTLASS into the SwiGLU-clamp allow-list) lets DSv4-Flash-NVFP4 serve | | FP8 / Marlin GEMM | `quantization/utils/fp8_utils.py`, `linear/scaled_mm/{cutlass,marlin}.py`, `csrc/.../marlin_moe_wna16/ops.cu` (the only C++) | SM12x e8m0→fp32 upcast + Marlin MoE SM12.0a cudagraph hardening | | cudagraph / compile / config | `config/vllm.py`, `compilation/breakable_cudagraph.py`, `passes/utility/fix_functionalization.py`, `config/compilation.py` | breakable-cudagraph auto-enable gate (MiniMax-only; DSv4 deliberately excluded), DSv4 custom-op defunctionalization + splitting-op registration | | OpenAI entrypoints / parsers | `chat_completion/protocol.py`, `serve/render/serving.py`, `tool_parsers/structural_tag_registry.py`, `chat_utils.py`, `engine/protocol.py`, `chat_completion/{serving,batch_serving}.py`, `reasoning/__init__.py` | expose DSv4 API semantics — `reasoning_content` / `thinking` param / tool-call streaming (jasl/vllm#19 instruction-following) | | Kernel warmup | `model_executor/warmup/deepseek_v4_sm12x_warmup.py` (new), `kernel_warmup.py` (+11) | DSv4 warmup passes (D512-split prefill precompile, paged-MQA rowwise, draft path) that avoid JIT-during-inference wedges. Kept in a separate module so `kernel_warmup.py` stays a two-line hook on upstream's file | | Weight loading | `weight_utils.py`, `default_loader.py` | fast-safetensors weight filter + EP-skip (lowers DSv4 load overhead on GB10) | | env / utils | `envs.py`, `utils/flashinfer.py`, `utils/import_utils.py`, `v1/worker/{gpu_model_runner,ubatch_utils}.py` | `VLLM_DEEPSEEK_V4_*` flags + `has_cutedsl` / `has_flashinfer_trtllm_sparse_mla` probes | Two notes for review: - The most invasive generic edits were **removed** in the 2026-06-21 audit cleanup: the scheduler carries a single +1-line change (the prefill-fairness heuristics were dropped) and the prefix-cache write-fence is gone. - A few hooks touch code paths shared with non-DSv4 models and are worth a closer look: the `kv_cache_coordinator` `cache_blocks` rewrite (affects hybrid-KV models; validated ≥ prior behavior), the proposer base-class change, and the OpenAI-entrypoint plumbing. Everything else (MoE oracle, fp8_utils, cudagraph gate, warmup, envs) is arch / quant / env-gated and inert for other models. ## Duplicate-work check The nearest open/merged PRs are related but not duplicates: | PR | Difference | | --- | --- | | #43477 | **Merged 2026-06-22.** Enables DeepSeek V4 + GLM-5.1 on SM120 via the FlashInfer-SM120 sparse-MLA route, but on its merged form **requires the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches** — on released/stock wheels its SM12x path raises at model construction. This PR is **reconciled on top of #43477** (merge `42657aca65`) and carries the **stock-deps** DSv4 SM120/121 path that runs on released wheels. | | #40929 | Earlier WIP Triton fallback effort. This PR is the maintained replacement branch with the broader scheduler, prefix-cache, parser, quant, warmup, and harness-validated fixes carried forward. | | #42856 | Focused workspace-bound fix that explicitly depends on / references this PR; a subset-style bugfix, not the full DeepSeek V4 SM12x enablement branch. | | #49335 | mxfp8 activation-scale swizzle after DP/EP dispatch — **carried in this branch** (unclaimed upstream). Inert at DP=1; taken for this branch's multi-node DP users. | | #50686 | Consecutive-assistant-message merging in DSv4 prompt encoding — **carried in this branch** (reproduced here before taking it). | | #50693 | B300-targeted prefill-workspace fix. **Test carried, code not needed**: this branch's `_prefill_workspace_topk_bound` returns early for `compress_ratio <= 1` and never reaches the affected buffer. | Upstream PRs whose fixes this branch previously carried as local deltas and has since **retired** in favour of upstream's own version: #48304, #48911, #48959 (via #49052). ## Fixed preview tags These tags are in `jasl/vllm` and give users stable pins while the PR is still moving: | Tag | Commit | Notes | | --- | --- | --- | | `sm120-pr-41834-stable-preview-20260809` | `aa0d513027` | **latest validated head** — 87 upstream commits incl. FlashInfer 0.6.16.post3; the V2 recall collapse root-caused as a prefix-cache ghost-block race and fixed (port of #42359); **default runner switched to V2**. See **Update 2026-08-09**. | | `sm120-pr-41834-stable-preview-20260804` | `0f59188db1` | 35 further upstream commits, four fixes from community reports (DSpark out-of-vocab draft token, eager scratch pool), two contributor PRs. Validated on **both** SM121 and SM120. See **Update 2026-08-04**. | | `sm120-pr-41834-stable-preview-20260802` | `9a94c54292` | 234 upstream commits, `DeepSeek-V4-Flash-0731` support, two DSpark config fixes, #49335 / #50686 absorbed. See **Update 2026-08-02**. | | `sm120-pr-41834-stable-preview-20260727d` | `d64074e6f0` | 209-commit upstream sync + torch 2.13 (tag `…-20260727`, `70a33886bd`); DSpark VRAM work (jasl/vllm#27) merged; bounded block-table gather in `compute_global_topk_indices_and_lens`. | | `sm120-pr-41834-stable-preview-20260721` | `832775efd1` | 79-commit upstream sync; #48911 dropped in favour of upstream's merged version; compact CPU KV offload (opt-in). | | `sm120-pr-41834-stable-preview-20260717` | `f63bfd3d7b` | 195-commit upstream sync; prefill ctx_pp +4.7% @ d8192. | | `sm120-pr-41834-stable-preview-20260711` | `b5c0d43b96` | 181-commit upstream sync; #48304 MTP unscaled-draft-rope; ~1097-line dead-kernel cleanup. | | `sm120-pr-41834-stable-preview-20260704` | `b43470e871` | @GanyX19 GB10 fixes: per-shape `constexpr`→runtime (stops the Triton recompile → unified-memory leak → hard-freeze) + fp8-einsum `tl.multiple_of(16)` (~24% decode @256K). | | `sm120-pr-41834-stable-preview-20260703` | `444fe3ac8b` | DSpark spec-decode (self-drafting block-5), V2 padded-Q OOM fix (jasl/vllm#26), exact non-cooperative `persistent_topk` for <128 KB-smem parts. | Older tags (`…-20260705` back to `…-20260612…`) remain in `jasl/vllm` for history. ## Update 2026-08-02 — `DeepSeek-V4-Flash-0731`, 234 upstream commits, two DSpark fixes Validated head **`9a94c54292`** (tag `sm120-pr-41834-stable-preview-20260802`), 234 upstream commits absorbed, level with `upstream/main` as of 2026-08-02. **What's in it** - **`DeepSeek-V4-Flash-0731` support.** The new checkpoint ships **no MTP heads** — `enorm`, `hnorm`, `e_proj`, `h_proj` and `shared_head` are absent from the weight index, and `mtp.{0,1,2}.*` now carries the DSpark-style `main_norm` / `main_proj` structure (matching `dspark_target_layer_ids: [40, 41, 42]`). DSpark is the speculative path going forward; the MTP code is retained for older checkpoints. - **`num_speculative_tokens` vs `dspark_block_size`** — *this rule was relaxed on 2026-08-04; see Update 2026-08-04. It now errors only BELOW the block size and warns above it.* The original reasoning and measurements follow. - The validator was tightened to require equality. The validator previously accepted `>=` and its error message recommended exceeding it. The drafter emits exactly one block per pass, so the extra slots are structurally unreachable — measured on a prose workload, the **7th draft position accepted 0.000 in every sample** (the 6th in all but one, 0.004 there), and `nst=7` drafts 40% more tokens per step for strictly worse acceptance: | configuration | mean acceptance length (3 samples) | avg draft acceptance rate | | --- | --- | --- | | nst=5 probabilistic | 2.15 / 2.16 / **2.19** | 22.9 / 23.2 / **23.8%** | | nst=7 probabilistic | 1.61 / 1.75 / 1.95 | 8.7 / 10.7 / 13.6% | | nst=5 greedy | 1.82 / 2.06 / **2.23** | 16.4 / 21.2 / **24.5%** | | nst=7 greedy | 1.57 / 1.66 / 1.75 | 8.2 / 9.5 / 10.8% | All samples are shown rather than a single figure: the probe reads whatever `SpecDecoding metrics` lines vLLM flushed inside its window, so a low sample means \"not much steady traffic in that slice\", not a worse drafter. Both `nst=7` runs also hit connection errors partway through, so their spread is noisier. - **`method: \"mtp\"` is no longer silently rewritten to `\"dspark\"`.** Auto-detection preserved an explicitly requested method only for `eagle`/`eagle3`/`dflash`/`dspark`. Since 0731 puts `dspark_block_size` in every DSv4 config, `method: \"mtp\"` fell through to the dspark branch, was rewritten, and then failed validation with a DSpark message the user never asked for. - **Absorbed** #49335 (mxfp8 activation-scale swizzle after DP/EP dispatch, verified non-vacuous) and #50686 (consecutive assistant messages merged in prompt encoding — it also removes a spurious mid-turn `<|end_of_sentence|>` on the default `drop_thinking=True` path). #50693's regression test is carried; its code fix is not reachable here. - **Three community reports re-tested and not reproduced** on this head: nst=7 draft-position collapse, TP=4 + expert-parallel `Check failed: num_tokens > 64`, and `FLASHMLA_SPARSE_DSV4` missing `tile_sched`. Details in [this comment](https://github.com/vllm-project/vllm/pull/41834#issuecomment-5152562124). **Validation (GB10 SM121, 2-node TP=2, `DeepSeek-V4-Flash-0731`, torch 2.13.0, FlashInfer 0.6.15.post1, nccl 2.30.7)** | | DSpark nst=5 | no speculation | | --- | --- | --- | | GSM8K 8-shot (flexible) | 0.9394 | 0.9500 | | GSM8K 8-shot (strict) | 0.9363 | 0.9484 | | instruction-following (jasl/vllm#19, JSON-only) | PASS | PASS | | long-context recall (arthur needle, c=1) | 2/2 | 2/2 | | illegal-access / assertion in serve log | 0 | 0 | | draft acceptance (prose) | mean 2.08, 21.7% | — | The GSM8K difference (1.06 pp flexible / 1.21 pp strict) is within this gate's measured single-run spread (~1.1 pp). **Resolved:** three runs per cell were collected and the arms interleave, so it was noise. `0731` is the first checkpoint where the strict and flexible extractors disagree at all; on every prior baseline they were identical. Perf — pinned llama-benchy standard (fp8 KV, prefix-cache on, `FULL_AND_PIECEWISE`, mml 49152, util 0.85; C=1, 3 runs), against the **full** recorded range of the prior MTP2 baselines. This crosses a checkpoint boundary, so read it as a sanity band rather than a controlled A/B: | metric | prior MTP2 range (n=10) | 0731 + DSpark | vs band | | --- | --- | --- | --- | | pp2048 @ d8192 | 1339.11 – 1400.81 | **1432.23** ± 11.74 | **above** | | pp2048 @ d16384 | 1308.77 – 1344.68 | **1356.56** ± 11.78 | **above** | | pp2048 @ d32768 | 1089.05 – 1226.63 | **1250.75** ± 2.18 | **above** | | ctx_pp @ d8192 | 1757.16 – 1876.01 | 1816.97 ± 5.89 | inside | | ctx_pp @ d16384 | 1769.85 – 1842.16 | 1817.43 ± 1.43 | inside | | ctx_pp @ d32768 | 1595.87 – 1756.01 | 1740.22 ± 2.87 | inside | | tg128 @ d8192 | 36.27 – 43.08 | 41.72 ± 5.09 | inside | | tg128 @ d16384 | 34.59 – 43.14 | 37.92 ± 9.92 | inside | | tg128 @ d32768 | 32.77 – 42.91 | 34.88 ± 5.78 | inside | | ctx_tg @ d8192 | 38.52 – 43.01 | 39.37 ± 2.34 | inside | | ctx_tg @ d16384 | 39.29 – 43.07 | **35.07** ± 0.67 | **below, −10.7%** | | ctx_tg @ d32768 | 38.02 – 42.73 | 40.70 ± 6.85 | inside | **Batched prefill (pp2048) is above the historical band at all three depths** (+2.2% / +0.9% / +2.0%) — the only consistent directional move here. Clearing the max of ten prior runs at all three depths says more than any single one of those margins would: +0.9% is inside this metric's own resolution, so read the consistency rather than the magnitudes. No sign of DSpark being slower than MTP2 was. One caveat reported rather than buried: **`ctx_tg @ d16384` sits 10.7% below its historical minimum**, the only metric outside its band. It is non-monotonic against our own neighbouring depths (39.37 at d8192, 40.70 at d32768, where history has d16384 ≈ d8192), which points at a single-run artifact rather than a depth-specific regression. **Resolved:** repeated on later heads and it did not recur. A measurement caveat for anyone benchmarking this branch: the `±` in a benchy row is the spread of the three runs *inside one invocation*, and it runs 5–30× smaller than the build-to-build spread. This branch's own history spans 31% on tg128 @ d32768 and ~1.3% on ctx_pp, so anything under ~15% on tg or ~2% on ctx_pp is not resolvable this way. ## Update 2026-08-04 — four fixes from community reports, 35 upstream commits, and first SM120 validation Validated head **`0f59188db1`** (tag `sm120-pr-41834-stable-preview-20260804`). **This is the first head validated on both SM121 and SM120.** Every SM120 discrete-GPU result on this PR up to now was a contributor's measurement we could not reproduce. We have since rebuilt a 2× RTX PRO 6000 Blackwell box as a first-party SM120 target. ### Fixes - **DSpark's fused Markov sampler could emit an out-of-vocab token id** (`e171c51036`). `_dspark_markov_probs_blocks_kernel` stores `vocab_size` as the filler for a block with no active lane. On a fully-masked row — every candidate `-inf`, which structured-output constraints can produce — *no* block has an active lane, so every block stores the filler and the reduce kernel returns it verbatim as the sampled token. Nothing downstream bounded it: the runner clamped `input_ids` with `min=0` only, and the DSv4 hash-MoE router indexes `tid2eid[token_id * 6 + lane]` on a `[vocab_size, 6]` table. Result is an illegal memory access on every TP rank. This is the producer on the **V1** path, which is this branch's default. @alexbi29's report traced the same class of defect to the V2 samplers (#50843) — a real defect, but a different tree. Fixed by folding out-of-range to `0` (matching `torch.argmax` on such a row, so the fused kernel stays bit-identical to the eager reference) and making the runner clamp two-sided. Worth stating plainly for anyone with similar gates: **our own gates could not have caught this**. The fused path is skipped when `all_greedy`, and both our long-context recall gate and GSM8K are greedy, so they are structurally incapable of executing that kernel. The new regression test is explicitly non-greedy. - **Adopted #50844** (`3df857ba50`) — bound `token_id` before the `tid2eid` gather. Defence in depth; `prompt_token_ids` reach that gather directly when `--skip-tokenizer-init` disables the engine's vocab check. **Not** taking #50843 (V2-tree only, inert on our default) or #50845, which has a defect reported on its own thread. - **Eager scratch pool is now OFF by default** (`d42b8d9f55`, `b1ef3033f4`), opt-in via `VLLM_DEEPSEEK_V4_EAGER_SCRATCH_POOL=1`. @tobymao bisected output corruption under concurrent mixed prefill+decode to it: pool active 7/7 rounds corrupt, disabled 0/2. We first removed the cross-template aliasing (`max()` → `sum()` sizing with per-family offsets); **they tested that commit directly and it was still corrupt in round 1**. Their diagnosis is the useful part: the pre-pool code was race-free *for free* because per-call transients go through the caching allocator, whose cross-stream reuse is event-guarded — the pool reuses memory without that machinery, so no static partitioning fixes it. Making the cross-layer reuse safe needs producer-waits-on-consumer events against the real stream graph; until then, off by default. - **Two contributor PRs merged** — jasl/vllm#37 (tuned FP8 W8A8 config for `N=4096,K=12288` on RTX PRO 6000) and jasl/vllm#38 (hoist the E8M0 block-scale upcast out of the FP8 GEMM hot path, 13,561 kernel launches removed per 25 decode steps), both from @alexbi29. - **`num_speculative_tokens` rule relaxed.** Upstream removed its own assertion in #50869 as \"invalid\". They were right that erroring *above* `dspark_block_size` is wrong — two users on this thread run `nst=7` against `block_size=5` and it demonstrably works. The two directions are not symmetric, so this branch now **errors below** the block size (that genuinely garbles output) and **warns above** it, quoting the acceptance cost. Strictly more permissive than what shipped before. ### Validation Full gate battery on **both** architectures, same branch: | gate | SM121 (2× GB10, 2-node TP=2) | SM120 (2× RTX PRO 6000, TP=2) | | --- | --- | --- | | serve, DSpark nst=5, `--block-size 256` | ✅ | ✅ | | instruction-following (jasl/vllm#19) | PASS | PASS | | long-context recall, arthur c=1 | 2/2 | 2/2 | | long-context recall, arthur c=12 | 22, 23, 22 / 24 | 22, 23 / 24 | | GSM8K 8-shot flexible | 0.9484 / 0.9507 / 0.9492 | 0.9371 | | GSM8K 8-shot strict | 0.9462 / 0.9477 / 0.9462 | 0.9303 | | tool-calling, 135 cases | 256/270 (94.8%) | — | | illegal-access / assertion lines | 0 | 0 | The ~1.1 pp GSM8K difference between architectures sits inside this gate's measured single-run spread and spans different silicon, different memory architecture and a 3× smaller KV cache (6.25 GiB vs ~18.5 GiB). We are not claiming a difference from it. **`Check failed: num_tokens > 64` does not reproduce on this branch.** @fuzzifikation reported stock 0.26.0 dying there on SM120 at `--block-size 256`, correctly tracing it to the DSv4 decode dispatch requiring `page_block_size == 64`. On our SM120 box, at the same `--block-size 256`, the serve comes up and the assertion never appears — the DSv4 packed KV cache is laid out in 64-token pages independent of vLLM's logical block size, and FlashInfer derives `page_block_size` from tensor geometry rather than the engine config. The SM120 packed decode path is confirmed engaged in the same run. Note the same assertion has **two** distinct gates (`page_block_size` and `(num_heads, topk)`, the latter being #50720 / flashinfer#3989), so patching one and still seeing it means checking the other. ### Prefill: V1 vs V2 model runner The 2026-08-02 V1-vs-V2 comparison never measured throughput. It has now been measured, blocked and pre-registered — 10 blocks, both arms inside each node pair, exact sign-flip permutation test, Holm-corrected across the six prefill cells, with the decision rule committed before any data was collected: | metric | V2 / V1 | 95% CI | exact p | | --- | --- | --- | --- | | ctx_pp @ d8192 | **+1.18%** | [+0.82, +1.53] | 0.0020 | | ctx_pp @ d16384 | **+1.11%** | [+0.35, +1.87] | 0.0137 | | ctx_pp @ d32768 | **+1.61%** | [+1.17, +2.04] | 0.0020 | | pp2048 @ d8192 | **+4.18%** | [+3.28, +5.09] | 0.0020 | | pp2048 @ d16384 | **+4.18%** | [+3.36, +5.00] | 0.0020 | | pp2048 @ d32768 | **+4.33%** | [+3.56, +5.11] | 0.0020 | All six survive Holm; both node pairs agree in direction on every cell. **Decode is not resolved in either direction** — `tg128` was declared unresolvable before the run (its within-build spread equals its entire historical range) and is reported for the record only. **V1 was the default when this was written; that was reversed on 2026-08-09 — see Update 2026-08-09.** The reasoning below was correct on the evidence available at the time, and the collapse it describes was real; it turned out not to be a property of the runner. Kept unedited because how the conclusion failed is the useful part. V2 is ahead on prefill, KV headroom (+4.70 GiB) and draft acceptance (+6.6%), but its long-context recall under concurrency is unreliable in a way that is worse than a consistent deficit: across 14 independent serves on the same build and configuration, roughly two thirds land in a state that loses most of the needles (arthur c=12 as low as 3/24), while the rest match V1 at 22–24/24. The mode is fixed at startup and stable within a serve, and nothing we have found predicts or detects it. A deployment could run clean for days and restart into the bad mode. The cause is **not identified**. Eliminated so far: the eager-scratch cross-template aliasing, the upstream merges, and the eager scratch pool as a whole (pool on 2 good / 6 bad vs pool off 3 good / 3 bad over 14 serves — no effect). The startup logs of a good and a bad serve are structurally identical, which rules out \"a different code path was taken\". Anyone opting into V2 with `VLLM_USE_V2_MODEL_RUNNER=1` should know this. ### Measurement note Two errors from our own process, since they affect how the numbers above should be read. The n=8 sampling that originally established V2's recall deficit took eight gate runs from **one serve** — it measured within-serve variance while the quantity that actually varies is across-serve. Raising n on the wrong axis. The 14-serve figures above use the inverted design: many serves, few gates each. And the `±` in a benchy row is the spread *within one invocation*; it runs 5–30× smaller than the build-to-build spread. The blocked design above exists because of that: a coarse range screen over the same 10 blocks returns \"no measurable difference\" on all six prefill cells, while the paired test finds all six. Had the screen been the decisive statistic, this section would have concluded the opposite and been wrong. ## Update 2026-08-09 — the V2 recall collapse was a prefix-cache race, not the runner; V2 becomes the default 70 upstream commits (to `643c125fab`), and the long-standing reason this branch pinned V1 is gone: it was **an unfixed upstream bug**, not a property of the V2 model runner. ### The defect `FullAttentionManager.cache_blocks()` commits prefix block hashes to the shared `BlockPool` **at scheduling time, before the forward pass writes their KV**. A request admitted later in the same step can match those hashes and read unwritten values. `MambaManager` has guarded this since #29387; no other manager does. This is **vllm-project/vllm#42359**, open and unmerged. Two more reports look like the same triple on different models — **#50188** (prefix caching + MTP spec decode + fp8 KV, byte-identical repeat requests, RTX 5090 / Qwen3.6-27B-NVFP4) and **#43559** (closed without a merged fix). Anyone on `--enable-prefix-caching` with speculative decoding is exposed; DeepSeek-V4 is not special here. What makes it hard to catch: **the damage persists**. A serve that loses the race keeps serving from the poisoned blocks for its lifetime, so a later *serial* request fails too — which is why it looked like a per-serve \"mode\" rather than a race. It is also stochastic, roughly half of cold serves. ### Evidence Same binary, `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT` the only variable, cache populated by the real gate, **4 fresh serves per arm** (a single clean serve proves nothing at ~50% incidence), 3 arthur c=12 runs each: | | serve 1 | serve 2 | serve 3 | serve 4 | mean | min | | --- | --- | --- | --- | --- | --- | --- | | guard **off** | 22/23/24 | **6/5/3** | **14/9/10** | **7/8/7** | 11.5 | 3 | | guard **on** | 23/22/21 | 20/22/20 | 22/23/23 | 23/22/23 | 22.0 | 20 | Mann-Whitney U, p = 0.0043. Every serve's runner and guard state was read back from the serve log rather than assumed. **It also fixes V1**, which was not expected: V1's arthur c=12 goes 20.7 → 23.0 with the guard on (24.0 with prefix caching disabled entirely). V1's own 2–4 needle shortfall was the same defect, not an inherent concurrency margin. ### Runner arbitration, re-run on the fixed tree Same tree, same guard mode, **runner the only variable**: | | V1 | V2 | | --- | --- | --- | | arthur c=12, 4 serves × 3 | 22.3 / 21.7 / 21.7 / 21.0 → **21.67** | 22.0 / 20.7 / 22.7 / 22.7 → **22.00** | | pp2048 d8192 / 16384 / 32768 | 1427 / 1363 / 1216 | **1472 / 1421 / 1303** (+3.1% / +4.2% / +7.1%) | | tg128 mean | 39.95 / 41.16 / 35.25 | **41.56 / 49.61 / 45.18** | | e2e TTFT | 1437 / 1506 / 1689 ms | **1393 / 1444 / 1576 ms** | | GPU KV cache | 339,194 tok | **423,752 tok** (+24.9%) | | GSM8K strict / flexible | 0.9378 / 0.9401 | 0.9401 / 0.9439 | | issue19 · multi-needle · c=1 | PASS · 48/48, 0 leaks · 2/2 | PASS · 48/48, 0 leaks · 2/2 | Recall: p = 0.697, neither side with a single-digit serve. GSM8K differs by <0.4 pp against ~1.1 pp single-run noise. **V2 is not behind anywhere and leads on throughput, latency and KV headroom, so it becomes the default.** `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Correction:** the *\"V2 +6.6% draft acceptance\"* figure in Update 2026-08-04 does **not** survive re-measurement — 2.772 (V1) vs 2.710 (V2) on the same formula and sample size, i.e. a tie. It was measured while V2 was poisoned. ### If you are running this branch **Nothing to set — the guard is on by default where it matters.** `KVCacheCoordinator` enables it whenever prefix caching and speculative decoding are both active, which is the only configuration in which a block hash can be published before its KV is written *and* a second request admitted in the same step to match it. This correction matters: an earlier revision of this update shipped V2 as the default while leaving the guard off by default, which would have handed a plain serve the exact combination measured at mean 11.5 with a 3/24 floor. Both changes looked like improvements in isolation. If you pulled `c054feedac`, take `aa0d513027` instead, or set the variable yourself. To turn it **off** (it is a real escape hatch, pinned by a test): ```bash VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0 ``` `1` is upstream's semantics, gated on `use_eagle`; on DeepSeek-V4 that covers only **2 of 5** managers and leaves the main MLA path unguarded — measured, not assumed, via a startup log line this branch adds that reports how many managers are actually guarded. `2` covers every group and is what the engine selects. Regression sweep on the merged tree: 261 passed, 1 failed, that one failing identically on the pre-merge tree `4ebd1fb698`. ### The two endpoints disagreed about the same model, twice `ResponsesRequest.reasoning` took its type from the OpenAI SDK, whose `ReasoningEffort` stops at `xhigh`, so DeepSeek's documented top tier `max` was rejected by schema validation on `/v1/responses` while `/v1/chat/completions` accepted it. Worse, and on the default path: with no thinking kwarg, `DeepSeekV4Tokenizer.apply_chat_template` defaults thinking **on** while `DeepSeekV4ReasoningParser` defaults it **off** and selects `IdentityReasoningParser`. The model reasoned and its reasoning, with a bare ``, came back inside `output_text` as though it were the answer — whenever a request omitted `reasoning`, which is exactly what a stock OpenAI SDK sends. Chat was immune only because it normalises thinking state at the protocol boundary, and its own docstring says why: *\"so the tokenizer and reasoning parser see the same effective state\"*. Responses never called that hook. The derivation now lives in `deepseek_v4_chat_kwargs` and both request types call it, so a third endpoint cannot repeat it. Measured on both checkpoints with no workaround flag set, 21/21 each: | | `DeepSeek-V4-Flash-0731` | `DeepSeek-V4-Flash` | | --- | --- | --- | | silence: reasoning in its own field | PASS | PASS | | silence: no `` in the answer | PASS | PASS | | `effort: none` disables thinking | PASS | PASS | | six spellings × two endpoints | PASS | PASS | | `high` reasons deeper than `low` | +85% / +110% | +19% / +35% | 26 unit cases accompany it, 11 of which fail on the unpatched tree. `tests/reasoning` 440 passed, `tests/tokenizers_/test_deepseek_v4.py` 45 passed. ### Acceptance on the exact published SHA Everything above was re-measured on `d44e224ab9` — the commit this tag points at, after a second upstream sync (17 further commits, FlashInfer 0.6.16.post3) — not on an ancestor assumed to be equivalent. 17 of 18 checks pass: | check | result | | --- | --- | | four nodes clean at the SHA, FlashInfer 0.6.16.post3 | PASS | | `tests/v1/core` | 509 passed, 1 pre-existing failure | | default serve with **nothing set**: boots, runner V2, guard 5/5, no NameError | PASS | | arthur c=12 ×3 / c=1 | 22 / 20 / 23 · 2/2 | | GSM8K strict · issue19 · multi-needle | 0.9363 · PASS · 48/48, 0 leaks | | pp2048 d8192 | 1480.77 (arbitration V2 arm 1471.8) | | `VLLM_USE_V2_MODEL_RUNNER=0` → V1, guard still 5/5, c=1 2/2 | PASS | | `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0` → guard 0/5, c=1 2/2 | PASS | **The one non-pass, and what it turned out to be.** `tests/v1/spec_decode` does not complete on this hardware — it wedges under a 30-minute bound on this head and on `4ebd1fb698` alike. Narrowed to `test_max_len.py` and measured both ways: | how it is run | result | | --- | --- | | whole file, one pytest process | wedges after ~7 min, 5 of 11 done | | each case in its own process | **11 of 11 pass**, free memory steady at 117 GiB | No individual case is broken. Each stands up a full engine, and repeated create/tear-down inside one process does not release resources fast enough on a **single-GPU** unified-memory node. That also explains why both trees wedge and why they stop at different points. It remains **unverified coverage rather than a pass**; running one process per case produces a verdict instead of a hang. `test_async_scheduling_pp_allows_rescheduling_with_output_placeholders` is the same class: it builds `pipeline_parallel_size=2`, and a GB10 node has one GPU, so it fails at config construction. It is the only case in `tests/v1/core` that needs more than one GPU; the other 509 pass. ### What this arbitration does and does not cover Everything above was measured on **one configuration**: 2-node TP=2, `DeepSeek-V4-Flash-0731`, DSpark `num_speculative_tokens: 5`, fp8 KV, `max_model_len` 131072, prefix caching on, GB10 (SM121). The default now applies to **every** DSpark config, including shapes not measured here — TP=4, other context lengths, the NVFP4 checkpoint, single-node setups. The reasoning generalises better than the numbers do: the race is in block publication and is not specific to a model shape, and V2's advantage comes from KV headroom and scheduling rather than anything config-specific. But if you run a materially different shape and see something worse, `VLLM_USE_V2_MODEL_RUNNER=0` returns you to V1 and a report would be welcome — that is a gap in our coverage, not a claim we have ruled out. ## Running DSpark DSpark is DeepSeek's self-drafting speculative-decode variant; on `0731` the draft weights are carried in the main checkpoint, so no separate `--speculative-model` is needed. ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --max-model-len 49152 \\ --max-num-seqs 64 \\ --max-num-batched-tokens 8192 \\ --gpu-memory-utilization 0.85 \\ --enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\"}' ``` - `num_speculative_tokens` **must equal** the checkpoint's `dspark_block_size` (5). Larger values are rejected: they are never accepted and only waste draft compute. - `--kv-cache-dtype fp8` is mandatory — DSv4's `fp8_ds_mla` attention asserts an fp8 KV layout, so the default `auto` fails at model construction. Not DSpark-specific. - Runs on the **V1 runner by default** (correct long-context recall). `VLLM_USE_V2_MODEL_RUNNER=1` opts into the V2 DSpark speculator; V2's long-context recall is correct after the jasl/vllm#26 padded-Q fix. - If you measure draft acceptance yourself, **use prose**. On counting or repeated text the Markov head alone reaches 68–100% acceptance even with the neural draft path degraded, which hides real regressions entirely. ## Dependencies (stock-deps path) Pins on the current head: torch 2.13.0 (triton 3.7.1) · `flashinfer-python` / `flashinfer-cubin` 0.6.15.post1 · tilelang 0.1.12 · `nvidia-cutlass-dsl[cu13]` 4.6.0 · `quack-kernels>=0.6.1` · nvidia-nccl-cu13 2.30.7 (multi-node, see below). - **FlashInfer** is pinned in `requirements/cuda.txt` (`flashinfer-python` **and** the GitHub-release `flashinfer-cubin`, which must be the same version); it ships the SM120 packed sparse-MLA kernels, so a stock build picks them up with no manual install dance. - **GB10 / multi-node**: pin `nvidia-nccl-cu13==2.30.7` on every node. A rebuild silently reverts it to torch's bundled version, and a per-node mismatch hangs the NCCL handshake. - The SM120 **decode** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_DECODE`) and **prefill** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_PREFILL`) FlashInfer sparse-MLA paths **default on**; set either `=0` to fall back to the FlashMLA / Triton path. Both are availability-gated, so stock installs without the kernel degrade gracefully rather than raising. ## Running the NVFP4 checkpoint This branch also serves `nvidia/DeepSeek-V4-Flash-NVFP4` on SM12x (RTX PRO 6000 / GB10). The NVFP4 MoE auto-selects the FlashInfer CUTLASS backend (the SwiGLU-clamp model gate accepts it), so no `--moe-backend` flag and no special FlashInfer build are required: ```bash vllm serve nvidia/DeepSeek-V4-Flash-NVFP4 \\ --trust-remote-code --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --tokenizer-mode deepseek_v4 ``` Expert-parallel off (plain TP) is the supported path. Accuracy matches MXFP4 (GSM8K 8-shot ~0.96 on both SM120 and SM121). On SM12x NVFP4 is **not** a memory or throughput win versus MXFP4: NVFP4 weights are ~4 GiB/GPU larger, leaving less KV-cache room; single-stream prefill is marginally faster and aggregate decode marginally slower. Its value here is checkpoint availability / parity with the SM100 datacenter path — MXFP4 remains the better practical choice on consumer Blackwell. ## AI assistance disclosure AI assistants, including OpenAI Codex/GPT models and Anthropic Claude models, were used for code review, refactoring support, regression-script writing, and benchmark analysis. The branch was validated through human review plus the commands and harness artifacts listed above; every performance and accuracy number quoted was measured on real SM120/SM121 hardware.", "url": "https://github.com/vllm-project/vllm/pull/41834", "createdAt": "2026-05-06T15:17:15Z", "updatedAt": "2026-08-12T20:16:37Z", "timestamp": "2026-08-12T20:16:37Z", "metrics": { "reactions": 40, "comments": 440 }, "labels": [ "new-model", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "tool-calling", "deepseek", "kv-connector", "nvidia", "quantization", "mrv2" ], "author": "jasl", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:99768f3b6e0352cb6d9c", "signalId": "github:vllm-project/vllm:pull_request:50268", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50268", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][AMD] Enable fused bf16→fp32 router GEMM on ROCm", "text": "## Purpose This pull request addresses issue #50267 by enabling fused bf16→fp32 GEMM operations on AMD ROCm hardware for MoE router gates. ## Problem Statement The MoE router gate requires `out_dtype=torch.float32` for the `grouped_topk` operation. However, on ROCm, the fused fp32-output GEMM implementation was gated exclusively to CUDA platforms. This caused a fallback to bf16 GEMM followed by a separate cast operation, creating a standalone \"bf16→fp32 copy kernel\" visible in performance traces between the router GEMM and `grouped_topk` for every MoE layer on each decode step. ## Solution The changes enable hipBLASLt's `out_dtype` epilogue support (analogous to CUDA's cuBLAS capability) by modifying `vllm/model_executor/layers/fused_moe/router/gate_linear.py`: - Adds ROCm as an alternative condition for `allow_cublas_router_gemm` when: no bias exists, weights are bf16, and output is fp32 - Updates both `__init__` and `set_out_dtype` methods to ensure consistency - The fused tier now executes on ROCm, folding the fp32 cast into the GEMM epilogue ## Impact Assessment **CUDA:** No changes; the condition uses `current_platform.is_rocm()`, which evaluates false on CUDA. **Correctness:** Verified on gfx942 hardware; results match fp32 reference within ~6e-5 tolerance. **Scope:** Router gate only; expert GEMMs and other fp32 operations remain unaffected. ## Performance Results Benchmarked on ROCm (gfx942) using the online serving benchmark: baseline vs. fused, 3 seeds each (123 / 456 / 789), max concurrency 4, 40 requests per run (40960 input + 40960 generated tokens). All runs completed 40/40 successful with 0 failures. Averaged across the 3 seeds: | Metric | Baseline | Fused (this PR) | Δ | |---------------------------|----------|-----------------|----------------| | Benchmark duration (s) | 343.18 | 337.69 | −5.5 s (−1.6%) | | Output throughput (tok/s) | 119.35 | 121.29 | +1.9 (+1.6%) | | Total throughput (tok/s) | 238.71 | 242.59 | +3.9 (+1.6%) | | Mean TPOT (ms) | 32.87 | 32.34 | −0.53 (−1.6%) | | Mean ITL (ms) | 32.87 | 32.34 | −0.53 (−1.6%) | | Mean TTFT (ms) | 689.65 | 688.97 | ≈flat (−0.1%) | **Interpretation:** Folding the fp32 cast into the router GEMM epilogue removes a per-MoE-layer, per-decode-step conversion kernel from the critical path. The decode-path metrics (TPOT, ITL, output throughput) all improve by a consistent ~1.6%, while TTFT is unchanged as expected since prefill is not dominated by that per-step kernel. Variance across seeds is small (throughput within ±0.2 tok/s, TPOT within ±0.07 ms), so the improvement is stable and above noise, if modest. ## Testing Plan - Confirm the standalone `bfloat16tofloat32_copy_kernel` is absent from ROCm traces - Validate router logits match previous behavior within tolerance - Unit tests added for the modification", "url": "https://github.com/vllm-project/vllm/pull/50268", "createdAt": "2026-07-29T13:26:44Z", "updatedAt": "2026-08-12T20:07:32Z", "timestamp": "2026-08-12T20:07:32Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm", "ready" ], "author": "mpashkovskii", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:dc52037924ee5a67e70e", "signalId": "github:vllm-project/vllm:pull_request:52028", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52028", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Pin DeepEP by its full commit hash", "text": "## Purpose `tools/ep_kernels/install_python_libraries.sh` pins DeepEP by a 10-character abbreviation: ```bash DEEPEP_COMMIT_HASH=${DEEPEP_COMMIT_HASH:-\"d4f41e4e93\"} ``` An abbreviated object ID is not a ref, so it cannot be fetched directly. GitHub serves any *complete* commit via `allowAnySHA1InWant`, but an abbreviation is never a valid want: ```console $ git fetch --depth 1 origin d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb * branch d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb -> FETCH_HEAD $ git fetch --depth 1 origin d4f41e4e93 fatal: couldn't find remote ref d4f41e4e93 ``` Cloning the whole repository and checking out afterwards resolves the abbreviation locally, which is why this goes unnoticed in the common path. It breaks any consumer that fetches only the pinned commit — a reasonable thing to do for a large dependency — and the resulting error names a network-shaped problem rather than the pin. This is the same hash, written in full. `--deepep-ref` and the `DEEPEP_COMMIT_HASH` environment override are unaffected. ## Test Plan `bash -n tools/ep_kernels/install_python_libraries.sh`, plus the fetch comparison above run against `github.com/deepseek-ai/DeepEP`. ## Test Result The full hash fetches successfully; the abbreviation fails. The commit resolved is unchanged: `d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb` (\"Add fence.proxy.async.shared::cta between mbarrier wait and TMA load. (#642)\").", "url": "https://github.com/vllm-project/vllm/pull/52028", "createdAt": "2026-08-12T17:31:10Z", "updatedAt": "2026-08-12T20:01:33Z", "timestamp": "2026-08-12T20:01:33Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready" ], "author": "tlrmchlsmth", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:add42bd6ce7482551e4f", "signalId": "github:vllm-project/vllm:pull_request:49139", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49139", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Kernel] Fix persistent top-k histogram reuse after short rows", "text": " ## Purpose Fix a correctness bug in `persistent_topk` when one persistent CTA group processes a radix row (`seq_len > 32768`), followed by a short or medium row (`seq_len <= 32768`), and then another radix row. The kernel previously used the outer row iteration counter to rotate its triple-buffered radix histograms. Short rows advanced that counter without executing `radix_topk`, causing the next radix row to reuse a histogram that could still contain counts from an earlier row. This PR introduces a separate `radix_iter` counter that advances only when `radix_topk` executes. It also adds a CUDA regression test that constructs the long-short-long scheduling pattern in one persistent CTA group. Related to #41748, which addresses workspace initialization between kernel launches. This PR fixes histogram-ring misalignment within a single launch. ## Test Plan - Compare the long-short-long case against `torch.topk`. - Run a consecutive-radix case as a control. - Repeat both cases ten times on an RTX 3080. - Exercise the adaptive CTA-group layout used by the regression test. - Run the checked-in regression test in CI: ```bash python -m pytest \\ tests/kernels/test_top_k_per_row.py::test_persistent_topk_reused_group_after_short_row \\ -v ``` - Run static checks: ```bash python -m py_compile tests/kernels/test_top_k_per_row.py ruff check tests/kernels/test_top_k_per_row.py git diff --check origin/main...HEAD git clang-format --diff origin/main -- \\ csrc/libtorch_stable/persistent_topk.cuh ``` ## Test Result RTX 3080 results before the fix: ```text middle_len=32768: match=False, set_diff=8 middle_len=32769: match=True, set_diff=0 ``` Results after rebuilding the production `topk.cu` with the fix: ```text middle_len=32768: failures=0/10 middle_len=32769: failures=0/10 ``` Adaptive regression layout: ```text ctas_per_group=7, num_groups=9, rows=(0, 9, 18) set_diff=0 ``` Python compilation, Ruff, `git diff --check`, and clang-format checks passed. The complete `_C_stable_libtorch` extension and checked-in pytest were not run locally; they are left for CI. Local GPU validation compiled the production `topk.cu` and modified `persistent_topk.cuh` for SM86. No documentation update is required because this is an internal kernel correctness fix with no public API or configuration changes. AI assistance disclosure: OpenAI Codex was used for source analysis, implementation, regression-test construction, and validation. I reviewed and understand all submitted changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results. - [x] No documentation update is required for this internal kernel fix.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/49139", "createdAt": "2026-07-20T02:07:09Z", "updatedAt": "2026-08-12T19:54:46Z", "timestamp": "2026-08-12T19:54:46Z", "metrics": { "reactions": 2, "comments": 12 }, "labels": [ "bug", "ready" ], "author": "fxfxfxfxfxfxfxfx", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:669c7b49256a99383b9a", "signalId": "github:vllm-project/vllm:pull_request:51843", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51843", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable fine-grained prefix-cache hits for incompatible hybrid KV layouts", "text": "## Summary Fine-grained prefix-cache hits are enabled for hybrid models containing Mamba `align` groups. However other groups, such as a sliding-window DSpark drafter, may use KV cache managers that only support block-aligned lookups. This previously caused an assertion when prefix caching was enabled. This change: - Enables fine-grained hits only when every KV cache manager supports the required lookup granularity. - Falls back to scheduler-block-aligned prefix hits for incompatible layouts. - Prevents the scheduler from creating partial-tail cache entries after that fallback. - Keeps the drafter’s native sliding-window KV representation, avoiding model-specific overrides. - Logs when fine-grained hits are disabled. ## Kimi K3 results with DSpark SWA - KV capacity is the same with and without: 7,815,323 tokens / 7.45× concurrency. - Repeated 66,007-token prompt reused 57,600 tokens. - The same hit remained after 300 GSM8K requests. - Mean acceptance length: 4.87, 4.83, 4.83. - GSM8K accuracy: 0.30, 0.32, 0.32. - Zero invalid responses. Short and medium prompts may receive no prefix-cache reuse because hits must land on the much larger scheduler alignment. ## Validation - 23 partial-prefix-cache tests passed. - 89 broader prefix-cache tests passed. - Pre-commit passed. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51843", "createdAt": "2026-08-11T16:11:07Z", "updatedAt": "2026-08-12T19:53:11Z", "timestamp": "2026-08-12T19:53:11Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "ready", "kimi" ], "author": "mgoin", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:96300b02a32dbbc789cd", "signalId": "github:vllm-project/vllm:issue:52040", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: MM input tensors are re-broadcast to all TP workers on every request, even when the prefix cache fully covers the image tokens (~19 ms per unique image)", "text": "## Environment - vLLM v0.25.1 (official `vllm/vllm-openai` Docker image), single API server process - TP=4 on one node (4 GPUs), `mm_processor_cache_type: lru` (behavior identical under `shm`) - Gemma-4-31B-IT with vision; `--mm-processor-kwargs {\"max_soft_tokens\": 560}` → `pixel_values` per image is `(1, 5040, 768)` fp32 ≈ **15.5 MB** - `--limit-mm-per-prompt {\"image\": 32}` ## Workload Assistant with a rolling visual-context window: each turn re-sends the same ~26 images (byte-identical, position-stable) plus one fresh frame. Prefix caching covers all history image tokens (this works great), and the mm processor cache hits 26/26 (also works great — with `lru`). Requests are therefore ~100% cached, and the encoder never runs for the history images. ## Problem Even in that fully-cached state, TTFT grows **linearly with the number of unique images attached**: ~23 ms per unique image, of which ~19 ms is the EngineCore re-serializing each unique tensor and re-copying it into the shm broadcast ring to the TP workers — on **every request**, although the workers will never consume those tensors (their soft-token spans are entirely within the prefix-cache-covered region, so no encoder run is ever scheduled). ## Measurements (all in-pod via localhost, byte-identical resends, all shapes warm) **1. Fully-cached TTFT floor vs unique image count** (queue = 0.0 ms, prefill ≈ 14 ms in all cases; `mm_cache_hits == queries` on every request): | unique images | TTFT (ms) | |---|---| | 2 | 69 | | 8 | 188 | | 13 | 310 | | 20 | 472 | | 26 | **634** | → slope ≈ **23 ms per unique image**, all CPU, none of it GPU. **2. Duplicate-image probe** — same request shape/token count, 26 image slots, but all 26 slots reference **one** image (byte-identical data URLs) vs 26 **distinct** images, everything cached in both cases: - 26 distinct cached images: **639 ms** - 26 copies of one cached image: **156 ms** → the cost is per-*unique* image, not per attached item. (Pickle's object memoization explains the collapse: the receiver cache returns the same tensor object per hash, so duplicates serialize once.) **3. py-spy on the EngineCore during a pure cache-hit request loop** (26 distinct cached images per request): of 29.4 s sampled, **6.4 s in `shm_broadcast.enqueue`** and **4.4 s in `torch.serialization._legacy_save`** (rest is idle `sched_yield`). The API server process is nearly idle under the same load (parse layer measured separately at ~0.9 ms/image). **4. Serializer baseline**: `MsgpackEncoder`/`MsgpackDecoder` (the path used API-server→EngineCore) round-trips the same 15.5 MB tensor in **0.01 ms** (zero-copy); `tensor.clone()` is 0.06 ms. The broadcast path's per-tensor cost is ~19 ms. ## Analysis (checked against current `main`) - API-server→EngineCore transport is already hash-only on hits — `MultiModalFeatureSpec.data`'s docstring says it explicitly: *\"Can be `None` if the item is cached, to skip IPC between API server and engine core processes\"* (`vllm/multimodal/inputs.py`). This half of the design works. - The EngineCore then re-populates the tensors from its receiver cache (`v1/engine/core.py:976`, `get_and_update_features`) and ships them to the TP workers inside `NewRequestData.mm_features` at admission (`v1/core/sched/output.py`) — per unique tensor, per request. Notably `NewRequestData` already carries `num_computed_tokens` in the same dataclass, so the information needed to skip is co-located with the thing to be skipped. - On v0.25.1 the broadcast serializes via pickle → `torch._legacy_save` (full byte rewrite of each 15.5 MB). Current `main` routes CPU tensors through `_reduce_tensor` / out-of-band buffers (`shm_broadcast.py:823-846`), which should remove most of the serialization half — but the ring copy remains, and more fundamentally **the tensors are shipped at all** for requests whose image placeholders are fully covered by the prefix cache. ## Proposal Attach an mm item's tensors to the **step where its encoder run is scheduled**, rather than to request admission: - At admission the scheduler already knows `num_computed_tokens` and each mm item's placeholder span; an item whose span lies entirely within the computed prefix needs no encoder run and therefore no tensor on the workers. The scheduler already plans encoder execution explicitly (`scheduled_encoder_inputs`, `v1/core/sched/scheduler.py:679`), so tensors could travel with that instead of with `NewRequestData`. - The receiver cache already retains tensors for the request lifetime, so preemption/recompute can still ship them lazily on demand. - For our workload this turns ~25 shipped tensors per request into 0–1, i.e. the fully-cached floor becomes flat in image count (~4 ms/image of unavoidable parse+hash remains). Happy to contribute a PR for this (and/or backport-friendly serializer improvements) with guidance from maintainers on the preferred shape. ## Reproduction sketch ```python # against any TP>1 vision deployment with prefix caching + processor cache enabled: # 1. build a chat request with N distinct images + fixed text; send twice (second send = fully cached) # 2. build the same request but with all N image slots referencing ONE image; send twice # 3. compare second-send TTFTs and watch vllm:mm_cache_* and queue/prefill counters: # distinct-N grows ~linearly in N; dup-N stays near the 1-image floor; queue/prefill flat in both. ```", "url": "https://github.com/vllm-project/vllm/issues/52040", "createdAt": "2026-08-12T19:44:46Z", "updatedAt": "2026-08-12T19:44:46Z", "timestamp": "2026-08-12T19:44:46Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "sseanliu", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c757557bd1e76dd91519", "signalId": "github:vllm-project/vllm:pull_request:46067", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46067", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][TurboQuant] Fix dangling decode scratch when workspace grows after cudagraph capture", "text": "## Summary Fixes a CUDA \"illegal memory access\" that kills the engine on the **first decode step** when serving with `--kv-cache-dtype turboquant_*` (see issue: #45670) and CUDA graphs enabled (the default `FULL_AND_PIECEWISE`). ## Root cause TurboQuant decode (`_decode_attention`) sourced its scratch buffers (`mid_o`/`output`/`lse`) from the **growable** `WorkspaceManager`. Decode runs inside the **FULL** cudagraph, so those buffer addresses are baked into the captured graphs at capture time. The `WorkspaceManager` frees and reallocates its buffer when it grows — and calls `empty_cache()`, unmapping the old address. CUDA graphs are captured for ascending batch sizes (B=1..max); each larger capture grows the workspace, freeing the address an earlier-captured (smaller-batch) graph still points at. Replaying that graph — e.g. the very first decode (B=1) — dereferences freed memory. The error surfaces asynchronously at the next synchronizing CUDA call (the H2D copy in `prepare_inputs`), which is why the reported traceback looked unrelated to attention. Symptoms that localized it: - `--enforce-eager` works (no graphs). - `--compilation-config '{\"cudagraph_mode\": \"PIECEWISE\"}'` works (decode attention runs eagerly, never captured). - Reproduces on every `turboquant_*` preset (scratch size is preset-independent); no non-turboquant dtype is affected. ## Fix Give TurboQuant decode a **dedicated scratch buffer**, allocated **once at the max cudagraph capture batch** and reused (sliced `[:B]`) by every TQ layer and every captured decode graph. Because it is never resized and is separate from the `WorkspaceManager`, its address never moves — so the addresses baked into the FULL graphs stay valid, including against a later long continuation-prefill that grows the workspace. The eager path (no cudagraphs, or a batch beyond the captured sizes) still uses the workspace, where there is no graph to dangle. Files: - `vllm/v1/attention/backends/turboquant_attn.py` — module-level `_DECODE_SCRATCH` + `_get_decode_scratch` + `reset_tq_decode_scratch`; `_decode_attention` routes captured batches to the fixed buffer, eager batches to the workspace. - `vllm/v1/worker/gpu/shutdown.py` — release the scratch on model-runner teardown. ## Not a duplicate Fix to the in-tree TurboQuant backend; no existing open PR addresses the FULL-cudagraph workspace-pointer staleness in TurboQuant decode. ## Test plan > Run on a GPU box (validated config: Qwen3-8B, GB200). Original failure (crashes before this PR, serves after): ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 ``` Eager / piecewise unaffected: ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 --enforce-eager vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_k8v4 \\ --compilation-config '{\"cudagraph_mode\": \"PIECEWISE\"}' ``` Other presets (same root cause): ``` vllm serve Qwen/Qwen3-8B --kv-cache-dtype turboquant_4bit_nc ``` Unit: ``` pytest tests/quantization/test_turboquant.py ``` **Results:** _to be filled in by the submitter after running on GPU_ — this branch was authored on a machine without a GPU, so the commands above have not yet been executed here. ## AI assistance Authored with the assistance of an AI coding assistant (Claude Code). All changed lines have been reviewed by the submitter, who is responsible for defending the change and running the tests above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/46067", "createdAt": "2026-06-18T15:59:03Z", "updatedAt": "2026-08-12T19:44:14Z", "timestamp": "2026-08-12T19:44:14Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "v1", "nvidia", "quantization", "mrv2" ], "author": "ldkhang1201", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9d5813bf126e42e25863", "signalId": "github:vllm-project/vllm:issue:51776", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51776", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: CuTe FA4 forward (flash_fwd.py SM80/SM120) fails on first call: NameError 'mDynamicCausal' and missing self.is_split_kv", "text": "### Summary Two independent Python-level bugs make `vllm/vllm_flash_attn/cute/flash_fwd.py` (`FlashAttentionForwardSm80`, and by inheritance `FlashAttentionForwardSm120`) fail on the very first call, before any kernel runs. They look like the code path has never been executed: SM80-class GPUs use the C++ FA2/FA3 kernels, so this CuTe DSL path is effectively dead there — but on **sm_120** it is selected, and anything routing through `vllm.vllm_flash_attn.cute.flash_attn_varlen_func` hits it. Both reproduce on unmodified `vllm/vllm-openai:v0.26.0` with a ~25-line script: single GPU, no model, no distributed init, batch size 1. One-line fixes for both are suggested below. Found while investigating a workaround for #51405 (paged KV not implemented for SM 12.0 in this same path). These two bugs are independent of that and of any particular model. ### Environment - 2× NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), **sm_120**, PCIe-only - Driver 595.84, CUDA 13.2, Ubuntu 25.10 - vLLM `0.26.0` (`vllm/vllm-openai:v0.26.0`), also reproduced on `0.26.1.dev1` built from source - `nvidia_cutlass_dsl` as shipped in that image ### Reproducer ```python import torch from vllm.vllm_flash_attn.cute import flash_attn_varlen_func SEQLEN_K, BATCH = 1024, 1 NUM_HEADS, NUM_KV_HEADS, HEAD_DIM = 16, 4, 128 dev, dtype = torch.device(\"cuda:0\"), torch.bfloat16 total_q = SEQLEN_K * BATCH q = torch.randn(total_q, NUM_HEADS, HEAD_DIM, device=dev, dtype=dtype) k = torch.randn(BATCH, SEQLEN_K, NUM_KV_HEADS, HEAD_DIM, device=dev, dtype=dtype) v = torch.randn_like(k) out = flash_attn_varlen_func( q=q, k=k, v=v, cu_seqlens_q=torch.arange(0, total_q + 1, SEQLEN_K, device=dev, dtype=torch.int32), seqused_k=torch.full((BATCH,), SEQLEN_K, device=dev, dtype=torch.int32), max_seqlen_q=SEQLEN_K, page_table=None, softmax_scale=HEAD_DIM ** -0.5, causal=True, num_splits=1, return_lse=False, ) torch.cuda.synchronize() print(\"OK\", (out[0] if isinstance(out, tuple) else out).shape) ``` ```bash docker run --rm --gpus all -e CUDA_VISIBLE_DEVICES=0 -v \"$PWD:/work\" \\ --entrypoint python3 vllm/vllm-openai:v0.26.0 /work/repro.py ``` Today this prints the Bug 1 error. With the Bug 1 patch applied it prints the Bug 2 error. With both applied it prints `OK`. --- ## Bug 1 — `NameError: name 'mDynamicCausal' is not defined` ``` error: NameError in `__call__`: name 'mDynamicCausal' is not defined --> /usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/cute/flash_fwd.py:840 = note: Caused exception: name 'mDynamicCausal' is not defined ``` `mDynamicCausal` is a parameter of the `__call__` **method** (`flash_fwd.py:681`): ```python mDynamicCausal: Optional[cute.Tensor] = None, ``` but it is referenced from the body of `@cute.kernel def kernel(...)` at `flash_fwd.py:840`: ```python psc = mDynamicCausal[batch_size] if const_expr(mDynamicCausal is not None) else None ``` `kernel` is a separate method of the class, not a closure nested inside `__call__`, so the name is not in scope there. The launch site (`self.kernel(...)`, `flash_fwd.py:761`) passes its arguments positionally and ends at `output_scale`, so it does not supply it either. Suggested fix — append to both the kernel signature and the positional launch list, leaving the order of existing arguments untouched: ```diff aux_data, fastdiv_mods, output_scale, + mDynamicCausal, ).launch( ``` ```diff aux_data: AuxData = AuxData(), fastdiv_mods=None, output_scale: Optional[cute.Tensor] = None, + mDynamicCausal: Optional[cute.Tensor] = None, ): # Thread index, block index ``` ## Bug 2 — `AttributeError: 'FlashAttentionForwardSm120' object has no attribute 'is_split_kv'` With Bug 1 fixed, the same script fails with: ``` AttributeError: 'FlashAttentionForwardSm120' object has no attribute 'is_split_kv' ``` `flash_fwd.py` reads `self.is_split_kv` in four places (≈ lines 365, 381, 408, 413), but `FlashAttentionForwardSm80.__init__` (`flash_fwd.py:620`) never assigns it. Both sibling implementations do: - `flash_fwd_sm90.py:70` — `self.is_split_kv = is_split_kv` - `flash_fwd_sm100.py:188` — `self.is_split_kv = is_split_kv` (The `is_split_kv` at `flash_fwd.py:194` is a parameter of `_check_type`, not of `__init__`.) The intended value for this path looks unambiguous: the kernel launch hardcodes `False, # is_split_kv`, and `interface.py` already asserts `not is_split_kv` for SM 12.0. Suggested fix: ```diff def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) assert self.output_quant_key is None, ( f\"Fused quant output not implemented for {type(self).__name__}\" ) + self.is_split_kv = False ``` --- ### After both fixes The kernel compiles and runs, and returns finite, plausible output across the configurations I tried (batch 1–8, seqlen_k 128–8192, causal, sliding window, decode-shaped q, with and without `score_mod`/`aux_tensors`). One regime still faults with an out-of-bounds device read — `aux_tensors` combined with batch > 1 — which I have written up separately in a comment on #51405, since reaching it at all requires working around that issue's paged-KV gap. I have sm_120 hardware available and am happy to test patches.", "url": "https://github.com/vllm-project/vllm/issues/51776", "createdAt": "2026-08-11T07:15:36Z", "updatedAt": "2026-08-12T19:42:54Z", "timestamp": "2026-08-12T19:42:54Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "IgnacyPawlak", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:afd54efbccd12163000a", "signalId": "github:vllm-project/vllm:pull_request:51949", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51949", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Enable LoRA support for tower and connector in Cosmos3-Edge", "text": "## Purpose An operator serving `nvidia/Cosmos3-Edge` can attach a LoRA adapter to the language model, but not to the vision **tower** or the **connector**, so visual feature extraction and its projection into the language model's space are frozen. Adapting the model to a new visual domain — a different imaging modality, an unusual camera, a specialised document type — currently means full fine-tuning. Today the engine refuses to start at all: `Cosmos3EdgeForConditionalGeneration does not support LoRA yet`. The machinery for this already exists; each model opts in by declaring `SupportsLoRA` and implementing two token-count helpers that convert an LLM sequence token count into the row counts entering the tower and the connector. Cosmos3-Edge implemented neither, although its `get_mm_mapping` and `packed_modules_mapping` were already correct. Getting those counts wrong fails **silently** rather than loudly, which is what shapes the verification below: the LoRA metadata buffer is not cleared between forward passes, so any row an undercount fails to cover reads a stale adapter index from the previous pass, and the operator receives plausible output produced by the wrong adapter. Nothing raises. For that reason the row counts here are **measured against real tensors**, not derived on paper. Part of #31479, claimed there on 2026-08-07. That issue tracks this feature across many models and stays open until they are all done, so this PR references it without a closing keyword — matching every merged sibling in the series. This PR: - Declares `SupportsLoRA` on the model class. - Implements `get_num_mm_encoder_tokens` and `get_num_mm_connector_tokens`. - Fills in the LoRA cell for this model in `docs/models/supported_models.md`. ## Technical Details The two helpers are **chained**: `get_num_mm_connector_tokens` receives the *encoder row* count returned by `get_num_mm_encoder_tokens`, not the LLM sequence token count. ``` LLM sequence tokens --x merge²--> encoder rows --// merge²--> connector rows ``` `merge` is `spatial_merge_size`. The connector body is the exact inverse of the encoder body, so the round trip returns the original count. **The merge factor is read from `self.visual.spatial_merge_size`**, which differs from Qwen2.5-VL and Qwen3-VL, which read `config.vision_config.spatial_merge_size`. The reason is specific to this checkpoint: `Cosmos3EdgeConfig.__init__` assigns ```python self.vision_config.spatial_merge_size = self.projector_config.spatial_merge_size ``` (`vllm/transformers_utils/configs/cosmos3_edge.py:126`). `Siglip2VisionConfig` has no such field, so the vision config's attribute is a synthesized alias; `projector_config` is where the checkpoint stores it and what the projector is built from (`cosmos3_edge.py:227`). `cosmos3_edge.py:597` already computes this same quantity the same way when splitting vision embeddings per item. Both helpers read the factor once and the connector body is the exact inverse of the encoder body, so the round trip cancels regardless of the factor's value. Integer division only, since the runtime mapping multiplies a list by the result. No divisibility assertion: `vllm/lora/model_manager.py:250` calls the encoder helper once at engine init with a whole-batch budget that carries no divisibility guarantee. A non-positive input returns zero — the connector floor-divides, so a negative count would otherwise yield a silently empty mapping rather than an error. ## Duplicate check Per `AGENTS.md`, re-run immediately before opening: ```bash gh issue view 31479 --repo vllm-project/vllm --comments gh pr list --repo vllm-project/vllm --state open --search \"31479 in:body\" gh pr list --repo vllm-project/vllm --state all --search \"Cosmos3-Edge LoRA\" gh pr list --repo vllm-project/vllm --state open --search \"tower connector lora\" gh pr list --repo vllm-project/vllm --state open --search \"Cosmos3EdgeForConditionalGeneration\" ``` No open or merged PR adds tower/connector LoRA to Cosmos3-Edge; the model-name and class-name searches return nothing. The eleven open PRs in this family target other models. Three neighbours a reviewer may find, none of which this duplicates: - **#51221** — adds EVS video pruning to this same model. Different feature, overlapping files; a rebase may be needed depending on merge order. See *Known limitations*. - **#42662** — migrates the helper interface to `get_mm_lora_token_counts`. Its default implementation delegates to the two methods added here, so this model migrates unchanged. - **#47400** — fixes a *different* defect in the same mapping builder: cache-served items are counted although the encoder forward skips them. Independent of this change. ## Test Plan ```bash # CI-visible: mocked models, no weights, no GPU .venv/bin/python -m pytest tests/v1/worker/test_gpu_model_runner.py -k \"cosmos3_edge or lora\" # Processing regression. tests/models/registry.py marks this model # is_available_online=False, so the file is skipped unless given a local path: SNAPSHOT=$(.venv/bin/python -c \\ \"from huggingface_hub import snapshot_download; print(snapshot_download('nvidia/Cosmos3-Edge'))\") COSMOS3_EDGE_MODEL_PATH=\"$SNAPSHOT\" .venv/bin/python -m pytest \\ tests/models/multimodal/processing/test_cosmos3_edge.py .venv/bin/python -m pytest tests/models/test_registry.py pre-commit run --all-files ``` End-to-end runs: single RTX 3090 (24 GB), CUDA 13.0, at this PR's base commit. ## Test Result **Before / after.** Starting the engine with `--enable-lora --enable-tower-connector-lora`: ``` # before — engine refuses to start ValueError: Cosmos3EdgeForConditionalGeneration does not support LoRA yet. $ echo $? 1 # after — engine starts with the feature active WARNING [model_manager.py:242] LoRA for the tower and connector of multimodal models is experimental and may contain bugs. Please report any related issues on GitHub if you encounter them. $ echo $? 0 ``` **Helper output vs. real tensor rows**, with `--enable-tower-connector-lora`. Forward hooks on `visual.encoder.encoder.layers.0.self_attn.qkv_proj` (tower) and `visual.projector.linear_fc1` (connector): | input | LLM tokens | encoder rows observed | helper | connector rows observed | helper | |---|---|---|---|---|---| | image | 1107 | 4428 | 4428 | 1107 | 1107 | | video, 4 frames | 4180 | 16720 | 16720 | 4180 | 4180 | The same counts hold with an adapter attached — applying LoRA does not perturb them. Extended on CPU across a range of image sizes and video lengths: encoder rows, connector rows and the round trip are exact in every case.
Script that produces the table above ```python import os os.environ.setdefault(\"VLLM_ENABLE_V1_MULTIPROCESSING\", \"0\") from types import SimpleNamespace from vllm import LLM, SamplingParams from vllm.assets.image import ImageAsset from vllm.assets.video import VideoAsset from vllm.model_executor.models.cosmos3_edge import ( Cosmos3EdgeForConditionalGeneration as C3E, ) from vllm.transformers_utils.config import get_config MODEL = \"nvidia/Cosmos3-Edge\" TOWER = \"visual.encoder.encoder.layers.0.self_attn.qkv_proj\" CONNECTOR = \"visual.projector.linear_fc1\" seen: dict[str, list[int]] = {\"tower\": [], \"connector\": []} def install_hooks(model): def record(key): def hook(_module, args): seen[key].append(args[0].shape[-2]) return hook for name, module in model.named_modules(): if name == TOWER: module.register_forward_pre_hook(record(\"tower\")) elif name == CONNECTOR: module.register_forward_pre_hook(record(\"connector\")) llm = LLM( model=MODEL, max_model_len=16384, limit_mm_per_prompt={\"image\": 1, \"video\": 1}, gpu_memory_utilization=0.85, enforce_eager=True, allowed_local_media_path=\"/\", mm_processor_cache_gb=0, ) llm.apply_model(install_hooks) config = get_config(MODEL, trust_remote_code=False) tokenizer = llm.get_tokenizer() start = tokenizer.decode([config.vision_start_token_id]) end = tokenizer.decode([config.vision_end_token_id]) image_pad = tokenizer.decode([config.image_token_id]) params = SamplingParams(temperature=0.0, max_tokens=1) def check(label, output, token_id): llm_tokens = output[0].prompt_token_ids.count(token_id) encoder_rows, connector_rows = seen[\"tower\"][-1], seen[\"connector\"][-1] stub = SimpleNamespace( visual=SimpleNamespace( spatial_merge_size=config.projector_config.spatial_merge_size ) ) predicted_encoder = C3E.get_num_mm_encoder_tokens(stub, llm_tokens) predicted_connector = C3E.get_num_mm_connector_tokens(stub, predicted_encoder) print( f\"{label:<8} llm_tokens={llm_tokens} \" f\"encoder observed={encoder_rows} helper={predicted_encoder} \" f\"{'OK' if encoder_rows == predicted_encoder else 'MISMATCH'} \" f\"connector observed={connector_rows} helper={predicted_connector} \" f\"{'OK' if connector_rows == predicted_connector else 'MISMATCH'}\" ) check( \"image\", llm.generate( { \"prompt\": f\"<|im_start|>user\\n{start}{image_pad}{end}Describe.\" f\"<|im_end|>\\n<|im_start|>assistant\\n\", \"multi_modal_data\": {\"image\": ImageAsset(\"stop_sign\").pil_image}, }, params, ), config.image_token_id, ) video = VideoAsset(name=\"baby_reading\", num_frames=4) check( \"video\", llm.chat( [ { \"role\": \"user\", \"content\": [ { \"type\": \"video_url\", \"video_url\": {\"url\": f\"file://{video.video_path}\"}, }, {\"type\": \"text\", \"text\": \"Describe.\"}, ], } ], params, ), config.video_token_id, ) ``` The video prompt goes through the chat template. A hand-built video placeholder omits the per-frame timestamp tokens, which silently misplaces the embeddings and would invalidate the comparison.
**The adapter is applied, not silently skipped.** Serving with `--enable-tower-connector-lora` and a rank-8 adapter targeting one tower layer and the connector: the engine emits the tower/connector-LoRA experimental warning at startup, the Punica shrink and expand kernels are invoked during the adapter request, and the output tensors of the wrapped modules differ from the no-adapter run for images and for video alike. This shows the adapter path is *reached*; it is not a model-quality measurement. **Unit and regression tests** — all pass: ``` tests/v1/worker/test_gpu_model_runner.py -k \"cosmos3_edge or lora\" tests/models/multimodal/processing/test_cosmos3_edge.py tests/models/test_registry.py pre-commit run --all-files ``` ## Known limitations - **Video pruning (EVS) is not addressed.** This model does not implement `SupportsMultiModalPruning` at HEAD, so the interaction is unreachable here. On models that do support pruning, `PlaceholderRange.get_num_embeds()` returns a post-prune count while the tower has already processed the unpruned set, so these helpers undercount by roughly `1/(1-q)`, bounded by the frame count. That follows from the shared call site at `vllm/v1/worker/gpu/mm/lora.py:45` rather than from any one model, and already applies to `Qwen2_5_VLForConditionalGeneration`, `Qwen3VLForConditionalGeneration` and `Qwen3VLMoeForConditionalGeneration`. Images are unaffected. This PR neither introduces nor widens it. - **`visual.encoder.embeddings.patch_embedding` is a plain `nn.Linear`**, so LoRA skips it. It lives in `lfm2_siglip2.py`, shared with two other models. Pre-existing, out of scope, unchanged here. ## Acceptance criteria - [x] New tests added that exercise the changed code path - [x] Existing test suite passes locally (model-runner, multi-modal processing, model registry) - [x] Follows the project style guide — `pre-commit run --all-files` clean, including mypy - [x] No breaking changes: the change is additive, and no existing behaviour is modified - [x] Documentation updated (`docs/models/supported_models.md`) - [x] Duplicate-work checks re-run immediately before opening, per `AGENTS.md` - [x] AI assistance disclosed ## AI assistance AI assistance was used. I reviewed every changed line, and the results above are from real runs on the stated hardware.
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51949", "createdAt": "2026-08-12T08:01:44Z", "updatedAt": "2026-08-12T19:40:41Z", "timestamp": "2026-08-12T19:40:41Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "charitarthchugh", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5f938ac4df54bc32cbbb", "signalId": "github:vllm-project/vllm:pull_request:52030", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52030", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix packed GDN decode launch for large batch-head grids", "text": "## Purpose Avoid a CUDA launch failure in packed GDN decode when `batch_size * num_value_heads` exceeds the maximum CUDA grid Y/Z dimension of 65,535. The existing launch is preserved for normal sizes. Only overflowing cases use a split `(value_tiles, value_heads, batch)` grid. ## Test Plan ## Test Result - Verified the failing Qwen shape (`B=1024`, `HV=64`, `K=V=128`) launches successfully. - Running `vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned94 -tp=2` doesn't crash anymore ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52030", "createdAt": "2026-08-12T17:44:32Z", "updatedAt": "2026-08-12T19:39:00Z", "timestamp": "2026-08-12T19:39:00Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bf0adf1efede33171bac", "signalId": "github:vllm-project/vllm:pull_request:39448", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:39448", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "AMD remove sync visible devices", "text": "## Purpose Use HIP_VISIBLE_DEVICES and device control environment variable on ROCm platform. Aligning with Pytorch and Ray frameworks. Syncing with CUDA_VISIBLE_DEVICES is no longer necessary ## Test Plan vllm serve with ray backend `vllm serve Qwen/Qwen2-7B-Instruct --tensor-parallel-size 4 --pipeline-parallel-size 2 --distributed-executor-backend ray --gpu-memory-utilization .9 --max-model-len 8192 --max-num-seqs 2000 --max-num-batched-tokens 131072` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/39448", "createdAt": "2026-04-09T19:53:00Z", "updatedAt": "2026-08-12T19:35:41Z", "timestamp": "2026-08-12T19:35:41Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "rocm", "verified" ], "author": "vickytsang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e24686b6803c2831d1c1", "signalId": "github:vllm-project/vllm:pull_request:50539", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50539", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Gemma4] Keep image bidirectional attention within the sliding window on the V2 model runner", "text": "## Purpose #47217 fixed Gemma4 vision models silently falling back to **causal-only** image attention whenever a single image's soft-token span exceeded the text sliding window (e.g. ~1120 soft tokens at `max_soft_tokens=1120` vs `sliding_window=1024`). The runner was dropping the *entire* bidirectional `mm_prefix` range for such an image; the fix gated that skip on a model capability flag, `mm_prefix_clamp_sliding_window`, so models that bound `mm_prefix` **in-kernel** keep the full range and let the kernel clamp it per-query — matching HF's `(causal OR blockwise) AND sliding_window` on local layers. That fix only landed in the **V1** `GPUModelRunner`. The V2 model runner has its own copy of the range-building logic in `vllm/v1/worker/gpu/attn_utils.py::compute_mm_prefix_ranges`, which still drops oversized ranges unconditionally: ```python for r in mm_feature.mm_position.extract_embeds_range(): if sliding_window is not None and (r[1] - r[0] + 1) > sliding_window: continue # <- no opt-out, unlike V1 ``` So the original defect is still reachable today on `VLLM_USE_V2_MODEL_RUNNER=1`. This matters more than it did at the time of #47217, because V2 is no longer opt-in for these configs — `VllmConfig.use_v2_model_runner` force-enables V2 for DSpark speculative decoding and for hybrid DFlash drafters, both of which are used with Gemma4. A user who serves Gemma4 with images and DSpark drafting gets the degraded mask with no warning and no error. The kernel-side clamp (`flash_attn.py`, `triton_attn.py`, `triton_unified_attention.py`) is already shared by both runners and needs no change — V2 simply never delivers the range to it. ### Fix - `compute_mm_prefix_ranges` gains a `clamp_in_kernel: bool = False` parameter that gates the oversized-range skip, mirroring V1's `_clamps_in_kernel`. The default preserves current behavior for every other PrefixLM model. - `DefaultModelState.build_attn_metadata` reads `mm_prefix_clamp_sliding_window` off the model and forwards it, exactly as `GPUModelRunner` does. No behavior change for V1, for non-Gemma4 models, or for images that already fit within the sliding window. ### Not a duplicate Checked before opening, per `AGENTS.md`: ```bash gh search issues \"compute_mm_prefix_ranges\" --repo vllm-project/vllm --include-prs # 0 results gh search issues \"mm_prefix_clamp_sliding_window\" --repo vllm-project/vllm --include-prs # only #47217 gh search issues \"mm_req_doc_ranges\" --repo vllm-project/vllm --include-prs ``` No open issue or PR covers the V2 side. The nearest open PRs touch adjacent code but not this logic: #47646 (revert of an FA4 mm_prefix *kernel* mask change, `flash_attn.py` only), #46558 (FlashInfer mm-prefix, `flashinfer.py` only), and #48907 / #46963 (V2 `attn_utils.py`, but KV-cache reshape). #47217 is the V1 fix being ported here — its changed-file list contains no V2 file. ## Test Plan Added four unit tests to `tests/v1/worker/test_attn_utils.py` covering `compute_mm_prefix_ranges` (previously untested): - `test_compute_mm_prefix_ranges_skips_oversized_range_by_default` — default path still drops an oversized span (regression guard for other models). - `test_compute_mm_prefix_ranges_keeps_oversized_range_when_clamped_in_kernel` — the fix: an oversized span survives when the model clamps in-kernel. - `test_compute_mm_prefix_ranges_keeps_fitting_range_either_way` — spans within the window are unaffected by the flag. - `test_compute_mm_prefix_ranges_ignores_non_visual_modalities` — audio is still skipped. ```bash pytest tests/v1/worker/test_attn_utils.py -q ``` ## Test Result All tests pass with the fix: ``` ........ [100%] 8 passed, 14 warnings in 4.15s ``` Reverting only the two source files and re-running (tests kept) fails, confirming the tests exercise the fixed behavior rather than passing vacuously: ``` FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_keeps_oversized_range_when_clamped_in_kernel FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_keeps_fitting_range_either_way FAILED tests/v1/worker/test_attn_utils.py::test_compute_mm_prefix_ranges_ignores_non_visual_modalities 3 failed, 5 passed ``` `ruff format --check` and `ruff check` are clean on all three changed files (the one remaining `UP038` in `attn_utils.py` is pre-existing on `main`, in `build_attn_metadata`, and untouched here). ### Model evaluation This change affects generated output for one narrow case: Gemma4 vision requests under the V2 model runner where a single image's soft-token span exceeds `sliding_window` (i.e. `max_soft_tokens=1120` against `sliding_window=1024`; the default `num_soft_tokens=280` on the released checkpoints is unaffected, so most serving configurations see no change at all). The expected effect is the one measured in #47217 — restoring blockwise-bidirectional image attention instead of causal-only — with V1 as the reference implementation. I have not re-run the end-to-end vision eval from #47217 under V2; I don't have a Hopper box free for it right now. Flagging that gap explicitly rather than implying coverage I don't have. If a reviewer wants those numbers before merging, say so and I'll get them. ## AI assistance AI assistance (Claude) was used on this PR: tracing the V1/V2 divergence, drafting the patch and tests, and drafting this description. The commit carries a `Co-authored-by:` trailer. I have reviewed every changed line, run the tests and linters reported above, and can defend the change end-to-end. ## Related - #47217 — the V1 fix this ports (merged) - #42175 — introduced Gemma4 FA4 + FlashAttention `mm_prefix` support", "url": "https://github.com/vllm-project/vllm/pull/50539", "createdAt": "2026-07-31T06:51:39Z", "updatedAt": "2026-08-12T19:29:36Z", "timestamp": "2026-08-12T19:29:36Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "mrv2" ], "author": "dhruvil237", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:62b774e74d669f74c976", "signalId": "github:vllm-project/vllm:issue:47839", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:47839", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Packed Variable Length Speculative Decoding", "text": "### Motivation. In speculative decoding, we can save verification cost by selectively verifying a different number of draft tokens per request in each step. There are various techniques of selecting and tuning which / how many tokens to submit for verification, but all share a common benefit: reducing verification overhead with per-request variable-length query lengths in a batch. ### Proposed Change. We will support verification input batches with heterogeneous query lengths for each decode request. We will separate the concerns of total verification token budget and allocation of that budget among the draft tokens. The scheduler chooses a host-visible total token budget, and the worker distributes that budget across requests entirely on-device, avoiding serialization of the query lengths to/from the host. This allows us to maintain asynchronous scheduling and CUDA graph compatibility. This will involve changes to the scheduler, model runner input preparation, cuda graph management, and speculator-scheduler implementations (for the actual logic to decide which tokens to propose). I propose to build off of the [batch-level Dynamic Speculative Decoding](https://github.com/vllm-project/vllm/pull/45953) support (which already has full-graph compatibility) in ModelRunnerV2. ### Feedback Period. _No response_ ### CC List. @TheEpicDolphin @WoosukKwon @LucasWilkinson @mgoin @ekagra-ranjan @simon-mo @zyongye ### Any Other Things. ## Challenges ### Varlen kernel support Attention kernels take as input the number of *requests* in addition to the query offset pointers. For decoding, many attention kernels omit the latter and simply assume either 1 (autoregressive) or N (uniform speculative) query lengths per request. Many, but not all, attention kernels already support variable length queries per request: - FlashAttention Decode (Hopper/Blackwell GQA): supports varlen queries via `flash_attn_varlen_func` arguments `max_seqlen_q` and `cu_seqlens_q`. (Definitely FA3, not sure about varlen full-graph for FA4) - FlashInfer TRTLLM-Gen Decode (Blackwell GQA): supports varlen queries via `max_q_len` and `cum_seq_lens_q`, but not plugged into vLLM yet - FlashInfer TRTLLM-Gen MLA Decode (Blackwell MLA): no support - FlashAttention MLA: Currently uniform-query CG compatibility, but supports varlen Q, might be compatible - DeepSeek V4 Indexed attention: Underlying Sparse MLA kernel supports varlen, but seems to need some plumbing to fully support varlen - MiniMax M3 Sparse attention: no support - FlashMLA (Hopper MLA): no support - GDN: seems possible with some extra work on the metadata builder - Mamba/Mamba2: seems possible with a bit of plumbing Thus we will have to gate support to only the attention backends which have varlen support implemented, and it'll take quite a bit of follow-up work to enable this for most backends. ### CUDA Graph Compatibility We currently have support for FULL CUDA Graphs with different query lengths for different batch sizes. This is done by keying the graphs by `num_reqs` _and_ `num_tokens` _and_ `uniform_token_count`. We can extend this: for each `num_reqs` (binned to powers of 2 / multiples of 8 / etc.) from `1` up to `max_num_seqs`, we record a separate set of graphs for each `num_query_tokens_per_request` from `1` (no spec tokens) up to (1 + max_num_speculative_tokens). To unlock true dynamism, we break the connection between `num_query_tokens_per_request` and `max_q_len`: we always permit up to `max_num_speculative_tokens` for a single request, a pass `max_q_len=max_num_speculative_tokens` in every attention kernel invocation, and thus allow some requests to have fewer than `num_query_tokens_per_request` and some to have more. This means that the scheduler effectively is only deciding `mean_num_query_tokens_per_request` (and will likely use heuristics from the workers to make that decision, see below). Then all we need to do is leverage `cu_seqlens_q` to have variable query start locations within the block, mark the remaining tokens in the suffix with `is_padding`, and pad `num_reqs` to `num_reqs_padded` by appending to `cu_seqlens_q` with 0-query-length requests which are ignored by the kernel. Thus, the attention kernel is, for a given `num_reqs_padded` and `num_tokens = int(mean_num_query_tokens_per_request * num_reqs_unpadded)`, always invoked with inputs of consistent shape and address. Note, that we can support non-integer `mean_num_query_tokens_per_request` as long as we record more graphs for `num_tokens` values in between `num_reqs * i` and `num_reqs * (i + 1)`, though they will not be perfectly uniform; we will have to support some extra padding or handle this specifically when doing capture. ### Supporting mixed batches Supporting mixed batches with FULL CUDA Graphs is appealing as host-overhead in short prefills is a real concern, but this is out-of-scope for this RFC. For PIECEWISE graphs, this is simpler. The main remaining consideration is compatibility with the `is_padding` optimization, which allows some kernels (e.g. MoE kernels) to skip work on graph-padding tokens. With variable query lengths in the decode, we may end up with some decode padding (to match the number of actual query tokens with the number of expected query tokens) in addition to graph padding. This should be avoided, either by packing the decodes tightly in the batch or reordering prefills and extends to the front of the batch, to share the query padding with the graph's total-token-count padding. It's unclear to me if it will be easy to implement the former, as the total number of used query tokens need not be known on the host, but either solution seems feasible here. The easiest short-term solution seems to be to **enforce no padding in the query block**. Since we always have more draft tokens than we will need to admit in our budget, we can just fill all allocated slots with draft tokens even if our performance model might indicate it's favorable to skip them. Even though they might be skippable via `is_padding`, there are still some attention and constant overheads to be paid, so the price shouldn't be too extreme here. ### Supporting Data Parallelism Some coordination across workers will be necessary to synchronize which graph is being executed, i.e. which (`num_reqs_padded` x `num_tokens_padded`) pair is active. Seems like this should fit naturally with our existing DP sync, but I'm not an expert in this area. ### Scheduler-Speculator awareness In vLLM, our scheduler lives in a separate process; yet I propose to let it decide `mean_num_query_tokens_per_request` when scheduling. In order to have the scheduler make decisions about how many total query tokens to permit, it will need some information from the model runner. I propose to extend the Speculator interface to allow the scheduler to \"query\" for \"hints\" such as how many query tokens should be issued on average per request. It is important to maintain asynchronous scheduling support here; in the DSpark report, they suggest using information delayed by 2 engine iterations in order to schedule the next step. This seems very reasonable, as acceptance rates over batches generally have significant temporal locality over multiple steps (reasoning v.s. final response, conversation domain, blocks of code/math/formatted output, etc). This would probably be easier to implement if the scheduler was replicated into each worker, so we don't need to cross an IPC boundary, but this is what we have today. I'm not certain that the scheduler needs to know about this at all. Conceptually I feel like it's the role of the scheduler to be choosing how many tokens to put in the batch, but besides allocating KV slots and selecting which requests to run (both of which are pretty agnostic to the variable-length decode tokens, the scheduler can just allocate the slots optimistically), I'm not sure why the current scheduler would need to be involved. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47839", "createdAt": "2026-07-07T08:04:55Z", "updatedAt": "2026-08-12T19:27:50Z", "timestamp": "2026-08-12T19:27:50Z", "metrics": { "reactions": 10, "comments": 2 }, "labels": [ "RFC", "dflash" ], "author": "benchislett", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:5b8d474d3f457d39d9d2", "signalId": "github:vllm-project/vllm:pull_request:52039", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52039", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix collect_env.py crash when pip list command fails", "text": "## Purpose Fixes the `collect_env.py` crash noted in #52027: in a uv venv with `uv` not on `PATH`, the pip-list command fails, `run_and_read_all` returns `None`, and `run_with_pip()` calls `None.splitlines()` → `AttributeError`. Separate from #52026 (the logging fix for the same issue), which doesn't touch this file. ## Changes - Return `None` when the list command fails, instead of calling `.splitlines()` on it (matches `get_conda_packages`; callers already handle `None`). - Remove a leftover `print(\"uv is set\")` debug line. ## Test `tests/test_collect_env.py`: calls `get_pip_packages` with a failing command and checks it returns `None` instead of crashing. ```bash .venv/bin/python -m pytest tests/test_collect_env.py -v ``` Passing; ruff clean. ## Related - Refs [#52027](https://github.com/vllm-project/vllm/issues/52027) (fixes the collect_env.py part) - Separate from #52026 (the logging fix for the same issue); doesn't touch this file Notes AI assistance was used; I reviewed every line and ran the test. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52039", "createdAt": "2026-08-12T19:14:16Z", "updatedAt": "2026-08-12T19:17:15Z", "timestamp": "2026-08-12T19:17:15Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "jahnavi-yelamanchi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dffad91d016482d00874", "signalId": "github:vllm-project/vllm:issue:52038", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52038", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: LoRA adapter support for DFlash speculative decoding draft models", "text": "### Motivation. vLLM supports serving many LoRA adapters over a single base target model, but speculative decoding still requires a separate, fully-trained draft model per domain. When serving many domains, this is expensive: each domain needs its own drafter (e.g. 0.8B) held in memory, with corresponding I/O cost to load and swap. This RFC proposes applying LoRA adapters to the DFlash drafter, so a single base drafter can be specialized per-domain via small adapters instead of full drafters. In our experiments an R64 adapter is ~28x smaller than the 0.8B drafter it replaces, substantially reducing memory footprint and I/O latency when serving many domains, while keeping drafter quality within ~2% of a fully-trained per-domain drafter (measured by acceptance length behavior). This mirrors the efficiency win vLLM already provides for target-model LoRA, extended to the speculative-decoding drafter. ### Proposed Change. Allow a LoRA adapter to be applied to the DFlash drafter, independent of the target model: - The adapter wraps all drafter layers except `lm_head` and `embed_tokens`, loaded from a stage-2-trained LoRA checkpoint. - The drafter LoRA is specified via additional arguments in the speculative config, distinct from any target-model adapter. - The correct adapter is routed to the drafter and applied along its forward path. Initial support targets DFlash drafters specifically (tested with a GPT-OSS-120B target and a 0.8B DFlash drafter). This is distinct from target-side multi-adapter work (#49705, #48297) and from prior Eagle-style draft-model attempts. **Prior art / known issues** Closed PR #11966 (V0, Eagle-era) attempted LoRA + spec decode and hit a vocab-size mismatch (e.g. 768 vs 4096) from differing vocab padding on draft vs. target, and disabled draft-side LoRA \"until the API interface is changed to inject the corresponding LoRA adapter for the draft model.\" In our DFlash testing we did not encounter this mismatch — likely because the adapter excludes `lm_head` and `embed_tokens`, avoiding the padded-vocab layers. We'd welcome maintainer input on whether this holds across all DFlash configurations. **Alternatives considered** - Full per-domain drafters (status quo): simple but costly — one full drafter per domain in memory. - Offline-merging the LoRA into drafter weights: avoids runtime adapter logic but still stores a full merged drafter per domain, forfeiting the 28x saving and per-request swappability. Serving the adapter dynamically preserves both the memory saving and per-request swappability. **Testing & benchmarks** Acceptance behavior and throughput comparing base drafter, LoRA'd drafter, and a fully-trained per-domain drafter; memory-footprint comparison (adapter vs. full drafter) across N domains. ### Feedback Period. One week. ### CC List. @LucasWilkinson @WoosukKwon ### Any Other Things. There are some details I can't share about the experiment results because that involved classified data and domains while my time at Capital One AI Foundations where I worked on this project, but I will share what I can based on your questions. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52038", "createdAt": "2026-08-12T19:09:15Z", "updatedAt": "2026-08-12T19:09:15Z", "timestamp": "2026-08-12T19:09:15Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "RFC" ], "author": "anshkaggarwal22", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:fc9e2911ecd200b72f5a", "signalId": "github:vllm-project/vllm:issue:50587", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50587", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: Kimi K3 Performance Optimization", "text": "### 🚀 The feature, motivation and pitch Tasks - [x] https://github.com/vllm-project/vllm/pull/50383 @jeejeelee - [x] https://github.com/vllm-project/vllm/pull/51146 @jeejeelee - [ ] https://github.com/vllm-project/vllm/pull/50478 @robertgshaw2-redhat - [x] https://github.com/vllm-project/vllm/pull/50484 @GirasoleY - [x] https://github.com/vllm-project/vllm/pull/50585 @yewentao256 - [ ] https://github.com/vllm-project/vllm/pull/50634 @JohnQinAMD - [x] https://github.com/vllm-project/vllm/pull/50656 @tlrmchlsmth - [x] https://github.com/vllm-project/vllm/pull/50912 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51070 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51311 @yewentao256 - [x] https://github.com/vllm-project/vllm/pull/51725 + https://github.com/vllm-project/vllm/pull/51726 @yewentao256 ### Alternatives _No response_ ### Additional context I don't have enough GPU resources to run an end-to-end benchmark currently. If you have available GPU capacity and are willing to share access, I would greatly appreciate it! ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/50587", "createdAt": "2026-07-31T15:49:29Z", "updatedAt": "2026-08-12T19:06:52Z", "timestamp": "2026-08-12T19:06:52Z", "metrics": { "reactions": 4, "comments": 0 }, "labels": [ "feature request", "kimi", "k3" ], "author": "yewentao256", "state": "open", "assignees": [ "yewentao256" ], "change": "updated" } }, { "id": "event:2e91ad739aa905290a82", "signalId": "github:vllm-project/vllm:pull_request:50532", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50532", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch", "text": "# [Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch ## Purpose Split out of #50488 at a maintainer's request, so each fix is reviewable on its own. V2 classified a batch as uniform decode from its shape alone (`num_tokens == num_reqs * max_query_len`) and nothing about whether the requests were actually decoding. A prompt chunk of exactly `1 + num_speculative_tokens` tokens has that shape, so a prefill was dispatched with `cg_mode=FULL` and replayed the captured spec-verify decode graph over prompt tokens. Without spec decoding the same collision exists between a 1-token prompt chunk and the q=1 decode graph. Dense models survive it, because spec-verify attention over a fresh sequence is the same computation as prefill attention. Recurrent-state models do not: the mis-dispatched batch still builds prefill metadata, so the GDN builder's persistent-buffer refresh is skipped and the replayed graph reads capture-time state indices. `state_idx <= 0` makes the fused GDN kernels skip the recurrent state read and store, so the prompt's conv/SSM state is never created. The gate is per batch, so genuine decodes sharing a batch with such a chunk lose their refresh too. This is the mechanism behind #49918. The predicate now requires every request in the batch to be past its prefill, not merely for the batch to have the right shape. Dummy batches (DP padding, memory profiling, warmup) are uniform by construction and have no request state to consult, so they stay classified by shape, mirroring `InputBatch.make_dummy`. The second commit is a performance follow-on in the same path: the shape test is O(1) and rejects every mixed prefill/decode batch, which is most of them, so it is hoisted ahead of the `np.fromiter` gather of request state rather than run after it. A third commit covers `multi_module_mtp/speculator.py`, which landed on main after this branch was cut. It is a copy of the autoregressive speculator, comment included, and repeats the shape-only call, so it carries the same defect. The guard added alongside it is written over every speculator in the package rather than over that one file, because copying is how the call reappeared; it fails on the offending line without the change and passes with it. ## Test Plan - `tests/v1/worker/test_gpu_batch_ordering.py` — the classification itself, including the prompt-chunk shapes that previously passed and the dummy-run path that must keep passing, plus the dispatch consequences: an 8-token prefill dispatches PIECEWISE and answers the prompt, control outputs are unchanged byte for byte, and genuine decode steps still replay their FULL graphs. A further case pins that the O(1) shape test rejects a non-uniform batch on its own, which is the ordering the second commit relies on. - `tests/v1/spec_decode/test_dynamic_sd_cug.py` — the dynamic speculative schedule path over the same dispatch. ## Test Result `pytest tests/v1/spec_decode/test_dynamic_sd_cug.py tests/v1/worker/test_gpu_batch_ordering.py` — **14 passed**, run on an H100 sandbox against this branch overlaid on the pinned nightly wheel (`0.26.1rc1.dev77+g6f91edf96`). Revert check: with the decoding clause removed from the predicate, `test_prompt_chunks_shaped_like_spec_decode_miss_the_full_graph` and `test_prompt_chunk_of_decode_query_len_is_not_uniform_decode` fail while the uniform-decode and dummy-run cases still pass. `ruff check` and `ruff format --check` are clean on every touched file. ### Model evaluation `tests/evals/gsm8k/gsm8k_eval.py`, 400 questions, 5-shot, greedy, on `Qwen/Qwen3-4B` with the published `z-lab/Qwen3-4B-DFlash-b16` drafter at `num_speculative_tokens=16`, V2 model runner, single H100: **accuracy 0.880, 0.000 invalid, 400 questions**. This was measured on the combined set of fixes before the split, so it covers this change together with the others rather than in isolation. ## Related - #49918 — reproduces this independently of proposer (MTP and ngram), quantization and batch size, with PIECEWISE or enforce-eager as workarounds. Split from #50488, alongside the warmup lookahead reservation fix and the CUDA graph capture-size fix. I checked for duplicate and overlapping open PRs (`gh pr list --search` on \"uniform decode cudagraph\", \"uniform decode dispatch\", \"prefill full cudagraph\") and found none besides #49918, which is an issue rather than a fix. I used AI assistance (Cursor) to draft, test, and validate this change, and I reviewed every changed line before submitting.", "url": "https://github.com/vllm-project/vllm/pull/50532", "createdAt": "2026-07-31T05:58:03Z", "updatedAt": "2026-08-12T19:05:14Z", "timestamp": "2026-08-12T19:05:14Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "speculative-decoding", "needs-rebase", "nvidia", "mrv2" ], "author": "rchalamala", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:80908163cf0314c3c892", "signalId": "github:vllm-project/vllm:pull_request:44969", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44969", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Gating more ROCm tests", "text": "Add 22 AMD mirrors for existing CI test groups as the Stage D gating expansion. ### `mi250_1` - Cudagraph - Docker Build Metadata - Kernels Mamba Test - Kernels Helion Test - Language Models Test (PPL) - Language Models Test (MTEB) ### `mi250_2` - Plugin Tests (2 GPUs) ### `mi300_1` - Model Runner V2 Spec Decode - PyTorch Compilation Unit Tests - Speculators Correctness - Language Models Test (Extended Generation) - Entrypoints Integration (Responses API) - Entrypoints Integration (Multimodal) - Async Engine, Inputs, Utils, Worker ### `mi300_2` - Model Runner V2 Distributed (2 GPUs) - Distributed Model Tests (2 GPUs) ### `mi300_4` - V1 e2e (4xH100-4xMI300) - LoRA TP (Distributed) ### `mi355_1` - Entrypoints Unit Tests - Multi-Modal Processor - Multi-Modal Models (Extended Generation 2) - Multi-Modal Models (Extended Generation 3)", "url": "https://github.com/vllm-project/vllm/pull/44969", "createdAt": "2026-06-09T06:56:07Z", "updatedAt": "2026-08-12T19:02:15Z", "timestamp": "2026-08-12T19:02:15Z", "metrics": { "reactions": 2, "comments": 17 }, "labels": [ "rocm", "ready", "ci/build", "v1", "nvidia" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9b169089b0674b4b8831", "signalId": "github:vllm-project/vllm:issue:49012", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:49012", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: nvfp4 reshape_and_cache_flash assumes NHD layout — silently mis-swizzles HND caches when num_kv_heads % 4 == 0", "text": "### Environment ``` GPU: RTX 5090 (SM120, 32 GB) | driver 595.71.05, CUDA 13.2 vLLM 0.25.1 | torch 2.11.0 ``` ### Bug `reshape_and_cache_nvfp4_dispatch` (`csrc/.../nvfp4_kv_cache_kernels.cu:210`) reads `block_size` from tensor dim-1, which is only correct for NHD caches. With an HND cache it picks up `num_kv_heads` instead: - if `num_kv_heads % 4 != 0` you get a loud error (\"block_size must be divisible by 4 for NVFP4 KV cache swizzle\") - confusing but at least visible - if `num_kv_heads % 4 == 0` (e.g. 8 KV heads) it swizzles across the wrong dimension and produces silently corrupt cache contents Repro is a ~30-line script: write random K/V through the op with an HND-shaped cache, dequantize, compare against the input. We hit this via `VLLM_KV_CACHE_LAYOUT=HND` on SM120 but the kernel behavior is layout-dependent regardless of arch. ### Good In NHD the kernel is excellent - we byte-compared its output (data + both scale-factor tensors) against flashinfer's `nvfp4_quantize_paged_kv_cache` on identical input and got 100% identical results, including with non-unit global scales (vLLM's `k_scale` convention lines up with `1/global_sf` on the flashinfer side). So this is purely a layout-indexing gap: either add HND indexing or reject non-NHD caches explicitly. Found while prototyping nvfp4-KV on SM120 (#49011).", "url": "https://github.com/vllm-project/vllm/issues/49012", "createdAt": "2026-07-18T04:24:00Z", "updatedAt": "2026-08-12T18:58:02Z", "timestamp": "2026-08-12T18:58:02Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "0xdespot", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5499d789eea18731eaa0", "signalId": "github:vllm-project/vllm:pull_request:52036", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52036", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails", "text": "## Purpose Fixes #51842. Serving `openbmb/MiniCPM-V-4_6` fails during weight loading, before the server starts: ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters ... are: {'qkv_proj.weight', 'qkv_proj.bias', 'out_proj.weight', 'out_proj.bias'} **Root cause.** The MiniCPM-V-4.6 ViT attention checkpoint stores separate `q_proj`/`k_proj`/`v_proj`, while vLLM fuses them into a single `QKVParallelLinear` (`qkv_proj`). `MiniCPMV4_6ViTWindowAttentionSelfAttn` tried to fuse them with a class-level `WeightsMapper(orig_to_new_stacked=...)` applied inside the submodule's own `load_weights`. Two problems make this a no-op: 1. `AutoWeightsLoader` strips the module prefix before dispatching to a submodule's `load_weights`, so the keys arriving here are bare (`k_proj.weight`), but the mapper substrings carry a leading dot (`.k_proj`) and never match. The unmapped `k_proj` then hits the \"no parameter named 'k_proj'\" error. 2. The leading dots cannot simply be dropped: `orig_to_new_stacked` applies every matching rule without breaking, and `qkv_proj` contains the substring `v_proj`, so a de-dotted mapper would re-map `k_proj → qkv_proj → qkqkv_proj` with the wrong shard id. ## Test Plan - Repro on 2× T4 (Turing, so `--dtype float16`), `tensor_parallel_size=2`, which triggers the failing weight-load path before and after the fix: ```python from vllm import LLM, SamplingParams llm = LLM(model=\"openbmb/MiniCPM-V-4_6\", trust_remote_code=True, dtype=\"float16\", tensor_parallel_size=2, max_model_len=4096, enforce_eager=True) print(llm.generate([\"Describe yourself in one sentence.\"], SamplingParams(max_tokens=32))[0].outputs[0].text) - Lint/compile on the change: ruff check and python -m py_compile. ## Test Result - Before: startup aborts with ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn . - After: model loads successfully ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52036", "createdAt": "2026-08-12T18:50:54Z", "updatedAt": "2026-08-12T18:56:09Z", "timestamp": "2026-08-12T18:56:09Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "songzhouran", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d4f2b79f4c0434841771", "signalId": "github:vllm-project/vllm:issue:51456", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51456", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: online FP8 (--quantization fp8) produces corrupted, non-EOS-terminating output on Qwen2.5-1.5B-Instruct", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 3.28.3 Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-79-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect GPU models and configuration : GPU 0: NVIDIA H200 Nvidia driver version : 570.124.06 HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.6 [pip3] numpy==2.1.2 [pip3] torch==2.10.0 [pip3] transformers==5.14.1 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== vLLM Version : 0.19.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled ``` (Full output, including the complete `pip freeze` and NIC topology dump, omitted here for length -- available on request. GPU/CUDA/vLLM-relevant fields above are the complete, unedited values from the script.)
### 🐛 Describe the bug `vllm serve --quantization fp8` (online/on-the-fly W8A8, checkpoint not pre-quantized) produces incoherent, mixed-script output from the *first* generated token and essentially never reaches EOS -- not degraded-but-readable text, and not the kind of subtle numerical drift normal quantization noise causes. **Repro:** ```bash vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8 ``` ```bash curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\": \"Qwen/Qwen2.5-1.5B-Instruct\", \"messages\": [ {\"role\": \"system\", \"content\": \"You are a warm, emotionally present conversational companion in a live voice call. Reply in one or two short spoken sentences. Never use lists or markdown.\"}, {\"role\": \"user\", \"content\": \"I have been thinking about something all week and I cannot shake it.\"} ], \"max_tokens\": 80, \"temperature\": 0.7, \"stream\": false, \"seed\": 0 }' ``` **Expected** (and what the identical request against the same checkpoint with no `--quantization` flag returns): ``` \"content\": \"It's okay to spend time on thoughts that matter to you. What is the thought or idea you've been pondering? Sometimes talking through what's on your mind can help clear things up.\" \"finish_reason\": \"stop\" ``` **Actual**, `--quantization fp8`, same prompt/seed, re-confirmed live immediately before filing this issue: ``` \"content\": \"袅 解.Resolve\\\";} yabǃ qualidade感じるolving (...ñas andaLab yab standardized zweolving � lastnameolics\\tresolvefeatures ectñas learlab尼亚resolve yab.resolveresolve clock benchmarks anda ǃ yab！ijase第八ourcem原文地址olv巬 elimination eliminatedany세resolution qi!\\n\\n\\n /> float: for _ in range(WARMUP): fn() torch.accelerator.synchronize() samples = [] for _ in range(REPEATS): start = time.perf_counter() for _ in range(ITERATIONS): fn() torch.accelerator.synchronize() samples.append((time.perf_counter() - start) * 1000 / ITERATIONS) return statistics.median(samples) def run_case(T: int) -> tuple[float, float]: num_seqs = min(N, T) q, k, v, g = [ torch.randn(1, T, H, D, dtype=torch.bfloat16, device=DEVICE) for _ in range(4) ] beta = torch.randn(1, T, H, dtype=torch.bfloat16, device=DEVICE) A_log = torch.randn(H, dtype=torch.float32, device=DEVICE) dt_bias = torch.randn(H, D, dtype=torch.float32, device=DEVICE) initial_state = torch.randn(num_seqs, H, D, D, dtype=torch.float32, device=DEVICE) cu_seqlens = torch.arange(num_seqs + 1, dtype=torch.int32, device=DEVICE) * ( T // num_seqs ) workspace_size = torch.ops._flashkda_C.get_workspace_size(T, H, num_seqs) core_out = torch.empty_like(v) state_cache = torch.empty_like(initial_state) final_state = torch.empty_like(initial_state) workspace = torch.empty(workspace_size, dtype=torch.uint8, device=DEVICE) def before() -> None: out = torch.empty_like(v) final = torch.empty_like(initial_state) ws = torch.empty(workspace_size, dtype=torch.uint8, device=DEVICE) torch.ops._flashkda_C.fwd( q, k, v, g, beta, D**-0.5, out, ws, A_log, dt_bias, -3.0, initial_state, final, cu_seqlens, ) core_out.copy_(out) state_cache.copy_(final) def after() -> None: _, final = _flashkda_prefill( q, k, v, g, beta, A_log, dt_bias, -3.0, initial_state, cu_seqlens, core_out, final_state, workspace, ) state_cache.copy_(final) return benchmark(before), benchmark(after) @torch.inference_mode() def main() -> None: print(f\"{'tokens':>8} {'before (ms)':>12} {'after (ms)':>11} {'speedup':>9}\") for tokens in TOKENS: before_ms, after_ms = run_case(tokens) print( f\"{tokens:>8} {before_ms:>12.3f} {after_ms:>11.3f} \" f\"{before_ms / after_ms:>8.3f}x\" ) if __name__ == \"__main__\": main() ``` And we can get ```bash tokens before (ms) after (ms) speedup 1 0.037 0.026 1.406x 32 0.042 0.034 1.263x 64 0.044 0.033 1.307x 128 0.042 0.033 1.278x 256 0.043 0.033 1.285x 512 0.043 0.033 1.324x 1024 0.042 0.033 1.283x 2048 0.048 0.044 1.113x ```", "url": "https://github.com/vllm-project/vllm/pull/51311", "createdAt": "2026-08-06T21:19:06Z", "updatedAt": "2026-08-12T18:51:50Z", "timestamp": "2026-08-12T18:51:50Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "ready", "kimi", "k3" ], "author": "yewentao256", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1431e8cd8df99145c6ca", "signalId": "github:vllm-project/vllm:pull_request:51723", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51723", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Clamp block table indices in align mode to prevent OOB gather", "text": "In mamba_cache_mode='align' with speculative decoding, sequences with seq_len near max_model_len caused indices_to_gather to exceed the block table column count, triggering a CUDA device-side assert in torch.gather. Fixes #42084 ## Purpose Fix an out-of-bounds `torch.gather` in `mamba_get_block_table_tensor` when `mamba_cache_mode='align'` is used together with speculative decoding and a large `max_model_len`. In `align` mode the Mamba block table has shape `(#requests, cdiv(max_model_len, block_size))` — one column per block over the full context window, with **no** extra speculative columns appended. When computing the gather indices: ```python start_indices = (seq_lens - 1) // kv_cache_spec.block_size offsets = torch.arange(1 + kv_cache_spec.num_speculative_blocks, ...) indices_to_gather = (start_indices.unsqueeze(1) + offsets).to(torch.int64) ``` for a sequence with `seq_len` near `max_model_len`, `start_indices` is at or near the last valid column. Adding `num_speculative_blocks` (which is `num_speculative_tokens` when using a draft model) then pushes `indices_to_gather` past the end of the block table, triggering: ``` torch.AcceleratorError: CUDA error: device-side assert triggered ``` This crashes the entire EngineCore process, killing all in-flight requests. **Fix:** add a single upper-bound clamp on `indices_to_gather` before the gather call, mirroring the identical `.clip(max=block_table.shape[1] - 1)` pattern already used in `make_local_attention_virtual_batches` (line 452 in the same file): ```python indices_to_gather.clamp_(max=block_table.shape[1] - 1) ``` Fixes #42084 ## Test Plan New CPU-only regression test added to `tests/v1/attention/test_mamba_update_block_table.py`: ```bash pytest tests/v1/attention/test_mamba_update_block_table.py::test_mamba_get_block_table_tensor_align_mode_oob_clamp -v ``` The test constructs a block table sized for `align` mode (`cdiv(max_model_len, block_size)` columns, no speculative tail), then calls `mamba_get_block_table_tensor` with `seq_lens` near `max_model_len` and `num_speculative_blocks=2`. Without the fix the test raises an index error; with the fix it passes and verifies the output shape and value bounds. ## Test Result **Before fix:** ``` IndexError: index out of bounds ``` **After fix:** ``` PASSED tests/v1/attention/test_mamba_update_block_table.py::test_mamba_get_block_table_tensor_align_mode_oob_clamp ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51723", "createdAt": "2026-08-10T19:13:48Z", "updatedAt": "2026-08-12T18:48:11Z", "timestamp": "2026-08-12T18:48:11Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "shernshiou", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:dce0d862c970cb0be327", "signalId": "github:vllm-project/vllm:pull_request:50388", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50388", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Fix ValueError on KV load failure with a hybrid KV cache", "text": "## Purpose `_update_requests_with_invalid_blocks` unpacks `get_block_ids()` -- which returns one block-id list per KV cache group -- into a single name: # TODO (davidb): add support for hybrid memory allocator (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id) so the first reported invalid block on a hybrid (Mamba/SSM plus attention) model raises `ValueError: too many values to unpack (expected 1)` inside the scheduler. Nothing catches it, so EngineCore calls `_send_engine_dead()` and exits: one recoverable read failure takes down every request on the instance, not just the one that failed. A hybrid model's groups sit on different block grids, so the same block index means a different token position in each and there is no single longest valid prefix to truncate at. Rather than build a block-level answer for that -- per- block error handling is being removed in https://github.com/vllm-project/vllm/pull/42461 -- affected hybrid requests recompute in full, which is the granularity that replacement moves to anyway. Single-group models keep the existing block-level truncation untouched. Blocks are still added to `marked_invalid_block_ids` so a request sharing one of them is rescheduled as before. The null block needs excluding on this path. In Mamba `align` mode a group's block list is mostly the shared null block -- a 9-chunk request looks like `[0, 0, 0, 0, 0, 0, 0, 0, 10]`, only the aligned snapshot is backed -- so without the guard a reported null block would restart unrelated requests, and `blocks_to_evict` would collect a block that belongs to no request. ## Test Plan Use Kimi-Linear-48B-A3B (TP=2, 4 KV cache groups). And also: `pytest -v tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py` ## Test Result Verified end to end on Kimi-Linear-48B-A3B (TP=2, 4 KV cache groups). ``` (vllm-dev) root@kebe-gpu-dev-0:~/vllm# pytest -v tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option \"asyncio_default_fixture_loop_scope\" is unset. The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: \"function\", \"class\", \"module\", \"package\", \"session\" warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET)) ========================================================================= test session starts ========================================================================= platform linux -- Python 3.12.12, pytest-8.3.5, pluggy-1.5.0 -- /home/jovyan/conda/envs/vllm-dev/bin/python3.12 cachedir: .pytest_cache rootdir: /home/jovyan/vllm configfile: pyproject.toml plugins: asyncio-1.0.0, anyio-4.9.0 asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 14 items tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs0] PASSED [ 7%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs1] PASSED [ 14%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs2] PASSED [ 21%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs0] PASSED [ 28%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs1] PASSED [ 35%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure[100-99-invalid_block_idxs2] PASSED [ 42%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs0] PASSED [ 50%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs1] PASSED [ 57%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_sync_load_failure_with_shared_blocks[100-99-50-invalid_block_idxs2] PASSED [ 64%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_progressive_load_failure[100-99-invalid_block_idxs0] PASSED [ 71%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_progressive_load_failure[100-99-invalid_block_idxs1] PASSED [ 78%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_recomputes_whole_request[0] PASSED [ 85%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_recomputes_whole_request[1] PASSED [ 92%] tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_hybrid_load_failure_ignores_null_block PASSED [100%] ========================================================================== warnings summary =========================================================================== ../conda/envs/vllm-dev/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py::test_async_load_failure[100-99-invalid_block_idxs0] /home/jovyan/conda/envs/vllm-dev/lib/python3.12/site-packages/transformers/models/gpt2/tokenization_gpt2.py:110: DeprecationWarning: Deprecated in 0.9.0: BPE.__init__ will not create from files anymore, try `BPE.from_file` instead BPE( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================== 14 passed, 15 warnings in 33.95s =================================================================== ``` ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50388", "createdAt": "2026-07-30T05:51:10Z", "updatedAt": "2026-08-12T18:45:35Z", "timestamp": "2026-08-12T18:45:35Z", "metrics": { "reactions": 3, "comments": 4 }, "labels": [ "v1", "kv-connector" ], "author": "kebe7jun", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b41316f1480ffab45ebc", "signalId": "github:vllm-project/vllm:pull_request:51994", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51994", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Model] Fix DiffusionGemma silently freezing attention mask under CUDA graph replay", "text": "## Purpose While looking into whether FULL CUDA graphs are worth the added complexity for DiffusionGemma, I found that `DiffusionGemmaModelState._causal_buf` (the per-request flag that switches each request between causal/encoder and bidirectional/denoise attention) stops being updated once `CUDAGraphMode.FULL` graphs are captured. It's bool, but FlashAttentionMetadataBuilder.build() casts non-int32 causal tensors via `causal.to(torch.int32)`, allocating a new tensor every call. A FULL-mode graph binds to the address it saw at capture time, so once captured it keeps reading that orphaned tensor forever. Later updates to `_causal_buf` never reach it. Any request served through an already-captured FULL graph gets whatever causal/bidirectional pattern existed at capture (warmup) time for its slot, for the graph's whole lifetime, silently. This isn't opt-in: vLLM's default optimization level (O2) resolves cudagraph_mode to FULL_AND_PIECEWISE, whose decode_mode() is FULL, so `vllm serve google/diffusiongemma-26B-A4B-it` with no flags already runs every denoise/decode step under FULL cudagraphs. I couldn't pin observed output corruption on this specifically: an E2E FULL-vs-PIECEWISE comparison was inconclusive, since DiffusionGemma's denoising has its own run-to-run variance of similar magnitude. The capture/replay test below reproduces the mechanism directly instead. So overall, this is a silent bug that likely has some unpredictable impact on output quality. Fix: make `_causal_buf` int32 from construction and update it in place (`.copy_()` / direct assignment) instead of = from a bool source, so build() takes its dtype-matched fast path and never reallocates. Tagging @LucasWilkinson, who authored the original FULL-cudagraph path in #45163. AI-assisted, every line reviewed and tested by the submitter. ## Test Plan `pytest tests/models/test_diffusion_gemma_cudagraph.py -v` ## Test Result New regression test constructs a real DiffusionGemmaModelState and drives _causal_buf through an actual torch.cuda.CUDAGraph capture/replay cycle mirroring FlashAttention's cast. Fails on main: replay returns the stale `[0, 0, 0, 0]` instead of the updated `[1, 1, 1, 1]`. Passes with this patch. Run on H100 (SM90).", "url": "https://github.com/vllm-project/vllm/pull/51994", "createdAt": "2026-08-12T13:49:00Z", "updatedAt": "2026-08-12T18:40:26Z", "timestamp": "2026-08-12T18:40:26Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia" ], "author": "fjosw", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:76f20a3842b4871dc2fb", "signalId": "github:vllm-project/vllm:pull_request:51207", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51207", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bolins/shm tensor arena", "text": " ## Purpose `MessageQueue.enqueue` (`shm_broadcast.py`) already routes CPU tensors out-of-band via `_reduce_tensor` (protocol-5 `PickleBuffer`, #48442), which removed the dominant cost of the old in-band path — copying tensor bytes into the pickle stream. Two costs remain for a large multimodal `pixel_values` tensor on a TP=N worker: 1. the out-of-band buffer is published **once per node-local reader** (N transport copies for TP=N), and 2. each reader's H2D copies from a **pageable** ZMQ frame (pageable staging + first-touch faults). This PR adds an opt-in **slotted shared-memory tensor arena** that layers *on top of* `_reduce_tensor`: the writer does **one** memcpy into a free slot, every reader takes a **zero-copy view** of the *same* slot (no per-reader transport copy), and the mapping is `cudaHostRegister`-pinned so the H2D is a true DMA. Large contiguous CPU tensors are diverted by `_ArenaPickler.reducer_override`; anything it declines (too small, non-contiguous, or arena exhausted) **falls through to `_reduce_tensor` unchanged**. Net for a ~200 MB image at TP=8: **1 memcpy + 8 pinned DMAs** instead of **8 transport copies + 8 pageable stagings**. The gain scales with TP degree and tensor size, and is zero at TP=1 (no multiproc broadcast) — both now confirmed empirically below. Design doc: `docs/design/shm_tensor_arena.md`. **Safety / scope:** controlled by the `--enable-shm-tensor-arena` / `--no-enable-shm-tensor-arena` CLI flag (`ParallelConfig.enable_shm_tensor_arena`, default on — happy to flip to default-off if preferred); activates only when all queue readers are node-local; the writer **never blocks** (no free/oversized slot → fall back to `_reduce_tensor`, deadlock structurally impossible); slot reuse is gated on an **H2D-completion CUDA event**, so the writer can't overwrite a slot whose async DMA is still in flight — correct under `--async-scheduling`. ## Test Plan A/B against **current `main`**: the baseline arm is the unmodified `vllm/vllm-openai:nightly` image (`nightly-65b7662d3f…`, includes #48442); the arena arm is the same image plus this PR's four files. A third arm runs this PR's code with `--no-enable-shm-tensor-arena` as a disable-control (must be behaviorally identical to base; also bounds run-to-run noise). Two setups, so the TP axis and the model-realism axis are both covered: 1. **TP-scaling sweep** — Qwen2.5-VL-3B, TP ∈ {1, 2, 4} (ViT MLP dim caps this family at TP≤4), image sizes 512²–3072², 150 prompts @ 2 req/s. 2. **Production-scale VLM** — Qwen3-VL-235B-A22B (ModelOpt NVFP4) on 8×B300, TP ∈ {4, 8}, image sizes 1024²/2048²/3072², 120 prompts @ 1 req/s. Common method: `vllm bench serve --dataset-name random-mm` (openai-chat backend), one image per request at a fixed size per cell, poisson arrivals at a **non-saturated** rate, 20 warmups (excludes the one-time arena pinning), seed 42, **2 repeats per cell**. Built-in negative controls: TP=1 (arena structurally inactive) and 512² images (~3 MB `pixel_values`, below the 8 MB divert threshold).
Repro commands Server (one per arm × TP; arena arms = same image + this PR's files): vllm serve $MODEL --tensor-parallel-size $TP --max-model-len 32768 \\ --mm-processor-cache-gb 0 [--enable-shm-tensor-arena | --no-enable-shm-tensor-arena] Load (per image size HW, per repeat): vllm bench serve --model $MODEL --backend openai-chat --endpoint /v1/chat/completions \\ --dataset-name random-mm --random-mm-bucket-config \"{($HW, $HW, 1): 1.0}\" \\ --random-mm-base-items-per-request 1 --random-mm-limit-mm-per-prompt '{\"image\": 1}' \\ --random-input-len 128 --random-output-len 64 --ignore-eos \\ --num-warmups 20 --num-prompts 150 --request-rate 2 --seed 42 \\ --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 --save-result
Still outstanding: port the standalone correctness harness (21 checks: byte-exact round-trip incl. bf16, zero-copy sharing, slot lifecycle with per-reader release, event-gated reuse, exhaustion/oversize/non-contiguous fallbacks) into `tests/distributed/test_shm_broadcast.py` alongside the #48442 tests. ## Test Result **Qwen3-VL-235B (NVFP4) on 8×B300 — E2E p99, base → arena (mean of 2 repeats):** | image (`pixel_values` size) | TP=4 | TP=8 | |---|---:|---:| | 1024² (~23 MB) | 1021 → 995 ms (−2.6%) | 1025 → 999 ms (−2.5%) | | 2048² (~96 MB) | 1584 → 1298 ms (**−18.1%**) | 1807 → 1363 ms (**−24.6%**) | | 3072² (~213 MB) | 3340 → 2771 ms (**−17.0%**) | 4132 → 3083 ms (**−25.4%**) | TTFT p99 moves the same direction (TP=8: −21.5% at 2048², −11.9% at 3072²), and at these payload sizes TTFT p50 improves too (TP=8/3072²: 1185 → 948 ms). Request throughput is identical in every cell (rate-limited by design). **TP-scaling sweep (Qwen2.5-VL-3B — separate setup from the table above):** TP=1 and 512² images are neutral (negative controls behave as designed — the arena never activates); gains appear only at TP≥2 with ≥8 MB tensors and grow with TP and size. On this small model the relative effect is larger than the 235B numbers above — up to −34% E2E p99 at 3072²/TP=4 (3020 → 1992 ms) — because the same transport cost is a much bigger fraction of a 3B's per-request time, and at 2 req/s its large-image cells carry queueing that the stall removal also drains. The 235B table above is the more conservative, production-representative measurement. The disable-control arm on this setup bounds noise at roughly ±5% per cell. **Statistical honesty:** with n=2 repeats, individual p99 cells are noisy (the 235B control shows single-cell swings up to ~±17%). The claim rests on the pattern, not any one cell: the arena is ahead in **all 12** TTFT/E2E comparisons across both models, monotonically in TP degree and tensor size — the shape the mechanism predicts (per-reader transport + pageable staging replaced by one memcpy + pinned DMA), and the E2E-tail bias matches head-of-line blocking on co-scheduled decodes. **Summary:** neutral outside the large-image + TP≥2 regime; **~18–25% p99 reduction** inside it on a production-scale VLM. **Process note:** the disable-control caught a bug in an earlier revision of this PR — `--no-enable-shm-tensor-arena` was silently ignored (the gate read `get_current_vllm_config()` outside its context and fell back to the default). Fixed by threading the flag explicitly through the executor; the control was re-validated inert (0 arena activations) before the numbers above.
Historical: original measurement vs the pre-#48442 in-band baseline Same-seed interactive multimodal workload (uncapped images, TP=4 ×2 workers on one 8-GPU node, ~1.2k aligned requests): TTFT p50 93→89, p90 401→241, p99 1321→862, max 2139→1375 ms; requests >1.5 s: 9→0. These numbers predate #48442 and therefore overstate the delta vs current main; kept only as provenance for the original problem analysis.
---
Essential Elements Checklist - [x] Purpose of the PR - [x] Test plan — executed (repro commands above); unit-test port to `tests/distributed/` still outstanding - [x] Test results — A/B vs current `main` on two models × TP 1–8, with disable-control - [x] Documentation — `docs/design/shm_tensor_arena.md`
", "url": "https://github.com/vllm-project/vllm/pull/51207", "createdAt": "2026-08-05T23:30:59Z", "updatedAt": "2026-08-12T18:29:04Z", "timestamp": "2026-08-12T18:29:04Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "documentation", "nvidia" ], "author": "BolinSNLHM", "state": "open", "assignees": [ "njhill", "Isotr0py", "BolinSNLHM" ], "change": "updated" } }, { "id": "event:91d6ebab250554c12803", "signalId": "github:vllm-project/vllm:pull_request:51359", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51359", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Initialize DeepGemmQuantScaleFMT oracle lazily; bound QuantFP8 UE8M0 packed path to group_size 128", "text": "## Purpose Fix a crash and a latent kernel-contract violation in `QuantFP8`'s DeepGEMM UE8M0 fast path, found by running the FP8 kernel suite on an RTX PRO 6000 (SM120). `DeepGemmQuantScaleFMT.from_oracle()` asserts its cache is populated, but the cache is only filled as a side effect of `_lazy_init()` running for some other DeepGEMM wrapper (it's called at the end of `_lazy_init`, #30898). A `QuantFP8` built with an explicit `use_ue8m0=True` — directly, or via the DeepSeek V3.2 attention path — can hit `from_oracle()` before any DeepGEMM kernel wrapper has run and crashes: ```text AssertionError: DeepGemmQuantScaleFMT oracle cache not initialized ``` On Hopper this path is dormant (the oracle resolves to `FLOAT32_CEIL_UE8M0`, not `UE8M0`), which is why CI never sees it; on Blackwell (SM100/SM120) `tests/kernels/quantization/test_fp8_quant_group.py` fails with this assertion today. Fixing the init order then surfaces two more problems in the same fast path (added in #33047): 1. the packed branch was taken for **any** group size, but `per_token_group_quant_8bit_packed` only supports `group_size == 128` — `group_size=64` raises `RuntimeError: ... only supports group_size==128, got 64`; 2. the test still asserted plain float32 `[B, num_groups]` scales, so it never passed on Blackwell where the fast path legitimately returns packed int32 UE8M0 scales. This PR makes `from_oracle()` initialize on first use (running `_lazy_init()` first so the decision is based on resolved DeepGEMM symbols; without DeepGEMM it yields `FLOAT32`, matching `is_deep_gemm_e8m0_used()`), gates the packed branch on `group_size == 128`, and updates the test to unpack the int32 UE8M0 scales and compare them against `forward_native`. I didn't find existing work on this: searched `oracle cache not initialized`, `DeepGemmQuantScaleFMT`, `init_oracle_cache`, and `test_fp8_quant_group` across issues/PRs — only the original #30898 (refactor that introduced the assert) and #33047 (refactor that introduced the fast path) touch this code. AI assistance (GitHub Copilot CLI) was used to prepare this PR. ## Test Plan On an RTX PRO 6000 Blackwell (SM120, torch 2.11.0+cu130), where the oracle resolves to `UE8M0`: ```bash pytest tests/kernels/quantization/test_fp8_quant_group.py -q # fails before, passes after pytest tests/kernels/quantization/test_fp8_quant.py -q # adjacent suite pre-commit run --files vllm/utils/deep_gemm.py \\ vllm/model_executor/layers/quantization/input_quant_fp8.py \\ tests/kernels/quantization/test_fp8_quant_group.py ``` Plus a standalone repro (fresh process, `QuantFP8(group_shape=GroupShape(1,128), use_ue8m0=True).forward_cuda(...)` under a bare `set_current_vllm_config`), a bit-exactness check of the packed path against `forward_native` for group_size 128 shapes, and an e2e smoke of `Qwen/Qwen3.6-35B-A3B-FP8` (block-FP8, DeepGEMM MoE backend) before/after. ## Test Result - `test_fp8_quant_group.py`: **3 failed on main** (assert crash for `[64,1024,64]` / `[128,2048,128]`, then shape mismatch once the crash is fixed) → **11 passed** with this PR; still 11 passed with the two source fixes reverted-and-reapplied individually confirming each layer. - Standalone repro: crash → works; `group_size=64` with `use_ue8m0=True` now takes the unpacked path instead of raising. - Packed path bit-exactness vs `forward_native` (group_size 128): quantized data 100% equal, unpacked UE8M0 scales exactly equal, for `[16,1024]` and `[128,2048]`. - `test_fp8_quant.py`: 241 passed. - Qwen3.6-35B-A3B-FP8 greedy e2e smoke: output identical before/after. - ruff / ruff-format / typos / mypy hooks and `git diff --check`: clean. Note: `tests/kernels/quantization/test_block_fp8.py` has 100+ pre-existing failures on this SM120 machine (DeepGEMM matmul tolerances, unrelated to this change — identical failures on clean `main`). ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51359", "createdAt": "2026-08-07T05:06:37Z", "updatedAt": "2026-08-12T18:27:51Z", "timestamp": "2026-08-12T18:27:51Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "bug", "ready", "quantization" ], "author": "BabyDrangoner", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:1a5ccd15e1ff91a6c131", "signalId": "github:vllm-project/vllm:pull_request:48116", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48116", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Structured Output][Tool Calling] Use xgrammar's reasoning parser instead vLLM's", "text": "## Purpose Filter reasoning for structured output and tool calling on Xgrammar's side instead of vLLM's side to simplify logic of scheduler. Recently many bugs related to combinations of structured output/tool callings with reasoning were reported in vLLM. All these issues can be resolved on vLLM's side but requires code changes inside scheduler that makes them potentially hard to implement and test correctly. Instead Xgrammar itself can handle reasoning, so no need for complex logic on vLLM's side. For tool calling we can simply turn off existing vLLM's reasoning filtering and use instead Xgrammar's one. Structured output (SO) is a bit more different case. SO can be one of the following formats: json object, json schema, structural tag. For structural tag Xgrammar has native reasoning support, for json it currently lacks support. I will do a PR to Xgrammar to implement native reasoning support for json. But as a current workaround we can translate json to structural tag and then reasoning support works as expected. In this PR this approach was implemented for now only for Qwen3.5 as a PoC. ## Test Result ### spec decoding + tool_choice auto/required + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/44006.
server ```bash export VLLM_ENFORCE_STRICT_TOOL_CALLING=1 vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 -tp 2 \\ --language-model-only \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --max-model-len 8192 \\ --enable-log-requests \\ --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ```
client ```bash run() { # $1 = tool_choice, $2 = N local fail=0 for i in $(seq 1 \"$2\"); do code=$(curl -s -o /tmp/r.json -w '%{http_code}' \\ http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ \"model\":\"Qwen/Qwen3.5-35B-A3B\", \"messages\":[{\"role\":\"user\",\"content\":\"What is the weather in Seoul? Think about which tool to use first.\"}], \"tool_choice\":\"'\"$1\"'\",\"max_tokens\":4096,\"temperature\":0.6,\"top_p\":0.95, \"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get current weather information\",\"parameters\":{\"type\":\"object\",\"properties\":{\"location\":{\"type\":\"string\"}},\"required\":[\"location\"]}}}] }') echo \"[$1][$i] HTTP $code\" [ \"$code\" = \"200\" ] || fail=$((fail+1)) done echo \"---- $1: failed $fail/$2\" } run required 30 run auto 30 ```
**Result**: correct; no 500 HTTP responses on server side Even though this bug has already been fixed by https://github.com/vllm-project/vllm/pull/44297 the fix itself is complicated, so probably it is better to process reasoning on xgrammar's side to keep vLLM's scheduler logic simple. This PR alone fixes the same issue but in a more simple way. ### spec decoding + structured output json schema + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/34650
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 -tp 2 \\ --language-model-only \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --max-model-len 8192 \\ --enable-log-requests \\ --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Pick a primary color and say how many letters its name has.\", } ], \"response_format\": { \"type\": \"json_schema\", \"json_schema\": { \"name\": \"color_fact\", \"schema\": { \"type\": \"object\", \"properties\": { \"color\": {\"type\": \"string\"}, \"letter_count\": {\"type\": \"integer\"}, }, \"required\": [\"color\", \"letter_count\"], }, }, }, \"chat_template_kwargs\": {\"enable_thinking\": True}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"color\\\": \\\"Blue\\\",\\n \\\"letter_count\\\": 4\\n}\"` -- correct; no 500 HTTP responses on server side ### spec decoding + structred output json object + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/48228
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 \\ --reasoning-parser qwen3 \\ --enable-auto-tool-choice \\ --tool-call-parser qwen3_coder \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":2}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Return a JSON object with keys name and age for Bob aged 30.\", } ], \"response_format\": {\"type\": \"json_object\"}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"name\\\": \\\"Bob\\\",\\n \\\"age\\\": 30\\n}\"` -- correct; no 500 HTTP responses on server side ### spec decoding + async scheduling + structred output json object + reasoning This case was reported in https://github.com/vllm-project/vllm/issues/43388
server ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --port 8000 \\ --reasoning-parser qwen3 \\ --async-scheduling \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":4}' ```
client ```python #!/usr/bin/env python3 import json import urllib.request from rich.console import Console url = \"http://localhost:8000/v1/chat/completions\" payload = { \"model\": \"Qwen/Qwen3.5-35B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Return a JSON object with keys color and letter_count for the color red.\", } ], \"response_format\": {\"type\": \"json_object\"}, \"chat_template_kwargs\": {\"enable_thinking\": True}, \"max_tokens\": 2048, \"temperature\": 0, } req = urllib.request.Request( url, data=json.dumps(payload).encode(), headers={\"Content-Type\": \"application/json\"}, method=\"POST\", ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) Console().print_json(data=data) ```
**Result:** `\"content\": \"\\n\\n{\\n \\\"color\\\": \\\"red\\\",\\n \\\"letter_count\\\": 3\\n}\"` -- correct; no 500 HTTP responses on server side ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48116", "createdAt": "2026-07-09T10:55:44Z", "updatedAt": "2026-08-12T18:21:30Z", "timestamp": "2026-08-12T18:21:30Z", "metrics": { "reactions": 4, "comments": 8 }, "labels": [ "structured-output", "ci/build", "v1", "tool-calling" ], "author": "arpera", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:3511cd93ad80c22a789f", "signalId": "github:vllm-project/vllm:pull_request:52031", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52031", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Rust Frontend][gRPC] Advertise LoRA capabilities", "text": "## Purpose Advertise the Rust frontend's LoRA capability and capacity through the existing engine ready handshake and gRPC discovery APIs. - Report whether LoRA is enabled and the configured maximum number of active adapters from each Python `EngineCoreProc`. - Decode the capability in the Rust engine-core client and reject internally inconsistent or mixed-capability engine ranks during startup. - Expose `max_loras` through `GetServerInfo` and `supports_lora` through `GetModelInfo`. - Preserve the existing inference and LoRA lifecycle behavior; this PR only reports and validates capability metadata. No overlapping open PR or issue was found after searching for Rust frontend LoRA capability reporting and the `supports_lora` and `max_loras` fields. ## Test Plan - `UV_PYTHON=/usr/bin/python3.12 cargo test --manifest-path rust/Cargo.toml -p vllm-engine-core-client` - `cargo test --manifest-path rust/Cargo.toml -p vllm-server --lib 'grpc::tests::'` - `cargo check --manifest-path rust/Cargo.toml -p vllm-server` - `cargo fmt --manifest-path rust/Cargo.toml --all -- --check` - `ruff check vllm/v1/engine/core.py vllm/v1/engine/__init__.py tests/v1/engine/test_engine_core_client.py rust/src/engine-core-client/src/tests/python_compat.py` - `ruff format --check vllm/v1/engine/core.py vllm/v1/engine/__init__.py tests/v1/engine/test_engine_core_client.py rust/src/engine-core-client/src/tests/python_compat.py` - `git diff --check upstream/main...HEAD` ## Test Result - Engine-core client: 97 passed. - Focused gRPC server suite: 31 passed. - Server compilation, Rust formatting, Python lint/format, and whitespace checks passed. - Model evaluation is not applicable because this change does not affect inference behavior, model output, or accuracy. **AI assistance disclosure:** This PR was authored with AI assistance and reviewed by the submitter. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described. - [x] The test plan is included. - [x] The test results are included. - [x] No documentation update is required for this focused capability-reporting change.
", "url": "https://github.com/vllm-project/vllm/pull/52031", "createdAt": "2026-08-12T18:06:10Z", "updatedAt": "2026-08-12T18:10:50Z", "timestamp": "2026-08-12T18:10:50Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rust" ], "author": "connorcarpenter15", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d22bdc191d62b6d0d0f0", "signalId": "github:vllm-project/vllm:pull_request:50786", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50786", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "docs(serving): note OpenAI client base_url for multi-model gateways", "text": "## Summary The OpenAI-compatible server docs already show the OpenAI Python client via `base_url`. This PR adds a one-line tip that the same client pattern works with OpenAI-compatible multi-model gateways when not running vLLM locally, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior changes. ## Test plan - [ ] Serving docs render - [ ] Local vLLM client example unchanged - [ ] Tip is clearly optional Co-Authored-By: Claude Fable 5 ", "url": "https://github.com/vllm-project/vllm/pull/50786", "createdAt": "2026-08-03T00:42:47Z", "updatedAt": "2026-08-12T18:09:00Z", "timestamp": "2026-08-12T18:09:00Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation" ], "author": "seven7763", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e5dcdd629d9b9237b484", "signalId": "github:vllm-project/vllm:pull_request:51209", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51209", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] IndexCache for DeepSeek-V4 (validated on V4-Flash-0731, including DSpark on to accelerate on top of", "text": "## Summary: This work is built to support Dspark + IndexerCache to achieve best deepseek v4 flash serving performance. ## Purpose Adds DSA IndexCache support to DeepSeek-V4, so C4A layers marked shared reuse the top-k the previous C4A layer left in the shared `topk_indices_buffer` instead of running their own indexer. Same config surface as the DeepSeek-V3.2 feature (`use_index_cache` / `index_topk_freq` / `index_topk_pattern`, see [docs/features/index_cache.md](https://github.com/vllm-project/vllm/blob/main/docs/features/index_cache.md)); addresses #45350. The indexer dominates decode KV traffic on this model at long context: per decode token per C4A layer it reads `(S/4) x 132 B` of its own K cache, while the main sparse MLA reads only `index_topk x 576 B`. Across DeepSeek-V4-Flash-0731's 21 C4A / 20 C128A / 44 SWA caches that is ~65% of decode KV traffic at 32K and ~81% at 128K, and the indexer is TP-replicated so every rank pays it in full. ### Relationship to #49085 #49085 implements the same feature for Hopper and is the earlier work; its authors (@wangyicong52, @zzz3bbb3) got there first on the two non-obvious points — per-PP-rank handling, and keeping skipped indexers' KV-cache specs registered while parking their linear weights on the meta device. Two behavioural differences worth a maintainer decision: 1. This PR gates on `use_index_cache`, matching the documented V3.2 surface. #49085 activates on `index_topk_freq > 1` alone, which would make `--hf-overrides '{\"index_topk_freq\": 4}'` silently enable on V4 while silently doing nothing on V3.2. 2. This PR uses `c4a_idx % index_topk_freq`; #49085 keeps V3.2's `index_skip_topk_offset` (default 2), so its first two C4A layers always compute. Theirs is more faithful to V3.2 and I am happy to switch. ## Design notes Only C4A layers (`compress_ratio == 4`) carry an indexer in V4, so **the frequency and pattern are indexed over the C4A layers, not over all layers**. Applying the V3.2 absolute-`layer_id` formula to DeepSeek-V4-Flash marks *every* C4A layer as shared (they are all even), so nothing populates the buffer and every layer reads stale contents — silently, with no error. A regression test pins this. Three further constraints, each covered by a test: - `topk_indices_buffer` is rank-local, so the first C4A layer **of each pipeline stage** computes its own top-k (derived from the same `get_pp_indices` split `make_layers` uses). - `compress_ratios` can be longer than `num_hidden_layers` (44 entries on V4-Flash, 46 on V4-Flash-0731, for the MTP/draft slots), so selection slices to `num_hidden_layers`. - A pattern is one `F`/`S` character per C4A layer — 21 on V4-Flash-0731, not 43 — and may not start with `S`. Wrong-length patterns raise rather than silently applying a prefix. Skipped layers keep their indexer allocated and simply never run it, so the change is confined to `attention.py` and needs no KV-cache-grouping or weight-loading changes. Memory savings are left to a follow-up. ## Test Plan ```bash pytest tests/models/test_deepseek_v4_index_cache.py -v ``` E2E hardware: single node, 8xH100 80GB, TP=8, upstream vLLM 0.26.0 with this patch, `--kv-cache-dtype fp8_ds_mla`, `deepseek-ai/DeepSeek-V4-Flash-0731`. ```bash # long context / frequency sweep vllm bench latency --model <0731> --tokenizer-mode deepseek_v4 -tp 8 \\ --kv-cache-dtype fp8_ds_mla --max-model-len 33792 \\ --input-len 32768 --output-len 128 --batch-size 2 \\ --hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": N}' # quality: GSM8K 200q 5-shot, MMLU 500q 5-shot, greedy # throughput: short prompt + 512 output tokens at fixed concurrency ``` ## Test Result 14/14 unit tests pass. ### Frequency sensitivity 32768-token input x batch 2, and MMLU 500 questions 5-shot. All arms are the same build with only `index_topk_freq` varying, so `freq=1`/off is a true no-skip control (confirmed by the log line firing only on freq>1 arms): | freq | indexers run | 32K latency | Δ | MMLU | |---|---|---|---|---| | off / 1 | 21/21 | 3.004 s | — | 85.4% | | 2 | 11/21 | 2.890 s | -3.8% | 85.6% | | 3 | 7/21 | 2.842 s | -5.4% | 85.0% | | 4 | 6/21 | 2.831 s | -5.8% | 84.8% | | 6 | 4/21 | 2.808 s | -6.5% | 85.0% | The latency curve saturates quickly — freq=2 captures most of the win, and skipping three times as many indexers (freq=6) only reaches -6.5%. MMLU spans 84.8-85.6%, within the ~1.6pp binomial noise at n=500. vLLM 0.25.1 and the older `DeepSeek-V4-Flash` checkpoint reproduce the freq=2/3 numbers within noise. ### Quality and throughput matrix (GSM8K, 200q 5-shot, best of 3 timed passes) | cell | DSpark | IndexCache | accuracy | throughput | |---|---|---|---|---| | c4 | off | off | 96.0%, 96.5% | 4087, 4525 tok/s | | c2 | off | **on** | 96.0%, 96.0% | 4322, 4081 tok/s | | c1 | **on** | off | 96.0%, 97.0% | 3449, 3742 tok/s | | c3 | **on** | **on** | 97.0% | 3873 tok/s | IndexCache changes neither accuracy nor throughput here. Note this shape is ~91% prefill tokens (180K prefill vs 17K decode) and saturates the GPU, so it is not informative about speculative decoding; the concurrency sweep below is. ### Throughput vs concurrency (short prompt, 512 output tokens) | concurrency | c4 baseline | c1 DSpark | c3 DSpark + IndexCache | |---|---|---|---| | 4 | 560 tok/s | **1419** | 1201 | | 16 | 1670 tok/s | 3289 | **3419** | | 64 | 4910 tok/s | 6975 | **7187** | The two features are complementary and stack: at the concurrencies that matter for serving, DSpark + IndexCache is the fastest configuration. Draft acceptance is 63-69% (mean accepted length 4.2-4.5 of 5) with IndexCache both on and off, so IndexCache does not measurably degrade the draft. ### Long-context correctness A needle-in-a-haystack probe (18,930-token prompt, needle at depth 0.15 / 0.5 / 0.85, greedy) retrieves correctly with IndexCache off, at freq=2, and at freq=3. ### DSpark interaction DeepSeek-V4's DSpark draft builds its decoder layers with `topk_indices_buffer=None` and layer ids >= `num_hidden_layers`, so draft layers resolve to `compress_ratio == 1` and never construct an indexer; the two features are structurally orthogonal. Enabling both is exercised end to end at every concurrency above. ## Notes Not covered, stated rather than implied: no Blackwell run (#49085 hard-rejects non-SM90 for this feature; this PR does not gate, which may be wrong); no PP>1 e2e run (the per-stage rule is unit-tested only); Meanwhile, if B200 number testing needed, I will provide later follow up test in NVFP4. AI Assistance are used for this development. --- ## Full measurement set (AIPerf) Everything below is `aiperf profile` against `vllm serve` on the same node: single 8xH100 80GB, TP=8, upstream vLLM 0.26.0 + this patch, `--kv-cache-dtype fp8_ds_mla`, `deepseek-ai/DeepSeek-V4-Flash-0731`, AIPerf 0.12.0. Output length is pinned with `ignore_eos:true` + `min_tokens` (without it the model stops early and throughput deltas measure differing OSL, not speed), `--request-count` = 16x concurrency, `--warmup-request-count` = concurrency. `ISL x concurrency` is chosen to fit the ~108K-token KV pool at each point; beyond that requests queue and the run becomes a scheduling test rather than a throughput test. ### Controls Two arms exist purely to bound what a real effect must beat: - **`freq=1`** enables IndexCache but skips nothing, so it must land on top of `off`. It does, everywhere. - **`off` re-run** gives the run-to-run noise floor. | control | throughput delta vs `off` | |---|---| | `freq=1`, all five workloads | -0.1% .. +0.1% | | `off` re-run, 3 workloads | -0.1% .. +0.0% | TTFT is the exception: identical configs vary by up to 5.8% on the 4K point, so no TTFT claim below ~6% is meaningful there. ### Output token throughput, delta vs `off` | workload | ISL/OSL | conc | freq=1 | freq=2 | freq=3 | freq=4 | |---|---|---|---|---|---|---| | summarisation (prefill-heavy) | 32K / 16 | 3 | -0.0% | +3.3% | +5.2% | **+5.6%** | | long chat | 32K / 256 | 3 | +0.1% | +3.5% | +4.9% | **+5.2%** | | mid chat | 16K / 256 | 6 | +0.0% | +2.8% | +3.9% | **+4.2%** | | RAG | 7K / 172 | 12 | +0.1% | — | **+3.2%** | — | | short chat | 4K / 256 | 24 | -0.1% | +2.0% | +2.8% | **+3.1%** | Absolute baselines: 13.63, 146.34, 285.82, 466.5, 1057.42 out tok/s respectively. ### Latency, delta vs `off` (freq=4; freq=3 for RAG) | workload | e2e | TTFT | ITL | |---|---|---|---| | 32K / 16 | -5.3% | -5.1% | — | | 32K / 256 | -5.0% | -5.1% | -4.8% | | 16K / 256 | -4.0% | -4.6% | -3.7% | | 7K / 172 | -3.1% | -2.7% | — | | 4K / 256 | -3.0% | -1.9% | -3.2% | ### Accuracy per frequency | freq | GSM8K (200q, 5-shot) | MMLU (500q, 5-shot) | |---|---|---| | off | 96.0% | 85.4% | | 1 (control) | 96.0% | 85.0% | | 2 | 95.5% | 85.4% | | 3 | 96.0% | **86.2%** | | 4 | **96.5%** | 85.0% | No trend: the best GSM8K score is at freq=4 and the best MMLU at freq=3, and the `freq=1` control differs from `off` by 0.4pp on MMLU while being semantically identical to it. All deltas sit inside that noise. **This is a no-harm result, not proof of quality preservation.** Both benchmarks' prompts (~900 and ~500 tokens, i.e. ~225 and ~125 compressed) are below `index_topk=512`, so every layer selects the whole compressed sequence and IndexCache is mathematically a no-op. Long-context quality evidence is limited to the needle probe (retrieved at all three depths with IndexCache off, freq=2 and freq=3); a RULER/LongBench run at 32K is still owed. ### Where the time actually goes A torch-profiler run (eager, prefix caching off, conc 12 at 8K) attributes GPU kernel time as: | bucket | freq=0 | freq=6 | |---|---|---| | TP all-reduce + other | 6176 ms | 5254 ms | | MoE | 1003 ms | 980 ms | | norm/rope | 395 ms | 394 ms | | **indexer** | **91.7 ms (1.2%)** | **30.4 ms (0.4%)** | | compressor | 43 ms | 42 ms | The indexer is 1.2% of kernel time; the single largest kernel is `cross_device_reduce_1stage` (TP all-reduce) at ~69%. Removing 81% of the indexer work saves 61 ms directly, yet wall-clock improves several percent — the rest comes from ranks spending less time waiting inside the all-reduce. This configuration is communication-bound, which is also why the gain saturates between freq=3 and freq=4 (+0.3-0.4pp) and why a faster indexer kernel would not help much. ### Interaction with DSpark speculative decoding Decode-heavy (short prompt, 512 output tokens, `ignore_eos`), output tok/s: | concurrency | baseline | DSpark | DSpark + IndexCache | |---|---|---|---| | 4 | 560 | **1419** | 1201 | | 16 | 1670 | 3289 | **3419** | | 64 | 4910 | 6975 | **7187** | At serving concurrencies the two stack and DSpark + IndexCache is the fastest configuration. Draft acceptance is 63-69% (mean accepted length 4.2-4.5 of 5) with IndexCache both on and off, so IndexCache does not measurably degrade the draft. ### Honest summary of magnitude **+3% to +5.6%** depending on shape and frequency, saturating at freq=3-4, free at inference time, with no measurable accuracy cost in the regimes tested. That is smaller than the 7-8% reported for vllm-ascend in #45350. On this hardware the gap is explained by the profile above (communication-bound, indexer at 1.2%), by the KV pool capping `ISL x concurrency` so the indexer's share cannot grow, and by Hopper reading an FP8 indexer cache at 132 B/token where `use_fp4_indexer_cache` would halve it. ### Measurement caveats - The e2e numbers were produced by the 0.26.0-based build; upstream restructured `attention_impl` in #51430 after 0.26.0, so the rebased branch carries the same gates at the new call sites but has not itself been re-run on GPU. - One RAG data point initially showed +9.8%; its `freq=1` control read +6.6%, which flagged the baseline as un-warmed. A repeat with two independent baseline arms put the true figure at +3.2%. The table above uses the corrected value. - Single node only: no Blackwell run, and no PP>1 run (the per-stage rule is unit-tested only).", "url": "https://github.com/vllm-project/vllm/pull/51209", "createdAt": "2026-08-05T23:47:09Z", "updatedAt": "2026-08-12T18:07:48Z", "timestamp": "2026-08-12T18:07:48Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "deepseek" ], "author": "DiegoCao", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2963aec385cbef91431d", "signalId": "github:vllm-project/vllm:pull_request:51318", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51318", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][DSv4] Make the C128A decode topk row stride capture-stable", "text": "# [Bugfix][DSv4] Make the C128A decode topk row stride capture-stable Fixes the intermittent output corruption on DeepSeek-V4-Flash under concurrent long-context load reported in the #41834 thread (mid-generation `<|begin_of_sentence|>` bursts, multilingual token salad — one request per batch, never the first). Root-cause analysis and A/B evidence: https://github.com/vllm-project/vllm/pull/41834#issuecomment-5209098845. The same fix is open against the GB10 deployment fork as jasl/vllm#41. ## Root cause `_build_c128a_metadata` derives the row stride of the persistent C128A decode buffer from the **batch's** `max_seq_len`: ```python active_topk_width = min(max(next_pow2(cm.max_seq_len // 128), 128), c128a_max) ``` The decode consumers of that layout run **inside FULL cudagraphs**, and cudagraph capture builds attention metadata with `max_seq_len = max_model_len` (`gpu_model_runner.py`, `for_cudagraph_capture` branch). The captured kernels therefore bake the widest stride — 8192 for a 1M-context model — while runtime builds re-lay rows out at whatever the current batch gives (128…4096 in practice). Row 0 lines up at offset 0 under any stride and is always read correctly. Every later decode row is read at the capture-time offset — stale bytes from earlier, differently-strided builds — which the decode kernel treats as global compressed-KV slot ids: the victim request attends over pages it does not own (token salad), or over uninitialized fp8 where ~2/256 random bytes decode to NaN. One NaN turns the whole logits row NaN, and `argmax` over an all-NaN row returns index 0 — token 0 is BOS. This explains the field observations that made the bug hard to pin down: the victim is never the first request in the batch; single-request traffic never reproduces (no row 1); `--enforce-eager` is clean (no baked stride); deployments whose `max_model_len` is close to their tested context are clean (the widths coincide); and the corruption is deterministic per shape, so run-it-twice determinism checks pass. ## Fix Lay decode rows out at `c128a_max_compressed` always. The only cost is wider `-1` padding writes in the build kernel; kernel reads stay bounded by `decode_lens` via `topk_length`, so decode-side work is unchanged. ## Reproduction / verification * `test_c128a_decode_row_addresses_survive_batch_width_changes` reproduces the mechanism directly on one GPU in seconds: build metadata once at `max_seq_len = max_model_len` (what capture does) and once for a small batch, then assert decode row 1's stride and address did not move. On unfixed `main` it fails with *\"stride changed with the batch's max_seq_len (8192 at capture vs 128 at runtime)\"*; on this branch it passes. Verified both ways on GB10 (SM121). * `test_c128a_decode_stride_is_batch_independent` is a no-GPU AST gate on the assignment itself, in the spirit of not asserting on a value the builder could stop computing. Also verified to fail on the unfixed tree. * System-level A/B on 4× GB10 / TP=4 / 1M context / FULL_AND_PIECEWISE / spec-off, 2×300k-token concurrent streams per round, with engine-side NaN detection at the sampling-row gather: **unfixed, 39 NaN events in round 1; fixed, zero events across 8 rounds / 16 streams / ~105k decode steps**, zero leaked specials, zero multi-script output. An in-graph trace (device ops captured into the FULL graphs, so replays record what they compute) additionally confirmed the NaN is born only at compress-ratio-128 layers, only in rows ≥ 1, while the builder's row 1 held valid slot ids and the captured kernel read a stale row. Test commands run (GB10 / SM121, CUDA container): ``` pytest tests/v1/attention/test_dsv4_c128a_capture_stable_stride.py -v # unfixed main: 1 failed (stride), 1 failed (AST gate), behavioral repro fails # this branch: all pass ``` ## Contribution notes * Not a duplicate: no open PR touches the C128A metadata build (searched `c128a`, `sparse_mla deepseek`, `41834 in:body`); jasl/vllm#41 is the same fix in the deployment fork, referenced here intentionally. * AI assistance was used throughout (instrumentation, analysis, and the patch); the change and its verification were reviewed end-to-end by the submitter, who operates the affected production deployment.", "url": "https://github.com/vllm-project/vllm/pull/51318", "createdAt": "2026-08-06T22:15:50Z", "updatedAt": "2026-08-12T18:04:38Z", "timestamp": "2026-08-12T18:04:38Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "tobymao", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:354e4b6bc7a10c063211", "signalId": "github:vllm-project/vllm:pull_request:51545", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51545", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][NVFP4 MoE] Support CuTeDSL for ModelOpt W4A16 and pad non-gated experts", "text": "**Closing since this is not a clean W4A16 as it will still quantize activations, so the comparison between the 2 backends is not apples to apples, and the correctness results below show degradation. I might open a separate PR for the padding fix** ## Purpose Enable the FlashInfer CuTeDSL NVFP4 MoE backend for ModelOpt W4A16 NVFP4 checkpoints on Blackwell, and fix the non-gated CuTeDSL shape handling needed by Nemotron-style MoE layers. This patch enables the existing CuTeDSL API for ModelOpt W4A16 by treating missing activation scales as identity scales at the wrapper boundary. Longer term, CuTeDSL should expose W4A16/no-activation-scale support as an explicit backend mode instead of requiring identity activation-scale tensors. This avoids the Marlin fallback for supported Blackwell W4A16 case; on GB200 with `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4`, auto-selected CuTeDSL improves output throughput by **38.9%** over forced Marlin and reduces mean TPOT by **30.3%**. This PR does three things: 1. Allows the NVFP4 MoE oracle to consider CuTeDSL for ModelOpt W4A16 checkpoints under `--moe-backend auto`. - ModelOpt W4A16 still enters backend selection as static NVFP4 weights with missing activation scales: `weight_key=kNvfp4Static`, `activation_key=None`. - CuTeDSL can run this case through its runtime activation-scale path, but because ModelOpt W4A16 checkpoints do not provide activation scales, the CuTeDSL wrapper materializes identity activation scales. - The oracle therefore uses `kNvfp4Dynamic` only for CuTeDSL's activation support predicate; other backends keep the original `activation_key=None` semantics. 2. Makes CuTeDSL tolerate missing W4A16 activation scales. - W4A16 checkpoints do not carry calibrated activation scales. - CuTeDSL materializes identity activation global scales for this case, matching the dtype/device of the global scale tensors they fold into. 3. Pads non-gated NVFP4 MoE intermediate rows to CuTeDSL's 128-row tile requirement. - The original unpadded intermediate size is preserved in the MoE config. - Padded regions are zero-filled so model outputs are unchanged. With this change, users can run ModelOpt W4A16 NVFP4 MoE with the normal backend selection path: ```bash vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\ --moe-backend auto ``` On supported Blackwell systems, auto selects: ``` Using 'FLASHINFER_CUTEDSL' NvFp4 MoE backend ``` ## Test Plan ### Unit tests Add and run focused cases in: 1. tests/quantization/test_modelopt.py 2. tests/quantization/test_trtllm_nvfp4_hidden_dim_padding.py ### Correctness Full GSM8K using lm-eval local chat completions: ``` python3 -m lm_eval \\ --model local-chat-completions \\ --apply_chat_template \\ --tasks gsm8k.yaml \\ --log_samples \\ --model_args \"model=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4,base_url=http://.../v1/chat/completions,max_length=32768\" \\ --gen_kwargs \"max_tokens=16384,temperature=0,top_p=1\" ``` ### Performance I compared `--moe-backend auto` after this PR, which selects CuTeDSL, against `--moe-backend auto` before this PR, which selects Marlin. Both tests ran with `vllm/vllm-openai:nightly-65b7662d3fcb773afaf751ab29ac6960a0cf011d`, with and without this PR applied. I used nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (W4A16) with GB200 (SM100) for the benchmark. Serve command: ```bash vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\ --moe-backend auto \\ --reasoning-parser nemotron_v3 ``` Bench command: ```bash vllm bench serve \\ --endpoint /v1/completions \\ --dataset-name random \\ --random-input-len 64 \\ --random-output-len 512 \\ --ignore-eos \\ --request-rate inf \\ --max-concurrency 128 \\ --num-prompts 512 ``` ## Test Result ### Unit tests Passed. ### Correctness ``` gsm8k strict-match exact_match: 0.9424 ± 0.0064 gsm8k flexible-extract exact_match: 0.9447 ± 0.0063 ``` ### Performance Backend config | Selected MoE backend | Successful requests | Output tok/s | Mean TPOT -- | -- | -- | -- | -- --moe-backend auto | FLASHINFER_CUTEDSL | 1536 / 1536 | 15235.59 ± 94.15 | 7.45 ± 0.07 ms --moe-backend marlin | MARLIN | 1536 / 1536 | 10972.30 ± 106.03 | 10.69 ± 0.04 ms ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51545", "createdAt": "2026-08-09T08:48:12Z", "updatedAt": "2026-08-12T18:01:08Z", "timestamp": "2026-08-12T18:01:08Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "nvidia", "quantization" ], "author": "fsaady", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:f5f0d59e1d74f9b30344", "signalId": "github:vllm-project/vllm:pull_request:25135", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:25135", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Llamas 3.1 405B fp4 changes upstreaming from 355_wip", "text": "Perf is the same: upstream tp1 ``` ============ Serving Benchmark Result ============ Successful requests: 320 Maximum request concurrency: 64 Benchmark duration (s): 413.07 Total input tokens: 326905 Total generated tokens: 327680 Request throughput (req/s): 0.77 Output token throughput (tok/s): 793.27 Total Token throughput (tok/s): 1584.66 ---------------Time to First Token---------------- Mean TTFT (ms): 5240.27 Median TTFT (ms): 5150.59 P99 TTFT (ms): 13542.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.58 Median TPOT (ms): 76.05 P99 TPOT (ms): 79.35 ---------------Inter-token Latency---------------- Mean ITL (ms): 75.58 Median ITL (ms): 66.27 P99 ITL (ms): 73.04 ----------------End-to-end Latency---------------- Mean E2EL (ms): 82558.95 Median E2EL (ms): 82994.12 P99 E2EL (ms): 89413.31 ================================================== ``` 355_wip tp1 ``` ============ Serving Benchmark Result ============ Successful requests: 320 Maximum request concurrency: 64 Benchmark duration (s): 415.51 Total input tokens: 326905 Total generated tokens: 327680 Request throughput (req/s): 0.77 Output token throughput (tok/s): 788.62 Total Token throughput (tok/s): 1575.37 ---------------Time to First Token---------------- Mean TTFT (ms): 5262.51 Median TTFT (ms): 5156.42 P99 TTFT (ms): 14143.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 76.03 Median TPOT (ms): 76.44 P99 TPOT (ms): 79.86 ---------------Inter-token Latency---------------- Mean ITL (ms): 76.03 Median ITL (ms): 66.64 P99 ITL (ms): 69.13 ----------------End-to-end Latency---------------- Mean E2EL (ms): 83044.74 Median E2EL (ms): 83387.81 P99 E2EL (ms): 90413.53 ================================================== ``` Command: ``` HIP_VISIBLE_DEVICES=7 \\ VLLM_DISABLE_COMPILE_CACHE=1 \\ USE_FASTSAFETENSOR=1 \\ SAFETENSORS_FAST_GPU=1 \\ VLLM_USE_V1=1 \\ AMDGCN_USE_BUFFER_OPS=1 \\ TRITON_HIP_ASYNC_COPY_BYPASS_PERMUTE=1 \\ TRITON_HIP_USE_ASYNC_COPY=1 \\ TRITON_HIP_USE_BLOCK_PINGPONG=1 \\ TRITON_HIP_ASYNC_FAST_SWIZZLE=1 \\ VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_MHA=0 \\ VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 \\ VLLM_USE_AITER_UNIFIED_ATTENTION=0 \\ VLLM_ROCM_USE_TRITON_ROPE=1 \\ VLLM_ROCM_USE_AITER_RMSNORM=1 \\ VLLM_ROCM_USE_AITER_FP4_ASM_GEMM=1 \\ vllm serve /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --host localhost \\ --port 30000 \\ --swap-space 64 \\ --disable-log-requests \\ --dtype auto \\ --max-model-len 8192 \\ --tensor-parallel-size 1 \\ --max-num-seqs 64 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.85 \\ --max-seq-len-to-capture 8192 \\ --no-enable-prefix-caching \\ --async-scheduling \\ --max-num-batched-tokens 8192 \\ --compilation-config='{\"pass_config\":{\"enable_attn_fusion\":true,\"enable_noop\":true,\"enable_fusion\":true},\"cudagraph_mode\":\"FULL\",\"custom_ops\":[\"+rms_norm\",\"+silu_and_mul\",\"+quant_fp8\"],\"splitting_ops\":[]}' ``` # Run the client benchmark ``` vllm bench serve \\ --host localhost \\ --port 30000 \\ --model /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --max-concurrency 64 \\ --num-prompts 320 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --ignore-eos ``` Correctness - shows reasonable answers for command: ``` HIP_VISIBLE_DEVICES=7 \\ VLLM_DISABLE_COMPILE_CACHE=1 \\ USE_FASTSAFETENSOR=1 \\ SAFETENSORS_FAST_GPU=1 \\ VLLM_USE_V1=1 \\ VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 \\ AMDGCN_USE_BUFFER_OPS=1 \\ VLLM_USE_AITER_TRITON_ROPE=1 \\ TRITON_HIP_ASYNC_COPY_BYPASS_PERMUTE=1 \\ TRITON_HIP_USE_ASYNC_COPY=1 \\ TRITON_HIP_USE_BLOCK_PINGPONG=1 \\ TRITON_HIP_ASYNC_FAST_SWIZZLE=1 \\ VLLM_ROCM_USE_AITER=1 \\ VLLM_ROCM_USE_AITER_MHA=0 \\ VLLM_ROCM_USE_AITER_RMSNORM=1 \\ VLLM_TRITON_FP4_GEMM_USE_ASM=1 \\ python /data/vllm-scripts/llm_test.py \\ --model /data/models/Llama-3.1-405B-Instruct-MXFP4-Preview \\ --dataset-path /data/models/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json \\ --batch-size 32 \\ --swap-space 64 \\ --dtype auto \\ --max-model-len 8192 \\ --tensor-parallel-size 1 \\ --max-num-seqs 1024 \\ --kv-cache-dtype fp8 \\ --gpu-memory-utilization 0.92 \\ --max-seq-len-to-capture 8192 \\ --no-enable-prefix-caching \\ --max-num-batched-tokens 8192 ```", "url": "https://github.com/vllm-project/vllm/pull/25135", "createdAt": "2025-09-18T04:18:56Z", "updatedAt": "2026-08-12T17:59:11Z", "timestamp": "2026-08-12T17:59:11Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [ "rocm", "ready", "v1", "llama" ], "author": "maleksan85", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:839224a089d7c14b41b0", "signalId": "github:vllm-project/vllm:pull_request:48922", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48922", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Guard tool call argument JSON parsing in chat message postprocessing", "text": "## Purpose ​ Fixes #47761. ​ `_postprocess_messages` in `vllm/entrypoints/chat_utils.py` calls `json.loads()` on the assistant tool-call `arguments` string that clients replay in conversation history, with no error handling: ​ ```python if content := function.get(\"arguments\"): if not isinstance(content, (dict, list)): parsed = json.loads(content) # raises on malformed input ``` ​ When that string is not valid JSON, the `JSONDecodeError` propagates out of `parse_chat_messages_async` and the request fails with an opaque `400 Unterminated string starting at: line 1 column N`. ​ Because the malformed string lives in **conversation history**, the failure is sticky. Every subsequent turn replays the same message and gets the same 400, so the session is unrecoverable rather than degraded, and the error message gives the user nothing to act on. ​ ### How the malformed string gets there ​ Two independent reports in #47761 reach this line by different routes: ​ | Reporter | Surface | Upstream cause | Payload | | --- | --- | --- | --- | | @rnogueira-tech | Chat Completions, Qwen3.6-35B-A3B-FP8, `qwen3_coder` | `max_tokens` cap hit mid-string; `finish_reason` still reported as `tool_calls`, so the client persisted the partial call | 18,356 chars, fails at char 13 | | @iMMIQ | Responses API, non-Harmony Simple path, DeepSeek-V4-Flash | Generation stopped at EOS with the string still open; `emit_simple_tool_call_done` wrote `state.accumulated_text` into `ResponseFunctionToolCall.arguments` with `status: \"completed\"` and no validation | 54 chars, 297 output tokens, fails at char 8 | ​ In both cases the reported offset is exactly the index of the opening quote of the value that never closes. Both were observed on 0.24.0 and 0.25.1. ​ Ruled out during investigation: ​ - **Not the tool parser.** `qwen3_coder`, `qwen3_xml` and `mimo` all resolve to `Qwen3EngineToolParser`, and `_qwen3_arg_converter` in `vllm/parser/qwen3.py` returns `json.dumps(params, ensure_ascii=False)` on both the full and partial paths, so it cannot emit invalid JSON. - **Not client mis-escaping.** Reconstructing the arguments from the SSE deltas and diffing against what the client replayed gives an identical 18,356 bytes, md5 `d1be3623820564162172c1f5ebff5893`. - **Not a config or version issue.** The reporter tried `hermes` and `qwen3_xml` parsers, patched chat templates, disabling the reasoning parser, and non-streaming mode; all still fail. Raising `max_tokens` to 8192 reproduces at the new cap, and 16384 overflows the 32768 context window. ​ ### Scope of this PR ​ This PR deliberately does not fix either upstream cause. Neither cause is fixable in one place: a `finish_reason` fix does nothing for the EOS case, and Responses-path validation does nothing for the Chat Completions case. `_postprocess_messages` is the only point the two have in common, and it is where vLLM turns data it emitted itself into an unhandled exception. ​ Two follow-ups will be filed separately so this PR stays single-purpose: ​ 1. Report `finish_reason: \"length\"` when a tool call is cut off by the token cap, so clients can discard the partial call instead of persisting it. 2. Validate JSON in `emit_simple_tool_call_done` on the Responses Simple path, and emit the done event that is currently missing for that item. ​ ### Design note ​ On the failure path this keeps the original string and logs a warning, rather than substituting `{}` or attempting a repair. ​ The tradeoff is real and worth stating. Passing through a truncated 18k-char file body preserves useful context. Passing through something like `{\"cmd\": \"mkdir -p /tmp/x && cat > /tmp/x` hands a half-written shell command back to the model as a completed `exec` call, which is arguably worse than an empty object. ​ Raw passthrough is chosen because it is tool-agnostic and non-destructive, and because substituting `{}` silently discards data in the much larger truncation case. This is a one-line change if reviewers prefer `{}` or a best-effort repair; say which and I will push it. ​ ## Test Plan ​ **Unit tests** — `tests/entrypoints/test_chat_utils_tool_call_arguments.py` (new): ​ ``` pytest tests/entrypoints/test_chat_utils_tool_call_arguments.py -q ``` ​ Cases covered: ​ - valid JSON object string is parsed to a dict (unchanged behaviour) - `arguments` already a dict or list is passed through untouched (unchanged behaviour) - absent / empty `arguments` becomes `{}` (unchanged behaviour) - truncated payload in the shape reported by @rnogueira-tech no longer raises - 54-byte `exec` payload reported by @iMMIQ no longer raises - a warning is logged on the failure path ​ **Manual reproducer** — contributed by @iMMIQ in #47761. No GPU work and no generation is involved, because `_postprocess_messages` runs before sampling, so `max_tokens: 1` against any served model is enough: ​ ```bash curl -sS http://:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{ \"model\": \"\", \"max_tokens\": 1, \"messages\": [ {\"role\": \"user\", \"content\": \"hi\"}, {\"role\": \"assistant\", \"content\": null, \"tool_calls\": [ {\"id\": \"c1\", \"type\": \"function\", \"function\": {\"name\": \"foo\", \"arguments\": \"{\\\"cmd\\\": \\\"mkdir -p /tmp/x\"}} ]} ] }' ``` ​ On `main` this returns: ​ ```json {\"error\":{\"message\":\"Unterminated string starting at: line 1 column 9 (char 8)\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}} ``` ​ The equivalent request against `POST /v1/responses` with a `function_call` input item fails the same way. The tool name is irrelevant. ​ ## Test Result ​ REPLACE THIS LINE with the real output of `pytest tests/entrypoints/test_chat_utils_tool_call_arguments.py -q`, pasted inside a code fence. ​ REPLACE THIS LINE with the curl result only if you actually ran it against a server. Otherwise delete this line and leave the reproducer as reported by @iMMIQ in the issue. ​ ## (Optional) Documentation Update ​ None required. Behaviour change is limited to no longer raising on malformed input that vLLM previously accepted into history. ​", "url": "https://github.com/vllm-project/vllm/pull/48922", "createdAt": "2026-07-17T07:15:19Z", "updatedAt": "2026-08-12T17:54:22Z", "timestamp": "2026-08-12T17:54:22Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "frontend" ], "author": "VBS2004", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b3a13d19fbbe5b1a47f6", "signalId": "github:vllm-project/vllm:pull_request:48970", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48970", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Test] e2e hybrid-Mamba prefix-cache corruption regression tests (#43559)", "text": "## Purpose End-to-end output-correctness regression tests for **#43559** — hybrid-Mamba prefix caching silently corrupting output under `mamba_cache_mode=\"align\"` + MTP/EAGLE speculative decoding. These are the e2e tests split out of #48361 (per review) and supersede the test-only **#48198**. Two graded arms, each compared against an `enable_prefix_caching=False` control using deterministic greedy needle recall: - **cold-race** (write-side): concurrent cold prefills fragment a prefill chunk mid Mamba block; the chunk-end state can be cached as the boundary snapshot. - **multi-turn** (read-side): reuse of blocks written during (speculative) decode, where the eagle-lookahead lookup can let the Mamba hit length overrun the attention-verified hit. Parametrized on `Nemotron-Super-120B-A12B-BF16` (TP4) and `Qwen3.6-27B-FP8`. Corruption raises `CorruptionDetected`; geometry / control-quality / cache-liveness problems hard-fail with dedicated exceptions (never `pytest.skip`, since the fork runner reports in-body skips as PASS). An **anti-vacuity liveness gate** hard-fails a \"clean\" run whose prefix cache is not demonstrably live (and now also hard-fails if the ≥3-block liveness probe cannot fit `max_model_len`, rather than silently disabling itself). Registered as a **non-gating optional** Buildkite step. ## What these tests actually guard (scope) These are **in-process** (`LLM`) tests; they exercise the `HybridKVCacheCoordinator` read path and the align-mode scheduler, **not** the Mooncake KV-connector. Concretely: - The **corruption arms** are a regression guard for the already-merged in-process #43559 fixes (**#46384** kv_cache_coordinator eagle-margin gate + **#47782** selective retention). On current `main` these arms are green because those fixes are already in; the RED baseline in the table below is against a **pre-#46384/#47782 tree**. If a future change regresses the in-process fix, the corruption arms go red. - The **cold-race liveness gate** additionally requires **#48668** — it reads `vllm:prefix_cache_hits`, which the offline `LLMEngine.step()` path under-reports to 0 (hits applied on non-output-bearing prefill steps are dropped) without #48668, which would make the gate fire even though the cache is live. **Dependency:** requires **#48668** to pass. It does **not** exercise #48361's Mooncake connector path (`_find_hit_blocks`) — #48361 is covered by its own unit test, not here. (Kept in the #43559 stack for context; merge order relative to #48361 does not matter for these tests.) ## Why not a duplicate Supersedes #48198 (test-only, draft). The production fixes live in #48361 (Mooncake connector) and #48668 (stats); this PR is the in-process e2e regression coverage for #46384/#47782 + the #48668 liveness gate. ## Test Plan / Test Result ```bash pytest -v -s -rA tests/v1/e2e/test_hybrid_mamba_prefix_cache_correctness.py ``` Validated on GB200 (Nemotron-Super-120B-A12B-BF16, TP4, current `main` + #48668; #48361 present but not exercised): | Arm | Result | Engagement | |---|---|---| | cold-race | **1 passed** | `liveness_probe_hits=4224`, no corruption | | multi-turn | **1 passed** | wave-2 hit ratio 0.477 (67584 hits), no corruption | RED baseline (pre-#46384/#47782 main) reproduced corruption on both arms (documented in issue #43559). ## AI assistance disclosure Developed with AI assistance (Claude Code), including GB200 validation and an adversarial self-review that corrected the dependency scope above; every line reviewed by the human submitter.", "url": "https://github.com/vllm-project/vllm/pull/48970", "createdAt": "2026-07-17T17:59:01Z", "updatedAt": "2026-08-12T17:48:40Z", "timestamp": "2026-08-12T17:48:40Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ready", "ci/build", "v1" ], "author": "puririshi98", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:baaa1fabe44364165c98", "signalId": "github:vllm-project/vllm:issue:47761", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:47761", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm 0.23.0 and 0.24.0 - Qwen3.6-35B-A3B-FP8 - Fails generating code- \"400 Unterminated string starting at\"", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-6.12.0-160000.35-default-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA A100-PCIE-40GB GPU 1: NVIDIA A100-PCIE-40GB Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 Stepping: 7 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities L1d cache: 1.3 MiB (40 instances) L1i cache: 1.3 MiB (40 instances) L2 cache: 40 MiB (40 instances) L3 cache: 55 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-9,40-49 NUMA node1 CPU(s): 10-19,50-59 NUMA node2 CPU(s): 20-29,60-69 NUMA node3 CPU(s): 30-39,70-79 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Indirect target selection: Mitigation; Aligned branch/return thunks Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Mitigation; TSX disabled Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.12 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cccl==13.3.3.3.1 [pip3] nvidia-cuda-crt==13.3.33 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvcc==13.2.78 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cuda-tileiras==13.2.78 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.25.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.5.2 [pip3] nvidia-cutlass-dsl-libs-base==4.5.2 [pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2 [pip3] nvidia-ml-py==13.610.43 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] nvidia-nvvm==13.2.78 [pip3] pyzmq==27.1.0 [pip3] tokenspeed-triton==3.7.10.post20260531 [pip3] torch==2.11.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchvision==0.26.0+cu130 [pip3] transformers==5.12.1 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.24.0 vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX 20-29,60-69 2 N/A GPU1 PIX X 20-29,60-69 2 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0 NVIDIA_DRIVER_CAPABILITIES=compute,utility NCCL_DEBUG=INFO VLLM_MARLIN_USE_ATOMIC_ADD=1 VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 NCCL_COMM_ID=127.0.0.1:12345 VLLM_ENABLE_CUDA_COMPATIBILITY=0 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 VLLM_NO_USAGE_STATS=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug When i request to the model to write some code more complex everything looks good at first, but during the code generation an exception occurs \"400 Unterminated string starting at: line 1 column 14 (char 13)\" ```exception [Extension Host] [@continuedev] \u001b[31merror\u001b[39m: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} logToConsole @ log.ts:117 console.ts:139 [Extension Host] [@continuedev] error: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} log @ console.ts:139 localProcessExtensionHost.ts:317 Extension Host localProcessExtensionHost.ts:318 [@continuedev] error: 400 Unterminated string starting at: line 1 column 14 (char 13) {\"context\":\"llm_stream_chat\",\"model\":\"Qwen/Qwen3.6-35B-A3B-FP8\",\"provider\":\"openai\",\"useOpenAIAdapter\":true,\"streamEnabled\":true,\"templateMessages\":false} ``` At first i was thinking it was related to this issue [39055] (https://github.com/vllm-project/vllm/pull/39055) , but after updating to the last version i am still getting the same error. i am also sharing my docker-composer.yaml to make sure that this problem is not a configuration problem: ```yaml vllm-instruct: image: vllm/vllm-openai:v0.24.0 container_name: vllm-Qwen3.6-35B-A3B-FP8 restart: unless-stopped command: [ \"Qwen/Qwen3.6-35B-A3B-FP8\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\", \"--tensor-parallel-size\", \"2\", \"--gpu-memory-utilization\", \"0.90\", \"--max-model-len\", \"32768\", \"--max-num-batched-tokens\", \"32768\", \"--max-num-seqs\", \"64\", \"--enable-prefix-caching\", \"--kv-cache-dtype\", \"fp8\", \"--enable-chunked-prefill\", #\"--disable-custom-all-reduce\", \"--attention-backend\", \"FLASHINFER\", \"--enable-auto-tool-choice\", \"--tool-call-parser\", \"qwen3_coder\", \"--reasoning-parser\", \"qwen3\", ] ports: - \"8001:8000\" env_file: - .env environment: - VLLM_API_KEY=${VLLM_API_KEY} - HF_TOKEN=${HF_TOKEN} - VLLM_NO_USAGE_STATS=1 - HF_HUB_DISABLE_TELEMETRY=1 - NVIDIA_VISIBLE_DEVICES=all - NCCL_DEBUG=INFO - NCCL_COMM_ID=127.0.0.1:12345 - GLOO_SOCKET_IFNAME=lo - VLLM_MARLIN_USE_ATOMIC_ADD=1 ipc: host shm_size: '32gb' volumes: - hf_cache:/root/.cache/huggingface - vllm_cache:/root/.cache/vllm deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] security_opt: - no-new-privileges:true healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8000/health\"] interval: 30s timeout: 10s retries: 3 start_period: 240s (...) ``` Can you please help me? ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47761", "createdAt": "2026-07-06T16:42:07Z", "updatedAt": "2026-08-12T17:46:25Z", "timestamp": "2026-08-12T17:46:25Z", "metrics": { "reactions": 0, "comments": 21 }, "labels": [ "bug" ], "author": "rnogueira-tech", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1377157b41dfd9db1645", "signalId": "github:vllm-project/vllm:pull_request:51980", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51980", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][MoE] Update AITER MXFP4 W4A16 tests to the renamed expert_mask", "text": "## Purpose Fixes two errors in `tests/kernels/moe/test_rocm_aiter_moe.py` on gfx950: ``` TypeError: rocm_aiter_fused_experts() got an unexpected keyword argument 'expert_map' ``` #49758 renamed that keyword from `expert_map` to `expert_mask` and updated the production callers (`AiterExperts.apply`, quark), but not these two test call sites. This renames them; both pass `None`, so nothing else changes. The other `expert_mask=` uses in the same file call `torch.ops.vllm.rocm_aiter_fused_moe`, which has always used that name and is unaffected. Both tests are behind `on_gfx950()`, so the CI run on #49758 skipped them. ## Test Plan MI355 (gfx950), ROCm: ``` pytest -v -s tests/kernels/moe/test_rocm_aiter_moe.py ``` ## Test Result Before: `test_aiter_fused_moe_mi350_mxfp4_w4a16_accuracy` and `test_aiter_fused_moe_mi350_mxfp4_w4a16_determinism` both error out with the `TypeError` above. After: both pass; the whole file is **31 passed**. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51980", "createdAt": "2026-08-12T11:08:29Z", "updatedAt": "2026-08-12T17:45:46Z", "timestamp": "2026-08-12T17:45:46Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "rocm", "ready" ], "author": "stefankoncarevic", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:0b5b03a44031501975b2", "signalId": "github:vllm-project/vllm:issue:52025", "event": "changed", "observedAt": "2026-08-13T13:48:00.446149Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52025", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Rust frontend renders caller-supplied chat templates with no evaluation budget: a 116-byte request body costs 55 CPU-seconds", "text": "### Your current environment vllm/collect_env.py output is not included: this reproduces entirely against the Rust workspace under rust/, driven by cargo test and the in-tree mock engine. The vLLM Python package is not installed on the repro machine and there is no GPU, so collect_env.py would report N/A for essentially every field it exists to collect. Nothing in the affected code path touches torch, CUDA, or the Python runtime. Happy to provide a full collect_env.py dump from a GPU install if that is needed for triage.
The output of python collect_env.py ```text Your output of `python collect_env.py` here ```
### 🐛 Describe the bug ### Summary The Rust frontend renders a caller-supplied Jinja `chat_template` from the request body with **no evaluation budget of any kind**. Cost is `O(N^depth)` with both terms written into the template by the caller, so a 116-byte body can occupy a request-runtime worker thread for 55 seconds. The operator cannot turn the feature off: `--trust-request-chat-template=false`, the setting Python vLLM defaults to, is a hard startup error in the Rust frontend. This is not \"templates can be expensive\". vLLM already bounds this class of cost elsewhere, and minijinja already bounds the same primitive — **and both bounds are walked around by the same omission**. That is the substance of the report. **Component:** `rust/src/chat/src/renderer/hf/template.rs`, reached from `POST /v1/chat/completions`. Read at commit `5c4fe4b`. **Severity (our reading):** remote, unauthenticated, repeatable denial of service against the request-runtime pool; ### The defect `poc.rs` below reproduces all of this on a plain release build of minijinja with vLLM's exact feature set. **Cost is `O(N^depth)`, and depth costs about 40 bytes a level:** ``` depth range bytes iterations seconds 2 300 78 90,000 0.005 2 1000 80 1,000,000 0.057 3 200 116 8,000,000 0.473 3 400 116 64,000,000 3.659 3 999 116 997,002,999 55.6 4 100 154 100,000,000 5.9 ``` The 116-byte template is: in jinja: ```jinja {% for x0 in range(999) %}{% for x1 in range(999) %}{% for x2 in range(999) %}{% endfor %}{% endfor %}{% endfor %}ok ``` Every `range()` in it is individually legal and individually small. **Guard 1 — minijinja bounds a single `range()`, and nesting walks around it.** ``` range( 100,000) accepted range( 1,000,000) REFUSED -- invalid operation: range has too many elements ``` The engine's authors did consider a bound on this exact primitive. It is per-range, so composing three legal ranges multiplies past it. minijinja also ships a `fuel` feature for bounding total evaluation work; `grep -rn set_fuel` over `rust/` returns nothing. **Guard 2 — vLLM isolates the runtime, and does not bound the task on it.** `rust/src/server/src/runtime.rs:14-16` introduces a dedicated request runtime with the comment: > *\"routes so CPU-heavy request preparation does not monopolize the HTTP runtime's worker > queue\"* `/v1/chat/completions` is in `OFFLOADED_PATHS` (`middleware/offload.rs:25-35`) and is spawned onto it (`offload.rs:82`). **This works, and it bounds the blast radius**: `/health` and `/metrics` stay responsive, so this is not a whole-server outage and we do not claim one. But that runtime is a bounded pool — `worker_threads = min(available_parallelism, 32)` (`runtime.rs:9,21`) — and the render is a plain synchronous call on it with no timeout: ```rust // rust/src/chat/src/renderer/hf/template.rs:124-127 pub fn apply(&self, ctx: TemplateContext<'_>) -> Result { let tmpl = self.env.get_template(\"chat\")?; tmpl.render(ctx).map_err(TemplateError::from) } ``` The same codebase uses `tokio::task::spawn_blocking` for its other CPU-heavy request preparation — image, audio and video preprocessing (`chat/src/multimodal/{image.rs:67, audio.rs:62, video.rs:85}`). The pattern is applied deliberately elsewhere; offloading alone would not fix this anyway without a cost bound. **So: at most 32 concurrent hostile requests occupy every worker and the chat-completions plane stops serving, while the health check stays green.** A monitor watching `/health` sees nothing. **No effective ceiling on template size.** `DEFAULT_JSON_BODY_LIMIT_BYTES = 32 * 1024 * 1024` (`rust/src/server/src/routes.rs:33`). 116 bytes already buys 55 seconds. ### Why this is the same class you already bound elsewhere The Security Guide documents decode limits for exactly this failure mode: | control | default | |---|---| | `VLLM_MAX_IMAGE_PIXELS` | ~179M pixels | | `VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` | 25 | | `VLLM_MAX_AUDIO_DECODE_DURATION_S` | 600 | with the note that setting any of them to `0` *\"removes the protection against resource-exhaustion attacks\"*. Multimodal decode is bounded; chat-template evaluation is not. ### Reproduction Needs no GPU and no model weights. ```toml # Cargo.toml — feature set copied from vLLM's rust/Cargo.toml [dependencies] # 2.18.0 is what vLLM's rust/Cargo.lock pins minijinja = { version = \"=2.18.0\", features = [ \"unstable_machinery\", \"json\", \"builtins\", \"loader\", \"loop_controls\", \"preserve_order\", ] } ``` ```rust use minijinja::{context, Environment}; use std::time::Instant; fn timed(tmpl: &str) -> (f64, Result) { let env = Environment::new(); let t = Instant::now(); let r = env.render_str(tmpl, context! {}); let dt = t.elapsed().as_secs_f64(); (dt, r.map(|s| s.len()).map_err(|e| e.to_string().chars().take(44).collect())) } fn nest(depth: usize, n: u64) -> String { let mut t = String::new(); for i in 0..depth { t.push_str(&format!(\"{{% for x{i} in range({n}) %}}\")); } for _ in 0..depth { t.push_str(\"{% endfor %}\"); } t.push_str(\"ok\"); t } fn main() { println!(\"== a single range() is bounded ==\"); for n in [100_000u64, 1_000_000] { let (_, r) = timed(&format!(\"{{% for i in range({n}) %}}{{% endfor %}}ok\")); println!(\" range({n:>9}) {}\", if r.is_ok() { \"accepted\".into() } else { format!(\"REFUSED -- {}\", r.unwrap_err()) }); } println!(\"\\n== nesting walks around it: O(N^depth) ==\"); println!(\" {:>5} {:>7} {:>7} {:>18} {:>10}\", \"depth\", \"range\", \"bytes\", \"iterations\", \"seconds\"); for (d, n) in [(2usize, 300u64), (2, 1000), (3, 200), (3, 400), (3, 999), (4, 100)] { let t = nest(d, n); let (dt, _) = timed(&t); println!(\" {d:>5} {n:>7} {:>7} {:>18} {dt:>10.3}\", t.len(), (n as u128).pow(d as u32)); } } ``` ### The operator cannot opt out Python vLLM gates this behind `trust_request_chat_template`, default **off**: ```python # vllm/entrypoints/chat_utils.py trust_request_chat_template: bool = False # \"Whether to trust the chat template provided in the request. If False, the server # will always use the chat template specified by --chat-template or the ones from # tokenizer.\" ``` The Rust frontend declares the same flag under `#[command(next_help_heading = \"Options not implemented in Rust frontend yet\")]` (`rust/src/cmd/src/cli/unsupported.rs`), where `Unsupported::from_str` always returns `Err`: ```console $ vllm-rs serve --no-trust-request-chat-template error: invalid value 'true' for '--trust-request-chat-template [<...>]': argument is not implemented in Rust frontend yet $ vllm-rs serve --trust-request-chat-template=false error: invalid value 'false' for '--trust-request-chat-template [<...>]': argument is not implemented in Rust frontend yet ``` Both spellings of the safe setting are hard startup errors, while the request field is accepted and reaches the renderer (`routes/openai/chat_completions/types.rs`, `convert.rs:160`). We recognise the flag is on a declared to-do list, and that \"implement the missing flag\" on its own would be a feature request. It is here because it is the reason the defect above cannot currently be mitigated by configuration — not as the defect. ### What we ruled out We expected a stronger primitive and did not find one. `build_environment` (`rust/src/chat/src/renderer/hf/template.rs:29-40`) is, in full: ```rust let mut env = Environment::new(); env.set_trim_blocks(true); env.set_lstrip_blocks(true); env.add_template_owned(\"chat\".to_owned(), template)?; env.set_unknown_method_callback(minijinja_contrib::pycompat::unknown_method_callback); env.add_filter(\"tojson\", hf_tojson_filter); ``` **No loader is registered.** We verified against minijinja directly that *with* a loader `{% include 'hostname' %}` renders that file's contents, and that without one `include`, `extends` and `import` all fail. vLLM enables minijinja's `loader` feature but never calls `set_loader`, so a request-supplied template **cannot read files**. The render context (`TemplateContext`, `template.rs:44-56`) carries messages, tools, documents, special tokens and template kwargs — nothing an unauthenticated caller did not already supply or know. Stated because it bounds the report, and so a reader need not wonder whether we looked. ### Suggested remediation 1. **Bound template evaluation.** Enable minijinja's `fuel`, or wrap `apply()` in a wall-clock budget. This closes the defect regardless of who supplied the template, and it also covers the **model-supplied** path, which no gate protects. 2. **Implement `trust_request_chat_template`, defaulting to `false`,** matching Python vLLM. That removes the request-supplied exposure and lets an operator who deliberately enables it keep the protection from (1). 3. Until either lands, consider rejecting `chat_template` in the request body rather than honouring a field the operator has no way to control. 4. Independently, `spawn_blocking` the render as the multimodal preprocessors already do. Hygiene, not a fix. ### Environment vLLM Rust frontend at `5c4fe4b`. **minijinja 2.18.0**, the version pinned in `rust/Cargo.lock`. rustc stable, release build, Linux x86-64, 32 cores, machine otherwise idle. Three repetitions of the 55.6 s row gave 55.382 / 55.436 / 55.397 — the figure is stable to about ±0.05 s. An earlier draft of this report quoted **86.7 s** for that row. That measurement was taken while the machine was running other builds and fuzzing campaigns, and overstated the cost by about 36%. It is corrected here rather than quietly restated. Absolute seconds will differ by machine in any case; the growth is the claim. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52025", "createdAt": "2026-08-12T17:11:53Z", "updatedAt": "2026-08-12T17:45:18Z", "timestamp": "2026-08-12T17:45:18Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "Yunzez", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3465eda6733e0816172f", "signalId": "github:vllm-project/vllm:pull_request:51415", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51415", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual act quant fusion llama", "text": "Starts the `ActivationQuantFusionPass` manual-fusion migration (RFC #43224, specific tracker https://github.com/vllm-project/vllm/issues/43501) on the producer side of the `QuantizedActivation` contract (#44260). Adds `maybe_fused_act_quant`: given an activation and the linear it feeds, it emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kernel when the linear advertises a consumable `input_quant_key`, and falls back to the plain activation otherwise. Llama's MLP forward calls it for `down_proj` — the model-code change is one line. When manual fusion fires the `silu_and_mul` pattern is already consumed, so the compiler `ActivationQuantFusionPass` finds nothing and the two never double-fuse. Only `(SiluAndMul, kFp8StaticTensorSym)` is registered for now; other schemes/activations are one table row each. Not a duplicate: builds on the landed contract (#44260) and is the activation-quant counterpart to the AR+RMSNorm manual fusion in #45855; no open PR covers the producer side. ## Test Smoke tested on `RedHatAI/Llama-3.2-1B-Instruct-FP8` (default compiled + CUDA-graph path): all 16 `mlp.down_proj` layers take the fused branch (`input_quant_key == kFp8StaticTensorSym`) and generations are coherent — the `QuantizedActivation` flows through the compiled `down_proj.forward` into `apply_weights` with no graph break. TP2 coherence and an A/B vs. the compiler pass are follow-ups. AI assistance (Claude) was used; all changed lines are human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/51415", "createdAt": "2026-08-07T15:44:04Z", "updatedAt": "2026-08-13T16:17:14Z", "timestamp": "2026-08-13T16:17:14Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "llama", "quantization" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6657c9a9b53a71678498", "signalId": "github:vllm-project/vllm:pull_request:52159", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Hardware][Intel] Disable compiled grouped_topk on XPU", "text": "## Purpose Unconditional `@torch.compile` on `grouped_topk` emits an XPU Inductor kernel that gathers `scores[topk_ids]` with **out-of-range expert ids**: ``` Assertion `index out of bounds: 0 <= tmp0 < 128` failed. ``` This fires at engine-init warmup for any path that uses the Python grouped-topk router on XPU (NemotronH / Nemotron-3.5 Lightning MTP draft MoE, and any fallback that is not fused inside `XPUExpertsWNA16`). The fused base WNA16 MoE kernel does **not** execute this Python router, which is why the backbone can load while the draft / compiled router dies. This is the same class of compiled-reduction bug vLLM already documents for other backends (see the PowerPC argmax note in `topk_topp_sampler.py`). **Fix:** decorate `grouped_topk` with `torch.compiler.disable` **only when** `current_platform.is_xpu()`. CUDA keeps `@torch.compile` + the existing fused / fallback path. `torch.compiler.disable` is a graph break at the router only; the rest of the model can stay compiled / XPU-graph-captured. This is complementary to https://github.com/vllm-project/vllm/pull/39145 (fused XPU `grouped_topk` registration). That PR is the right long-term kernel path. This PR unblocks the eager/native router that still runs when that kernel is not selected (draft MoE, fused-flag off, or registration not yet landed). It does **not** add a second routing algorithm, expert-id clamps, or a CUDA-breaking top-level XPU import. Related kernels PR: https://github.com/vllm-project/vllm-xpu-kernels/pull/524 — zero the Xe2 grouped-GEMM scheduler counter (`at::empty` → `at::zeros`). Without that, graph-mode MoE can still be non-deterministic even if this router is correct. ## Test Plan ```bash pytest tests/kernels/moe/test_grouped_topk_xpu.py ``` The new file is skipped on non-XPU. On XPU it: - calls `grouped_topk` for softmax/sigmoid × bias/no-bias with `num_experts=128`, `num_expert_group=8`, `topk=8`, `topk_group=4`; - asserts every id is in `[0, num_experts)` and weights are finite; - wraps the same call in `torch.compile` and asserts it does **not** device-assert. Existing CUDA `tests/kernels/moe/test_grouped_topk.py` is unchanged. Manual serve repro (Intel XPU, NemotronH / Nemotron-3.5 Lightning with a draft that hits Python `grouped_topk`): ``` vllm serve --quantization gptq ... ``` Pre-fix: EngineCore dies during warmup with `0 <= tmp0 < 128`. Post-fix: router no longer asserts. ## Test Result On Intel Arc Pro B70: - Pre-fix: compiled `grouped_topk` device-asserts `0 <= tmp0 < 128` at engine-init warmup for NemotronH MTP draft MoE. - Post-fix: same serve path no longer asserts in the router. `torch.compile(grouped_topk)` on-device returns in-range ids. - CUDA path is a no-op (decorator still `@torch.compile`). - Full-eager + this router is deterministic at temp-0. Compiled/graph mode can still diverge in *other* XPU compiled kernels (sampler / GEMM without the kernels `at::zeros` fix). This PR does **not** claim “XPU is now deterministic.” No MTP prenorm, V1-bypass, W4A8, converters, or expert-id clamps in this PR.", "url": "https://github.com/vllm-project/vllm/pull/52159", "createdAt": "2026-08-13T11:20:36Z", "updatedAt": "2026-08-13T16:16:51Z", "timestamp": "2026-08-13T16:16:51Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "SergiioB", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:cc8040d2597591a8a7dd", "signalId": "github:vllm-project/vllm:pull_request:52091", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Auto-ping Cohere on related issues", "text": "## Purpose As discussed offline ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52091", "createdAt": "2026-08-13T03:37:59Z", "updatedAt": "2026-08-13T16:16:46Z", "timestamp": "2026-08-13T16:16:46Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "ci/build" ], "author": "DarkLight1337", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4d2df6a2e3bba028f7d7", "signalId": "github:vllm-project/vllm:pull_request:52189", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52189", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Fail fast when MLA head dimensions are not supported by the CPU decode kernel", "text": "## Purpose Partial implementation of **option B** from #52008: fail fast with a clear error when an MLA model's head dimensions are not supported by the CPU MLA decode kernel, instead of crashing deep inside kernel dispatch after a confusing chain of contradictory mamba / prefix-caching validation errors. The CPU MLA decode kernel (`csrc/cpu/mla_decode.cpp`) only compiles for the DeepSeek-V2/V3 cache layout (`head_dim=576, v_head_dim=512, block_size=16`). Models with different MLA head dimensions — e.g. the `inference-optimization/Kimi-K3-0.40B` tiny variant (`head_dim=160, v_head_dim=64`) — cannot run on the CPU backend at all, but previously only failed deep in kernel dispatch with an unrelated-looking error. Now `CpuPlatform.check_and_update_config` validates the dimensions at startup and raises: ``` ValueError: The CPU MLA decode kernel only supports head_dim=576, v_head_dim=512 (block_size=16); got head_dim=160, v_head_dim=64. Model cannot run on the CPU backend. ``` DeepSeek-V2/V3 standard dimensions still pass the guard and keep running; the existing MLA handling (forcing chunked prefill and prefix caching off) is unchanged. This PR does **not** implement option A (CPU MLA prefix caching / chunked prefill) or option C (honoring `--hf-overrides {\"use_mla\": false}`). This is deliberately more precise than the architecture-level fail-fast suggested in #52008: only models whose MLA head dimensions the CPU decode kernel cannot run are rejected, so DeepSeek-V2/V3 with standard dimensions remain supported on CPU. Complementary to #52045 (which works around the contradictory mamba validation so supported configs can start); #52045 explicitly lists option B as out of scope, so there is no overlap. ## Test Plan ``` .venv/bin/python -m pytest tests/config/test_cpu_mla_fail_fast.py .venv/bin/python -m pytest tests/config/test_multimodal_config.py .venv/bin/python -m ruff check vllm/platforms/cpu.py tests/config/test_cpu_mla_fail_fast.py .venv/bin/python -m ruff format --check vllm/platforms/cpu.py tests/config/test_cpu_mla_fail_fast.py ``` Local synthetic config only (EngineArgs + stubbed `hf_text_config`). No weight download, no GPU, no serve/generate. ## Test Result ``` 3 passed tests/config/test_cpu_mla_fail_fast.py 42 passed tests/config/test_multimodal_config.py (regression) All checks passed! (ruff check) 2 files already formatted (ruff format) ``` Manual verification with the real Kimi-K3-0.40B config (cached locally, `use_mla=True, kv_lora_rank=128, qk_rope_head_dim=32, v_head_dim=64`): `CpuPlatform.check_and_update_config` raises the fail-fast error above at startup; the DeepSeek-V3 layout (512/64/512) passes the guard and keeps the existing behavior (chunked prefill and prefix caching forced off). ## AI Assistance This PR was prepared with AI assistance (deepseek). The human submitter reviewed every changed line and realize the detail. **Why this is not a duplicate:** searched open PRs referencing #52008 before opening. #52045 exists but implements a different fix (mamba cache-mode validation workaround, so supported configs can start) and explicitly lists option B — a hard \"architecture not supported on CPU\" fail-fast — as out of scope. This PR is the complementary option B. **Model evaluation:** N/A. This change is a startup-time configuration guard; it does not affect inference output, accuracy, or serving behavior. ## Docs None. This does not make `Kimi-K3-0.40B` a supported CPU serving path — it turns a deep confusing crash into a clear startup error.", "url": "https://github.com/vllm-project/vllm/pull/52189", "createdAt": "2026-08-13T16:14:27Z", "updatedAt": "2026-08-13T16:15:10Z", "timestamp": "2026-08-13T16:15:10Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "cpu" ], "author": "kunkunblueberry", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f7490ad906b88015c534", "signalId": "github:vllm-project/vllm:pull_request:52017", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52017", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X causal paged attention backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency and shared lazy-import integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) causal paged-attention backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing attention backend interface. It does not modify generic model-runner behavior or introduce a new attention abstraction. Supported paths include: - Causal paged MHA, MQA, and GQA with BF16 model dtype. - BF16 and FP8 E4M3 KV caches. - Prefill, decode, and mixed batches. - Speculative verification, sliding-window attention, and attention sinks. - CUDA graph capture and replay. MLA, non-causal attention, ALiBi, logits soft capping, and context parallelism are unsupported. The documentation covers explicit `--attention-backend B12X_ATTN` selection, supported cache formats and head sizes, and unsupported attention modes. This is the causal-attention component split from #51696, which it supersedes rather than duplicates. Related open work targets different backend boundaries and does not provide this standalone optional-package attention integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused backend and reference-correctness coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/v1/attention/test_b12x_attn.py \\ tests/v1/attention/test_attention_backends.py \\ -k b12x -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `Qwen/Qwen3-8B` on an RTX PRO 6000 Blackwell Max-Q GPU with BF16 model weights, TP1, an 8192-token input, and a 512-token output. Hold linear execution constant and compare B12X with FlashInfer using both BF16 and FP8 E4M3 KV caches. Use three warmup and ten measured requests per result. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 41 passed, 124 deselected in 38.59s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | BF16-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 88.5677 | 89.6471 | +1.22% | | FP8-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 91.0632 | 92.3668 | +1.43% | The attention measurements exclude prefill and use 5,110 post-first-token outputs across ten requests per result. All 40 measured requests completed successfully. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52017", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:13:42Z", "timestamp": "2026-08-13T16:13:42Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:2cf3be17daab679c4fd2", "signalId": "github:vllm-project/vllm:pull_request:52018", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52018", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X FP4 MoE backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency, shared lazy imports, packed-storage reuse, and warmup integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) FP4 MoE backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing fused-MoE backend interfaces. It does not introduce a new MoE abstraction. Supported paths include: - Native NVFP4 and MXFP4 W4A4. - W4A16 and supported dynamic W4A8 activation modes. - MXFP4 defaults to MXFP8 activations where supported; NVFP4 retains its checkpoint activation format. - `VLLM_B12X_MOE_FP4_FORCE_A16=1` forces both FP4 formats onto the BF16 activation path. - Tensor parallelism, warmup, reload-stable packed storage, and CUDA graph replay. Expert parallelism, expert maps, EXL3, and NF3 are unsupported. B12X 1.2.4 owns and registers the custom operator used by the backend. Warmup supplies the complete set of serving sizes available to vLLM. B12X deduplicates sizes that resolve to identical internal policy choices; vLLM does not duplicate or guess those package-internal policies. The documentation covers explicit `--moe-backend b12x` selection, supported activation modes, the A16 override, and unsupported expert configurations. This is the MoE component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 do not provide this standalone optional-package MoE integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the B12X MoE backend-selection, correctness, warmup, storage-reuse, and CUDA graph coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/kernels/moe/test_b12x_moe.py -q .venv/bin/python -m pytest \\ tests/model_executor/test_b12x_warmup.py \\ tests/quantization/test_auto_round.py::test_inc_mxfp4_moe_method_registers_weights_and_builds_kernel \\ -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `nvidia/MiniMax-M2.7-NVFP4` on an RTX PRO 6000 Blackwell Max-Q GPU, comparing B12X with FlashInfer CUTLASS MoE at TP2. Use five warmup and ten measured requests. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 54 MoE backend tests passed in 23.63s 2 warmup and weight-lifecycle tests passed in 1.09s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | NVFP4 MoE | MiniMax-M2.7-NVFP4 | 2 | FlashInfer CUTLASS MoE | 132.1558 | 137.8276 | +4.29% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52018", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:11:53Z", "timestamp": "2026-08-13T16:11:53Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build", "quantization" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ac98afacffd44639b137", "signalId": "github:vllm-project/vllm:pull_request:52188", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52188", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec decode] Support Kimi-K3 DCP with DSpark", "text": "## Purpose This PR adds support for running Kimi-K3 decode context parallel with DSpark with FlashinferMLA and Tokenspeed as target causal attention backend and Tokenspeed as the draft non-causal backend. ## Test Plan Kimi K3 GSM8k with the different backend combination. ## Test Result Default (no backend specified): ``` vllm serve moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 \\ -dcp 8 \\ --load-format fastsafetensors \\ --no-enable-flashinfer-autotune \\ --trust-remote-code \\ --language-model-only \\ --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\ --kv-cache-dtype fp8 \\ --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"rejection_sample_method\":\"standard\"}' |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9606|± |0.0054| | | |strict-match | 5|exact_match|↑ |0.9606|± |0.0054| ``` FlashinferMLA target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9613|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9613|± |0.0053| ``` Tokenspeed target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9621|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9621|± |0.0053| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52188", "createdAt": "2026-08-13T16:10:03Z", "updatedAt": "2026-08-13T16:10:43Z", "timestamp": "2026-08-13T16:10:43Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia", "mrv2", "kimi", "k3" ], "author": "wzhao18", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:cda1d3c1820175bd279b", "signalId": "github:vllm-project/vllm:pull_request:49585", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49585", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] Added Build Connector Worker Meta for EC Connector", "text": "## Purpose **Why this is still needed after #38390.** The PR implemented the V2 model runner EC Connector, but EC still has no worker -> scheduler metadata channel. KV connectors have a complete one: `build_connector_worker_meta()` -> `KVConnectorOutput.kv_connector_worker_meta` -> `KVOutputAggregator` -> scheduler-side KV connector. EC had none of those three pieces, so a worker-side`ECConnector` has no way to report per-step state — e.g. \"this `mm_hash` is now durably stored\" — back to its scheduler-side EC connector. That blocks the CPU-offload based on CUDA events and NIXL P2P EC connectors, which need exactly this report (We are retiring the StepTracker in the CPU EC Connector, because using CUDA events is the right way to handle \"copy done\" notifications, instead of waiting until the batch is done). This PR adds that channel, and fixes the exit paths where the worker's `ec_connector_output` was computed and then dropped before reaching the scheduler. ### The channel - `ec_connector/base.py`: `ECConnectorWorkerMetadata` ABC with an abstract `aggregate()`, plus `ECConnectorBase.build_connector_worker_meta()`. - `v1/outputs.py`: `ec_connector_worker_meta` field on `ECConnectorOutput`. - `ec_connector/utils.py`: `ECOutputAggregator`, mirroring `KVOutputAggregator` - merges the EC output of the rank that ran the connector onto the rank whose `ModelRunnerOutput` reaches the scheduler. - `engine/core.py`, `executor/abstract.py`, `executor/multiproc_executor.py`: wiring so the aggregator runs when an EC connector is configured. - Both runners (MRv1, MRv2) populate the field via `build_connector_worker_meta()`. Default behavior is unchanged for existing connectors: nothing overrides `build_connector_worker_meta()` yet, so it returns `None`. ## Test Plan ```bash # Existing EC connector unit tests (unaffected areas) .venv/bin/python -m pytest tests/v1/ec_connector/unit/test_metadata.py \\ tests/v1/ec_connector/unit/test_ec_example_connector.py \\ tests/v1/ec_connector/unit/cpu/test_connector.py -v ## Test Result ``` tests/v1/ec_connector/unit/test_metadata.py ..................... PASSED tests/v1/ec_connector/unit/test_ec_example_connector.py ......... PASSED tests/v1/ec_connector/unit/cpu/test_connector.py ................ PASSED 36 passed, 2 warnings pre-commit: ruff check / ruff format / mypy / SPDX headers / etc. — all Passed ```", "url": "https://github.com/vllm-project/vllm/pull/49585", "createdAt": "2026-07-23T12:33:50Z", "updatedAt": "2026-08-13T16:10:36Z", "timestamp": "2026-08-13T16:10:36Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "ready", "ci/build", "v1", "kv-connector", "mrv2" ], "author": "omerpaz95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:36e3ec4f75ce9134010f", "signalId": "github:vllm-project/vllm:pull_request:50932", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "buffer size insuffient Dspark sd for FlashInfer MNNVL allreduce", "text": "fixes [#50877](https://github.com/vllm-project/vllm/issues/50877) ## Purpose FlashInferAllReduce.should_use_fi_ar gates on: self.max_num_tokens = max_workspace_size // (hidden_dim * element_size) max_workspace_size is the size of the whole MNNVL allocation (2 MB for TP8). But the MNNVL backend is Lamport-based and rotates through NUM_LAMPORT_BUFFERS=3 buffers, so only ~1/3 of the budget backs any single all-reduce: budget 2,097,152 bytes → gate says 146 tokens per-buffer 698,368 bytes → real capacity is 48 tokens Any tensor between 49–146 tokens passes the gate and then aborts inside the kernel. The drafter produces B × (1 + num_speculative_tokens) tokens per step — e.g. 14 requests × 4 = 56 tokens — landing exactly in this window. Without a drafter, decode batches produce B tokens and stay below 48. The same defect exists in fused_allreduce_gemma_rms_norm._can_use_flashinfer, which its docstring notes \"mirrors FlashInferAllReduce\". ## Fix After the existing budget bound (kept as a cheap pre-filter), call the workspace's own is_buffer_size_sufficient(tp_size, num_tokens, hidden_dim, dtype) method rather than reimplementing the Lamport arithmetic. This is on the abstract base class (flashinfer-python==0.6.15.post1, the pinned version), correct for both mnnvl and trtllm, and version-proof. ## What this does NOT fix AllReduceFusionPass has the same too-permissive bound but self.max_token_num there also drives the workspace allocation request — fixing it requires separating those two concerns and verifying the one-shot/two-shot split, which needs GB200 hardware. Leaving that for a follow-up. ## Workaround (no code change needed) VLLM_ALLREDUCE_USE_FLASHINFER=0 Disables the eager FlashInferAllReduce path; the compiled fusion pass is unaffected. ## Test Plan I have added test file and you can see output below, every assertion in the capacity-accounting logic changed ran and succeeded. pytest tests/compile/passes/distributed/test_fusion_all_reduce.py -v , already existing test. On GB200 python -m vllm.entrypoints.openai.api_server \\ --model --speculative-config '{\"model\": \"\"}' --tensor-parallel-size 8 if reporter can test on GB200 that would be perfect. ## Test Result Cannot test as I do not have GB200, if @ilmarkov can test it, that would be best. ## AI assistance This fix was developed with Claude (claude-opus-4-6) assistance. I have reviewed every changed line and understand the root cause and the design. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/50932", "createdAt": "2026-08-03T23:46:19Z", "updatedAt": "2026-08-13T16:10:15Z", "timestamp": "2026-08-13T16:10:15Z", "metrics": { "reactions": 2, "comments": 21 }, "labels": [ "bug", "ready", "nvidia" ], "author": "khushali9", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e4138cf793fc0095c156", "signalId": "github:vllm-project/vllm:pull_request:51323", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51323", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[1/N] HiSparse: host-resident sparse-MLA decode hot-buffering", "text": "Initial implementation based on #46326, with substantial improvements; notably, a heirarchical design which includes a standard device resident KV cache in addition to the hot buffer and host cache. Includes support for DeepSeek v4. Supersedes #46326 Preliminary results on B300: \"hisparse_pd_staged_pareto\" \"hisparse_blog_shapes\"", "url": "https://github.com/vllm-project/vllm/pull/51323", "createdAt": "2026-08-07T00:21:12Z", "updatedAt": "2026-08-13T16:09:56Z", "timestamp": "2026-08-13T16:09:56Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "documentation", "needs-rebase", "ci/build", "v1", "deepseek", "kv-connector", "nvidia", "mrv2" ], "author": "MatthewBonanni", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5333a786b361bd8e4019", "signalId": "github:vllm-project/vllm:pull_request:52016", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X dense linear backends", "text": " ## Purpose This PR integrates [B12X](https://github.com/local-inference-lab/b12x) dense linear kernels for NVIDIA SM120 and SM121 GPUs through the existing vLLM linear backend interfaces. B12X is an optional dependency installed with `vllm[b12x]` and pinned to `b12x==1.2.4`; it is a pure-Python CuTe DSL package and requires no additional vLLM build step. Supported linear paths are: - Per-tensor FP8. - 128x128 block-scaled FP8. - MXFP8. - NVFP4 and MXFP4. B12X participates in automatic kernel selection after established optimized backends and before emulation. Users can also request it with `--linear-backend b12x`; linear families that B12X does not implement, such as dense W4A16, retain normal backend selection so mixed-format models continue to work. Warmup is limited to layers that selected a B12X kernel. It covers every configured CUDA-graph capture size plus `max_num_batched_tokens`. B12X internally deduplicates shapes that select the same kernel policy, while vLLM deliberately avoids duplicating B12X policy-selection heuristics. The documentation covers installation, selection, supported formats, and fallback behavior. This is the linear and shared-integration component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 target FlashInfer-embedded B12X paths or narrower integrations and do not provide this standalone optional linear backend. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused dense-kernel and warmup coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/model_executor/kernels/test_b12x_mxfp4_linear.py \\ tests/model_executor/kernels/test_b12x_nvfp4_linear.py \\ tests/model_executor/kernels/test_b12x_mxfp8_linear.py \\ tests/model_executor/test_b12x_warmup.py \\ tests/kernels/quantization/test_block_fp8.py \\ -k b12x -q ``` Run pre-commit over every changed file: ```bash mapfile -t changed_files < <(git diff --name-only upstream/main...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check upstream/main...HEAD ``` Benchmark `Qwen/Qwen3.6-27B-FP8` on an RTX PRO 6000 Blackwell Max-Q GPU, holding the attention backend constant and comparing B12X with CUTLASS. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10 at the exact rebased PR head: ```text 52 passed, 473 deselected in 24.86s ``` All changed-file pre-commit hooks passed, including Ruff, formatting, mypy, and markdownlint; `git diff --check` also passed. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | Dense block FP8 | Qwen3.6-27B-FP8 | 1 | CUTLASS | 54.3975 | 56.7518 | +4.33% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52016", "createdAt": "2026-08-12T16:04:49Z", "updatedAt": "2026-08-13T16:09:53Z", "timestamp": "2026-08-13T16:09:53Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:eb0d372c5e601844373f", "signalId": "github:vllm-project/vllm:pull_request:52187", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52187", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][Perf] Zero new KV blocks by cache group", "text": "## Purpose #51749 correctly generalized recycled KV-page zeroing to every allocating `AttentionSpec`, preventing stale FP8 sliding-window pages. However, the scheduler currently flattens the newly allocated block IDs from every cache manager, and `KVBlockZeroer` applies that union to every attention-cache segment. On a hybrid or otherwise multi-group layout this creates unnecessary `all new block IDs x all cache groups` work. This PR preserves the cache-group association through `SchedulerOutput`, builds zeroing metadata per group, and zeros each newly allocated block only in its owning group's physical cache pages. It retains #51749's correctness coverage and #52058's bounded 3-D launch geometry. This is generic V1 KV-cache infrastructure, not an SM120-specific path. The DeepSeek-V4 SM120 configuration below is the reproducer because it has five active MLA/sliding-window managers. ### Relationship to adjacent work - #51749 supplies the required all-`AttentionSpec` correctness coverage. - #52058 fixed the major launch-geometry scaling problem and restored most of the prefill regression by using wide masked chunks on a 3-D grid. - #50485 is alternative 3-D launch-geometry work. - #52062 proposes reverting #51749; this PR instead preserves its stale-page fix while reducing the remaining redundant work. Open PR and issue searches for `KVBlockZeroer group zeroing`, `new_block_ids_to_zero`, `zeroing \"cache group\"`, and `\"owning group\" KV cache` found no equivalent group-aware change. ## Test Plan ```bash python -m pytest -q tests/v1/worker/test_kv_block_zeroer.py python -m pytest -q \\ tests/v1/kv_connector/unit/test_nixl_connector_hma.py::test_scheduler_filters_connector_loaded_blocks_from_zeroing \\ tests/v1/kv_connector/unit/test_nixl_connector_hma.py::test_failed_load_rezeroes_unwritten_skipped_blocks \\ tests/v1/worker/test_kv_block_zeroer.py::test_large_dsv4_launch_geometry \\ tests/v1/core/test_single_type_kv_cache_manager.py::test_sliding_window_records_new_blocks_for_zeroing \\ tests/v1/core/test_single_type_kv_cache_manager.py::test_chunked_local_attention_records_new_blocks_for_zeroing ruff format --check ruff check ``` Live A/B: DeepSeek-V4-Flash-0731, 2x RTX PRO 6000 Blackwell, TP2 + EP, FP8 KV, five cache managers, max model length 1M, FlashInfer 0.6.17. Both arms use the same rebuilt tree and dependency set; the control is the parent commit and the test arm adds only this commit. Each size has five sequential C=1 runs. ## Test Result - KV block zeroer GPU suite: `9 passed` - Focused scheduler/manager/geometry suite: `5 passed` - MLA gather compatibility slice: `34 passed, 2437 deselected` - Ruff format/check: passed - Cold-load non-thinking HTTP smoke: `200`, exact `OK.` Matched median prefill results after #52058: | Prompt | Control TTFT | This PR TTFT | Control tok/s | This PR tok/s | | ---: | ---: | ---: | ---: | ---: | | 128 | 166 ms | 156 ms (-6.0%) | 919 | 966 (+5.1%) | | 1K | 158 ms | 155 ms (-1.9%) | 6,641 | 6,753 (+1.7%) | | 4K | 498 ms | 503 ms (+1.0%) | 8,268 | 8,189 (-1.0%) | | 16K | 2.221 s | 2.224 s (+0.1%) | 7,382 | 7,376 (-0.1%) | The benefit is concentrated in short-prefill latency; 4K and 16K are flat within about 1%. Three live decode runs averaged 304.7 tok/s, unchanged from the prior 304.0 tok/s five-run result. AI assistance was used to bisect the original regression, implement the change, and draft this description. I reviewed the diff and ran the validation above.", "url": "https://github.com/vllm-project/vllm/pull/52187", "createdAt": "2026-08-13T16:06:09Z", "updatedAt": "2026-08-13T16:06:45Z", "timestamp": "2026-08-13T16:06:45Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "kv-connector", "mrv2" ], "author": "alexbi29", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:8311648ca547cbdfcc83", "signalId": "github:vllm-project/vllm:pull_request:50082", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add Kimi K3 MoE support to benchmark_moe.py", "text": "## Purpose `benchmarks/kernels/benchmark_moe.py` cannot tune the fused MoE Triton kernel for Kimi K3. `get_model_params()` does not recognize `KimiK3ForConditionalGeneration`, so it falls through to the Mixtral default branch and raises `AttributeError` (Kimi's config has no `num_local_experts`). Kimi K3 is multimodal and stores its MoE parameters in a nested `KimiLinearConfig` `text_config`, and it exposes the routing top-k as `num_experts_per_token` rather than the more common `num_experts_per_tok` used by the existing handlers. This adds a dedicated branch, following the same pattern as the Qwen3.5 handler added in #37068. It also covers the text-only `KimiLinearForCausalLM`, for which `get_text_config()` returns the config itself. ## Test Plan Parse the released Kimi K3 config and confirm the extracted MoE parameters: ```python from vllm.transformers_utils.config import get_config from benchmark_moe import get_model_params cfg = get_config(\"\", trust_remote_code=True) print(get_model_params(cfg)) ``` ## Test Result ``` architectures = ['KimiK3ForConditionalGeneration'] (text_config: KimiLinearConfig) get_model_params -> (E=896, topk=16, moe_intermediate_size=3072, hidden_size=7168) ``` The values match the model config. With `--tp-size 8 --enable-expert-parallel` this yields `E_local=112` and shard `N=6144`. Before this change the script raised `AttributeError` before tuning could start.", "url": "https://github.com/vllm-project/vllm/pull/50082", "createdAt": "2026-07-28T04:27:24Z", "updatedAt": "2026-08-13T16:04:01Z", "timestamp": "2026-08-13T16:04:01Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "performance", "ready", "kimi", "k3" ], "author": "vanshbhatia-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c26dafd656ff6832e025", "signalId": "github:vllm-project/vllm:pull_request:51886", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51886", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KVConnector] Add retention interval to OffloadingConnector", "text": "## Purpose Add retention interval support to `OffloadingConnector`. The Mooncake KV connector supports retention interval support via the `envs.VLLM_PREFIX_CACHE_RETENTION_INTERVAL` env variable. It is used to avoid storing/loading blocks that are outside of the sliding window of the attention mechanism. This is implemented via block masks. The implementation in `OffloadingConnector` uses the same block mask mechanism to implement retention interval support. cc @varun-sundar-rabindranath Note: parts of the code generated with claude ## Test Plan - Run kv_offloading unit tests - Add new unit tests - run gptoss w/retention interval enabled. ## Test Result gpt-oss-20b results ``` Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-20b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":4294967296,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-20b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.315 │ 0.185 │ 36.5 │ 1284.8 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.305 │ 0.145 │ 33.4 │ 1434.2 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.325 │ 0.170 │ 25.2 │ 1840.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.325 │ 0.160 │ 29.8 │ 1577.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.345 │ 0.200 │ 22.3 │ 2138.7 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.320 │ 0.165 │ 28.1 │ 1678.6 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` gpt-oss-120b results ``` ● gpt-oss-120b — Retention Interval Eval Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-120b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-120b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.540 │ 0.145 │ 58.9 │ 725.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.540 │ 0.100 │ 51.7 │ 851.9 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.575 │ 0.085 │ 57.0 │ 769.3 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.555 │ 0.080 │ 50.9 │ 870.1 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.570 │ 0.095 │ 40.3 │ 1059.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.570 │ 0.080 │ 50.9 │ 851.3 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` DeepSeekV4 results showing reduction in memory stores for different intervals. ``` DeepSeek-V4-Flash (GSM8K, 200 questions, 8-shot, max_model_len=16384) Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve deepseek-ai/DeepSeek-V4-Flash \\ --enforce-eager \\ --max-model-len 16384 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --block-size 256 Where INTERVAL is unset (baseline), 512 (sparse), or 0 (most sparse). Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 8 \\ --host http://localhost \\ --port 8001 Accuracy: ┌──────────────────────────┬────────┬──────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.960 │ 48.5 │ 371.1 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.970 │ 53.0 │ 345.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ fresh │ 0.965 │ 54.3 │ 331.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ reload │ 0.955 │ 47.9 │ 380.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.970 │ 52.6 │ 343.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.960 │ 46.3 │ 389.6 │ └──────────────────────────┴────────┴──────────┴─────────────┴──────────┘ Prometheus Offload Metrics: ┌──────────────────────────┬─────────────────────────┬───────────────────────┐ │ Configuration │ Store Bytes │ Reduction vs Baseline │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Baseline (no interval) │ 4,610,304,000 (4.61 GB) │ — │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (interval=512) │ 192,430,080 (192 MB) │ 95.8% │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Most sparse (interval=0) │ 104,232,960 (104 MB) │ 97.7% │ └──────────────────────────┴─────────────────────────┴───────────────────────┘ ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51886", "createdAt": "2026-08-11T23:11:05Z", "updatedAt": "2026-08-13T16:03:52Z", "timestamp": "2026-08-13T16:03:52Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "bnellnm", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ecc1e3776419329388cb", "signalId": "github:vllm-project/vllm:pull_request:51784", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51784", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix][EPLB] Log correct avg/max token counts and balancedness", "text": "## Purpose Here is a fix for a small but confusing logging bug that I ran into: the EPLB balancedness log line reduced `avg_tokens`/`max_tokens` over the layer axis instead of the rank axis, so logged values were wrong whenever `num_moe_layers != num_ranks` leading to wrong conclusions about expert balance. Now both reduce the trailing rank axis. Log-only, actual rebalancing unaffected. AI-assisted, every line reviewed and tested by the submitter. ## Test Plan `pytest tests/distributed/test_eplb_state.py -v` (new CPU regression test, registered in the EPLB CI lane incl. AMD mirror). I added the test to pin down the problem. Happy to remove it if you don't see any value in it. ## Test Result Fails on `main` (`balancedness=0.517` vs correct `0.876`), passes with fix. No model evals: log-only change.", "url": "https://github.com/vllm-project/vllm/pull/51784", "createdAt": "2026-08-11T08:27:15Z", "updatedAt": "2026-08-13T16:02:29Z", "timestamp": "2026-08-13T16:02:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "needs-rebase", "ci/build" ], "author": "fjosw", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:164c483840217d1873be", "signalId": "github:vllm-project/vllm:pull_request:52079", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52079", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kimi-K3] Add GEMM-RS for sequence parallelism", "text": "## Purpose Add GEMM-RS kernel for Blackwell, based on https://github.com/NVIDIA/cutlass/blob/dcf215a/examples/python/CuTeDSL/cute/blackwell/kernel/distributed/distributed_gemm_reduce_scatter_blackwell.py (`multimem.ld_reduce`) - Supports any value of M e.g. M=1023. However, only uses GEMM-RS when M>=128 since the kernel was not optimized for small/medium M. Only supports TP<=16, and requires all rank on the same NVLink domain. - The sharding behavior follows existing RS logic i.e. eac rank holds `ceil(M / world_size)`, with the exception of the last rank - Requires an opt-in flag `VLLM_KIMI_K3_GEMM_RS`, which enables GEMM-RS for O-proj and shared experts+dense MLP down-proj - Symmetric memory workspace (per-GPU): `max_num_batched_tokens x 7168 x 2 bytes` = 448 MiB for MNBT=32k. Confirmed in vLLM logs KV memory 39.99 GiB (before) -> 39.78 GiB (after) -> not much Initialization and runtime logic - Whether to initialize GEMM-RS: done in `maybe_init_gemm_rs()`, which also logs the reason if it fails. When `VLLM_KIMI_K3_GEMM_RS=0`, it doesn't do anything - At each layer's `__init__()`, we call `self.run_gemm_rs = get_gemm_rs().can_run(self.down_proj.weight)`. This is to further validate supported weight shapes and dtype - When the checks fail, we fallback to standard behavior. - In `forward()`, we check again with `should_run()`, which is the heuristics M>=128. The kernel supports any values of M, but right now the baseline is better for M<128 Though technically this can work with any SP in general, this PR only enables GEMM-RS for Kimi-K3. A future extension is to make this into GEMM-AR by adding `multimem.st` (all-gather) after `multimem.ld_reduce` (reduce-scatter). ## Microbenchmark `benchmarks/kernels/benchmark_kimi_k3_gemm_rs.py` in this PR. CUDA graph with rotating buffers. All benchmarks were done with GB300. ### TP4 Note: K=1536 is shared expert down-proj, K=3072 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 1536 | 48.13 | 46.27 | 38.74 | 1.242 | 1.195 | | 512 | 7168 | 1536 | 84.64 | 60.51 | 50.69 | 1.67 | 1.194 | | 2048 | 7168 | 1536 | 119.92 | 110.83 | 83.74 | 1.432 | 1.323 | | 8192 | 7168 | 1536 | 286.58 | 343.22 | 209.25 | 1.37 | 1.64 | | 32768 | 7168 | 1536 | 1044.66 | 1265.86 | 719.7 | 1.452 | 1.759 | | 128 | 7168 | 3072 | 52.37 | 48.27 | 43.38 | 1.207 | 1.113 | | 512 | 7168 | 3072 | 92.35 | 66 | 53.09 | 1.74 | 1.243 | | 2048 | 7168 | 3072 | 145.6 | 133.81 | 89.44 | 1.628 | 1.496 | | 8192 | 7168 | 3072 | 383.65 | 431.82 | 234.38 | 1.637 | 1.842 | | 32768 | 7168 | 3072 | 1402.05 | 1628.43 | 1001.49 | 1.4 | 1.626 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 1536 | 17.65 | 39.89 | 38.74 | | 512 | 7168 | 1536 | 21.52 | 50.94 | 50.69 | | 2048 | 7168 | 1536 | 36.54 | 90.37 | 83.74 | | 8192 | 7168 | 1536 | 104.14 | 191.68 | 209.25 | | 32768 | 7168 | 1536 | 401.31 | 652.77 | 719.7 | | 128 | 7168 | 3072 | 18.96 | 41.38 | 43.38 | | 512 | 7168 | 3072 | 28.21 | 50.48 | 53.09 | | 2048 | 7168 | 3072 | 59.25 | 90.56 | 89.44 | | 8192 | 7168 | 3072 | 195.92 | 196.91 | 234.38 | | 32768 | 7168 | 3072 | 763.71 | 650.18 | 1001.49 | ### TP8 Note: K=768 is shared expert down-proj, K=1536 is O-proj | M | N | K | Torch GEMM + NCCL RS (RING_LL) (us) | Torch GEMM + NCCL RS (LDMC) (us) | GEMM-RS (us) | Speedup vs RING_LL | Speedup vs LDMC | |------:|-----:|-----:|--------------------------------------:|-----------------------------------:|---------------:|---------------------:|------------------:| | 128 | 7168 | 768 | 49.47 | 44.93 | 40.08 | 1.234 | 1.121 | | 512 | 7168 | 768 | 70.29 | 56.58 | 47.46 | 1.481 | 1.192 | | 2048 | 7168 | 768 | 110.34 | 100.75 | 81.1 | 1.36 | 1.242 | | 8192 | 7168 | 768 | 264.43 | 300.98 | 202.27 | 1.307 | 1.488 | | 32768 | 7168 | 768 | 928.83 | 1109.26 | 697.5 | 1.332 | 1.59 | | 128 | 7168 | 1536 | 51.26 | 46.86 | 41.63 | 1.231 | 1.126 | | 512 | 7168 | 1536 | 72.22 | 62.64 | 49.82 | 1.45 | 1.257 | | 2048 | 7168 | 1536 | 121.71 | 109.84 | 82.08 | 1.483 | 1.338 | | 8192 | 7168 | 1536 | 306.91 | 347.92 | 209.04 | 1.468 | 1.664 | | 32768 | 7168 | 1536 | 1108.27 | 1289.57 | 706.02 | 1.57 | 1.827 | Component breakdown | M | N | K | Torch GEMM (us) | NCCL RS (best) (us) | GEMM-RS (us) | |------:|-----:|-----:|------------------:|----------------------:|---------------:| | 128 | 7168 | 768 | 17.3 | 39.2 | 40.08 | | 512 | 7168 | 768 | 18.61 | 49.44 | 47.46 | | 2048 | 7168 | 768 | 26.64 | 91.39 | 81.1 | | 8192 | 7168 | 768 | 60.18 | 209.14 | 202.27 | | 32768 | 7168 | 768 | 216.29 | 714.27 | 697.5 | | 128 | 7168 | 1536 | 17.22 | 39.66 | 41.63 | | 512 | 7168 | 1536 | 23.17 | 51.39 | 49.82 | | 2048 | 7168 | 1536 | 36.64 | 90.43 | 82.08 | | 8192 | 7168 | 1536 | 104.14 | 210.16 | 209.04 | | 32768 | 7168 | 1536 | 400.02 | 713.71 | 706.02 | ## E2E prefill-only benchmark All benchmarks were done with 8xGB300, TP8+EP+SP (DeepGEMM MegaMoE), `--max-num-batched-tokens 32768`, 8k input - 1 output requests. Baseline is 7aa248f Concurrency | Baseline TTFT (median) | GEMM-RS TTFT (median) | Baseline TPGS | GEMM-RS TPGS -- | -- | -- | -- | -- C1 | 312.81 ms | 298.44 ms (-4.59%) | 3,172.7 tok/GPU/s | 3,432.2 tok/GPU/s (+8.18%) C32 | 7,240.71 ms | 6,799.51 ms (-6.09%) | 4,500.5 tok/GPU/s | 4,794.0 tok/GPU/s (+6.52%) ## Test Plan Unit test (also added to distributed CI) ``` tests/kernels/test_kimi_k3_gemm_rs.py ``` E2E testing, TP8+EP+SP (DeepGEMM MegaMoE) - GSM8K: 96.82% - OCRBench: 88.40% ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
**BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions)", "url": "https://github.com/vllm-project/vllm/pull/52079", "createdAt": "2026-08-13T01:15:53Z", "updatedAt": "2026-08-13T16:02:28Z", "timestamp": "2026-08-13T16:02:28Z", "metrics": { "reactions": 3, "comments": 8 }, "labels": [ "performance", "ci/build", "kimi", "k3" ], "author": "gau-nernst", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:e617e11baeba8c4fadc9", "signalId": "github:vllm-project/vllm:pull_request:50595", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50595", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Structured Output] Mask request stop tokens in xgrammar until grammar terminates", "text": "## Purpose Following #49227's merge, remove the patch used in HarmonyParser and update test to mirror production path. ## Test Plan `pytest tests/parser/test_harmony.py` ## Test Result `64 passed, 38 warnings in 26.74s` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50595", "createdAt": "2026-07-31T16:41:25Z", "updatedAt": "2026-08-13T16:02:27Z", "timestamp": "2026-08-13T16:02:27Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "structured-output", "tool-calling", "verified" ], "author": "yzong-rh", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:5c637816505ca4e6d9c7", "signalId": "github:vllm-project/vllm:pull_request:46847", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46847", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Bugfix] window-correct shuffled fp8 decode for SWA layers", "text": "## Purpose `AiterFlashAttentionImpl.forward` routes sliding-window (and sinks / multi-token spec-decode) decode through a branch that calls `flash_attn_with_kvcache` or `unified_attention`, both of which read the plain (non-shuffled) KV layout. That branch opens with `assert not rocm_aiter_ops.is_shuffle_kv_cache_enabled()`, so a sliding-window model with the shuffled fp8 cache enabled hard-fails at engine warmup: ``` AssertionError: Shuffle KV cache layout is not supported with sliding window, sinks, or speculative decoding (multi-token decode). ``` So you currently cannot run a sliding-window model with the shuffled fp8 KV layout. The non-windowed shuffled path (`paged_attention_common`) is fine; only the windowed case is blocked, because there was no kernel that reads the shuffled layout and applies a window. This replaces that blanket assert: when the shuffled layout is enabled and the case is sliding-window decode (one query token, no sinks), it reshapes the cache to the shuffled view and calls `paged_attention_decode_shuffle_swa`, which reads the shuffled fp8 layout and masks the window. Sinks and multi-token spec decode over the shuffled layout are still unsupported, so the assert is kept for those. ## Test Plan Built vLLM from source at current `main` on MI300X (gfx942), applied this patch and the companion aiter kernel (`paged_attention_decode_shuffle_swa`). Mistral-family SWA model (head 128, `sliding_window=4096`), `--kv-cache-dtype fp8`, `VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1`, `--attention-backend ROCM_AITER_FA`, `--max-model-len 8192` (above the window so SWA is actually active). Prompt longer than the window so decode crosses it. Compared against the bf16 default SWA path; decode dispatch confirmed with a per-call sentinel. ## Test Result gfx942, torch 2.10 / ROCm 7.2.1, GPU clock-locked: - Before this patch the build aborts at engine warmup with the assertion above. After: the SWA shuffled decode kernel fires on every decode step (at a 6722-token prompt the dispatch logged 2592 calls = 81 decode steps x 32 layers). A 7222-token-prompt run produces coherent, on-topic output. - fp8 shuffled vs bf16 default, batch 32: fp8 gives 2x KV-cache capacity (2,413,877 vs 1,206,938 tokens; max concurrency 294x vs 147x) at the same ~163.6 GiB reservation, with decode throughput at parity within run-to-run spread (bf16 1037-1293, fp8 1166-1226 tok/s). The point of the PR is that the windowed shuffled path now works at all instead of asserting; the deterministic win is the 2x capacity. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/46847", "createdAt": "2026-06-26T17:11:04Z", "updatedAt": "2026-08-13T16:01:37Z", "timestamp": "2026-08-13T16:01:37Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [ "bug", "rocm", "v1" ], "author": "reger-men", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:362e7ddf803bdec1de02", "signalId": "github:vllm-project/vllm:pull_request:44343", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44343", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DO NOT MERGE][Perf][Feat] Add generic cuteDSL LL FP32 router (GEMM)", "text": "Depends on: https://github.com/vllm-project/vllm/pull/50174 ## Motivation This kernel is intended to be a generic low-latency fp32-router kernel. We should not expect it to outperform highly specialized PTX/CUDA kernels tuned for one exact router shape. The goal is different: make the CuTeDSL implementation competitive with those specialized kernels while keeping the code flexible enough to support new router shapes without writing a new kernel each time. That flexibility matters because router shapes keep changing across new models. Without a dedicated shape-specific CUDA path, the fallback is typically cuBLAS/`torch.mm`, and the kernel-wise results show why a generic low-latency router kernel is still useful: even when it only ties or slightly trails the specialized dispatch kernel, it is dramatically faster than cuBLAS across these low-M router workloads. ## Kernel-level results Speedup is computed as `baseline_us / ll_fp32w_us`; values above `1.0x` mean `ll_fp32w` is faster. `dispatch` == `fp32_router_gemm_dispatch` ### MiniMax-M3 `(K=6144, N=128)` | M | Config | ll_fp32w us | dispatch us | cuBLAS us | Speedup vs dispatch | Speedup vs cuBLAS | |---:|---|---:|---:|---:|---:|---:| | 1 | `(384,1,1)` | 1.729 | 1.722 | 6.924 | 0.996x | 4.005x | | 2 | `(384,1,1)` | 1.825 | 1.856 | 13.618 | 1.017x | 7.462x | | 3 | `(384,1,1)` | 1.958 | 2.033 | 13.709 | 1.038x | 7.002x | | 4 | `(384,1,1)` | 2.172 | 2.212 | 13.920 | 1.018x | 6.409x | | 5 | `(384,1,1)` | 2.301 | 2.349 | 13.885 | 1.021x | 6.034x | | 6 | `(384,2,1)` | 2.422 | 2.565 | 15.746 | 1.059x | 6.501x | | 7 | `(256,1,1)` | 2.580 | 2.655 | 15.743 | 1.029x | 6.102x | | 8 | `(384,2,1)` | 2.614 | 2.813 | 14.011 | 1.076x | 5.360x | | 9 | `(384,1,1)` | 2.956 | 3.021 | 14.117 | 1.022x | 4.776x | | 10 | `(384,2,1)` | 2.983 | 3.085 | 14.107 | 1.034x | 4.729x | | 11 | `(384,1,1)` | 3.212 | 3.195 | 14.229 | 0.995x | 4.430x | | 12 | `(192,2,1)` | 2.964 | 3.158 | 14.238 | 1.065x | 4.804x | | 13 | `(384,1,1)` | 3.374 | 3.477 | 14.255 | 1.031x | 4.225x | | 14 | `(192,2,1)` | 3.256 | 3.290 | 14.300 | 1.010x | 4.392x | | 15 | `(256,1,1)` | 3.683 | 3.662 | 14.311 | 0.994x | 3.886x | | 16 | `(192,2,1)` | 3.529 | 3.493 | 14.393 | 0.990x | 4.078x | | 17 | `(384,1,1)` | 3.891 | 3.919 | 15.516 | 1.007x | 3.988x | | 18 | `(192,2,1)` | 3.635 | 3.669 | 15.678 | 1.009x | 4.313x | | 19 | `(256,1,1)` | 4.018 | 4.074 | 15.735 | 1.014x | 3.916x | | 20 | `(192,2,1)` | 3.900 | 3.937 | 15.851 | 1.009x | 4.064x | | 21 | `(256,1,1)` | 4.273 | 4.317 | 15.826 | 1.010x | 3.704x | | 22 | `(192,2,1)` | 4.145 | 4.163 | 15.730 | 1.004x | 3.795x | | 23 | `(384,1,1)` | 4.560 | 4.689 | 15.888 | 1.028x | 3.484x | | 24 | `(192,2,1)` | 4.317 | 4.329 | 15.731 | 1.003x | 3.644x | | 25 | `(384,1,1)` | 4.836 | 4.867 | 15.777 | 1.006x | 3.262x | | 26 | `(128,2,1)` | 4.314 | 4.467 | 15.798 | 1.035x | 3.662x | | 27 | `(384,1,1)` | 5.052 | 5.121 | 15.899 | 1.014x | 3.147x | | 28 | `(128,2,1)` | 4.683 | 4.682 | 15.959 | 1.000x | 3.408x | | 29 | `(384,1,1)` | 5.318 | 5.442 | 16.048 | 1.023x | 3.018x | | 30 | `(128,2,1)` | 4.761 | 4.860 | 15.927 | 1.021x | 3.345x | | 31 | `(384,1,1)` | 5.531 | 5.559 | 16.046 | 1.005x | 2.901x | | 32 | `(192,2,1)` | 5.044 | 5.044 | 15.955 | 1.000x | 3.163x | ### GLM5.2 `(K=6144, N=256)` | M | Config | ll_fp32w us | dispatch us | cuBLAS us | Speedup vs dispatch | Speedup vs cuBLAS | |---:|---|---:|---:|---:|---:|---:| | 1 | `(384,1,1)` | 1.899 | 2.078 | 6.763 | 1.094x | 3.561x | | 2 | `(384,1,1)` | 2.102 | 2.300 | 21.317 | 1.094x | 10.141x | | 3 | `(384,1,1)` | 2.302 | 2.607 | 21.377 | 1.132x | 9.286x | | 4 | `(128,1,1)` | 2.615 | 2.938 | 21.326 | 1.124x | 8.155x | | 5 | `(128,1,2)` | 2.790 | 2.877 | 21.633 | 1.031x | 7.754x | | 6 | `(128,1,1)` | 2.964 | 3.103 | 15.648 | 1.047x | 5.279x | | 7 | `(128,1,2)` | 3.100 | 3.248 | 15.757 | 1.048x | 5.083x | | 8 | `(128,2,2)` | 3.197 | 3.476 | 20.018 | 1.087x | 6.261x | | 9 | `(128,1,2)` | 3.458 | 3.702 | 20.244 | 1.071x | 5.854x | | 10 | `(128,2,2)` | 3.359 | 3.716 | 20.326 | 1.106x | 6.051x | | 11 | `(128,1,1)` | 3.850 | 3.880 | 20.396 | 1.008x | 5.298x | | 12 | `(128,2,2)` | 3.758 | 4.033 | 20.414 | 1.073x | 5.432x | | 13 | `(256,1,2)` | 4.203 | 4.190 | 20.516 | 0.997x | 4.881x | | 14 | `(128,2,2)` | 3.857 | 4.433 | 20.524 | 1.149x | 5.321x | | 15 | `(256,1,2)` | 4.556 | 4.526 | 20.718 | 0.993x | 4.547x | | 16 | `(128,2,2)` | 4.138 | 4.434 | 20.798 | 1.072x | 5.026x | | 17 | `(256,1,2)` | 4.844 | 5.002 | 17.310 | 1.033x | 3.573x | | 18 | `(128,2,2)` | 4.433 | 4.751 | 17.400 | 1.072x | 3.925x | | 19 | `(256,1,2)` | 5.173 | 5.135 | 17.421 | 0.993x | 3.368x | | 20 | `(128,2,2)` | 4.653 | 4.848 | 17.405 | 1.042x | 3.741x | | 21 | `(256,1,2)` | 5.427 | 5.561 | 17.370 | 1.025x | 3.201x | | 22 | `(128,2,2)` | 4.909 | 5.031 | 17.493 | 1.025x | 3.563x | | 23 | `(256,1,2)` | 5.793 | 5.808 | 17.641 | 1.003x | 3.045x | | 24 | `(128,2,2)` | 5.360 | 5.252 | 17.626 | 0.980x | 3.288x | | 25 | `(256,1,2)` | 6.141 | 6.199 | 17.617 | 1.009x | 2.869x | | 26 | `(128,2,2)` | 5.631 | 5.542 | 17.686 | 0.984x | 3.141x | | 27 | `(256,1,2)` | 6.458 | 6.563 | 17.537 | 1.016x | 2.716x | | 28 | `(128,2,2)` | 5.850 | 5.914 | 17.588 | 1.011x | 3.006x | | 29 | `(256,1,2)` | 6.763 | 6.891 | 17.694 | 1.019x | 2.616x | | 30 | `(128,2,2)` | 6.139 | 6.223 | 17.623 | 1.014x | 2.871x | | 31 | `(256,1,2)` | 7.099 | 7.162 | 17.849 | 1.009x | 2.514x | | 32 | `(128,2,2)` | 6.399 | 6.431 | 17.834 | 1.005x | 2.787x | ## E2E serving results (MiniMax-M3) | Batch | main run | pr run | main median ITL | pr median ITL | median ITL diff | main tok/s | pr tok/s | tok/s diff | |---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1 | 2 | 3 | 5.73 | 5.73 | +0.0% | 173.79 | 173.79 | +0.0% | | 2 | 3 | 3 | 6.44 | 6.44 | +0.0% | 308.71 | 308.72 | +0.0% | | 3 | 3 | 2 | 7.60 | 7.64 | +0.5% | 392.90 | 391.41 | -0.4% | | 4 | 3 | 3 | 7.55 | 7.53 | -0.3% | 527.77 | 529.02 | +0.2% | | 5 | 2 | 3 | 8.62 | 8.53 | -1.0% | 574.68 | 579.33 | +0.8% | | 6 | 3 | 2 | 8.97 | 8.92 | -0.6% | 662.75 | 668.50 | +0.9% | | 7 | 2 | 2 | 9.29 | 9.31 | +0.2% | 745.72 | 747.07 | +0.2% | | 8 | 2 | 2 | 9.26 | 9.23 | -0.3% | 857.08 | 859.11 | +0.2% | | 9 | 2 | 3 | 10.18 | 10.19 | +0.1% | 878.64 | 873.63 | -0.6% | | 10 | 3 | 3 | 10.42 | 10.40 | -0.2% | 955.08 | 954.98 | -0.0% | | 11 | 3 | 2 | 10.72 | 10.72 | +0.0% | 1019.76 | 1021.32 | +0.2% | | 12 | 3 | 2 | 11.07 | 10.99 | -0.7% | 1077.26 | 1081.18 | +0.4% | | 13 | 3 | 3 | 11.25 | 11.32 | +0.6% | 1147.66 | 1138.11 | -0.8% | | 14 | 2 | 3 | 11.44 | 11.42 | -0.2% | 1209.73 | 1206.35 | -0.3% | | 15 | 2 | 2 | 11.79 | 11.74 | -0.4% | 1260.54 | 1264.29 | +0.3% | | 16 | 2 | 3 | 11.70 | 11.66 | -0.3% | 1348.45 | 1356.88 | +0.6% |", "url": "https://github.com/vllm-project/vllm/pull/44343", "createdAt": "2026-06-02T16:54:43Z", "updatedAt": "2026-08-13T16:00:58Z", "timestamp": "2026-08-13T16:00:58Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "ci/build" ], "author": "LopezCastroRoberto", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a6d3f2a8b75eae7e462f", "signalId": "github:vllm-project/vllm:pull_request:52178", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52178", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Propagate fatal EngineCore shutdown", "text": "## Purpose Fixes #48966. When `EngineCore` dies unexpectedly, the watchdog asks Uvicorn to stop, but `serve_http()` currently returns the same no-op shutdown coroutine used for a normal server stop. The caller therefore completes cleanly and the serving process exits with status 0, so supervisors using `Restart=on-failure` do not restart it. This change keeps a reference to the `EngineClient` across HTTP teardown. After the server task finishes, it returns a coroutine that re-raises the recorded `dead_error` only when the engine is dead and shutdown was not initiated by a user signal. SIGINT/SIGTERM and a healthy server stop retain successful exit semantics. This PR is opened as a draft to confirm the process exit contract and the placement of the check in the HTTP launcher before marking it ready. The exception is not consumed by an intermediate cleanup handler on the single-server path: `serve_http()` returns the failure coroutine, `run_server_worker()` awaits it after leaving the engine-client context and closes the socket in a `finally`, `run_server()` lets it propagate, and the CLI calls `uvloop.run(run_server(args))` without a broad catch. The process-level result below verifies the end of that chain rather than only the coroutine. ## Test Plan 1. Unit-test the four relevant terminal states through `serve_http()`: unexpected engine death, intentional signal shutdown, a signal observed alongside engine death, and a healthy server stop. If both a user signal and engine death are observed before HTTP teardown completes, the deliberate signal takes precedence and preserves exit 0. The signal callback records this flag synchronously rather than scheduling a coroutine to set it later. 2. Start the OpenAI API server on one RTX 4090, wait for `/health`, send one successful completion request, then kill only `EngineCore` and record the top-level exit status. 3. Repeat the same startup and request path, but send SIGTERM to the API server as the control case. 4. Repeat the process-level validation with TP=2 on two RTX 4090 GPUs, including killing only one TP worker to exercise worker death propagation through EngineCore. ## Test Result ```text $ python -m pytest tests/entrypoints/unit_tests/test_launcher.py -q .... [100%] 4 passed in 1.39s $ python -m ruff check \\ vllm/entrypoints/launcher.py \\ vllm/entrypoints/openai/api_server.py \\ tests/entrypoints/unit_tests/test_launcher.py All checks passed! $ python -m ruff format --check \\ vllm/entrypoints/launcher.py \\ vllm/entrypoints/openai/api_server.py \\ tests/entrypoints/unit_tests/test_launcher.py 3 files already formatted ``` Process-level validation on NVIDIA GeForce RTX 4090, PyTorch 2.13.0+cu130: ```text EngineCore SIGKILL: ready=1 request_status=200 wrapper_rc=1 server_rc=1 Intentional SIGTERM: ready=1 request_status=200 wrapper_rc=0 server_rc=0 ``` TP=2 process-level validation on two NVIDIA GeForce RTX 4090 GPUs: | Injection after `/health` and HTTP 200 | wrapper_rc | server_rc | Timed out | | --- | ---: | ---: | ---: | | SIGKILL only `VllmWorker-0` | 1 | 1 | 0 | | SIGKILL only EngineCore | 1 | 1 | 0 | | SIGTERM API server | 0 | 0 | 0 | The worker trial exercised the full propagation chain: the EngineCore log reported `VllmWorker-0 died unexpectedly`, followed by an EngineCore fatal error and `EngineDeadError` in the API server. All three trials exited without leaving an API server, EngineCore, or TP worker process behind. The multi-port DP supervisor remains a separate parent-process lifecycle path and is not claimed as covered by this focused change. Follow-up: audit and, if necessary, separately fix nonzero child-exit propagation through the multi-port `DPSupervisor` parent process. That path has different lifecycle semantics and should have its own focused regression. ## Duplicate-work check Before preparing this draft, issue #48966 had no linked development and the open-PR searches for `48966` and `EngineCore exit status` returned no matching fix. ## AI assistance disclosure AI assistance was used for code review, test design, and drafting this PR description. The submitter reviewed the changed lines and ran the tests and process-level validation above.", "url": "https://github.com/vllm-project/vllm/pull/52178", "createdAt": "2026-08-13T14:30:50Z", "updatedAt": "2026-08-13T16:00:36Z", "timestamp": "2026-08-13T16:00:36Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "bug", "frontend" ], "author": "jackLei0901", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:261d0779438adfacb2c4", "signalId": "github:vllm-project/vllm:pull_request:51846", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51846", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Frontend] Fix prompt_logprobs=0 bypassing admission guards and chat echo", "text": "## Summary `prompt_logprobs=0` is a valid, meaningful `SamplingParams` value (\"return the prompt token's own logprob, no alternatives\"), but several checks along the request path test it with plain truthiness instead of `is not None`, so a value of `0` silently slips through: - `ChatCompletionRequest.to_sampling_params` inherits `top_logprobs` (which defaults to `0`, not `None`) for *any* `echo=True` chat request, even when the client never asked for logprobs at all. Every echo request therefore does unrequested prompt-logprobs work (extra `compute_logits` pass, `gather_logprobs`, forced device sync per prefill step, and an unrequested `prompt_logprobs` array in the response). `cast_bool(null)`-style truthy checks miss `0` the same way `bool(0)` does anywhere else in Python. - `AsyncLLM.add_request` and `check_logprobs` (both chat and completion protocol) use truthy checks, so `prompt_logprobs=0` bypasses the `--kv-sharing-fast-prefill` incompatibility guard and the documented `stream=True` rejection — while `GPUModelRunner` downstream checks `is not None` and hits a bare `assert`, which aborts the whole `execute_model` step and kills the engine-core process for **every** in-flight and future client, not just the offending request. - `LLMEngine.add_request` (the sync path used by `LLM.generate()`) had no such guard at all, for any value — so the same crash is reachable there too, for any `prompt_logprobs`, not just `0`. ## Fix - Only inherit `top_logprobs` for chat `echo` when `logprobs` was actually requested (`ChatCompletionRequest.logprobs`), mirroring `CompletionRequest`'s existing, already-correct idiom (its source field, `logprobs`, defaults to `None`, so its analogous branch is a no-op unless the client explicitly set it). - Use `is not None` consistently for the `--kv-sharing-fast-prefill` admission guard in both `AsyncLLM.add_request` (async path) and `LLMEngine.add_request` (sync path, guard added — there was none). - `check_logprobs` now rejects any non-`None` `prompt_logprobs` under `stream=True`, in both the chat and completion protocols, instead of only `> 0` / `== -1`. - Fixed a stray \"tokens, tokens,\" typo in the `GPUModelRunner` backstop assert message, touched while already on that exact line. Diff is intentionally scoped to these checks — no changes to the grammar, scheduler, or any GPU/kernel code. ## Not a duplicate of #49622 #49622 (open) adds the sync-path admission check to `LLMEngine.add_request` and fixes the assert typo, but keeps the **truthy** check (`params.prompt_logprobs`), so it does not close the `prompt_logprobs=0` gap, and it doesn't touch the chat `echo` implicit-enablement bug or the `stream=True` rejection gap. This PR's `LLMEngine.add_request` change and typo fix overlap with #49622 at the file level; if #49622 merges first this one becomes a small `is not None` correction on top of it (and drops the typo hunk), rather than a conflicting approach — happy to rebase either way. ## Test Plan Unit tests added: - `tests/entrypoints/openai/chat_completion/test_chat.py` — `test_chat_completion_request_echo_without_logprobs_does_not_set_prompt_logprobs`, `test_chat_completion_request_echo_with_logprobs_inherits_top_logprobs`, `test_chat_completion_request_explicit_prompt_logprobs_zero_is_preserved`, `test_chat_completion_request_rejects_prompt_logprobs_zero_with_stream` - `tests/entrypoints/openai/completion/test_completion.py` — `test_completion_request_rejects_prompt_logprobs_zero_with_stream` - `tests/v1/engine/test_async_llm.py` — `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` - `tests/v1/engine/test_llm_engine.py` — `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` Commands run (Linux, real install, not a container/CI runner): ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=cpu python -m pytest tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_echo_without_logprobs_does_not_set_prompt_logprobs \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_echo_with_logprobs_inherits_top_logprobs \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_explicit_prompt_logprobs_zero_is_preserved \\ tests/entrypoints/openai/chat_completion/test_chat.py::test_chat_completion_request_rejects_prompt_logprobs_zero_with_stream \\ tests/entrypoints/openai/completion/test_completion.py::test_completion_request_rejects_prompt_logprobs_zero_with_stream \\ -v # 5 passed # Red/green check: reverted only the 5 source files to their pre-fix state # (git checkout HEAD~1 -- ) and reran the same 3 assertion-bearing # tests -> all 3 FAILED (assert 0 is None / DID NOT RAISE VLLMValidationError), # confirming they genuinely exercise the bug. Restored the fix -> green again. ruff check # All checks passed! ruff format --check # 9 files already formatted ``` `test_kv_sharing_fast_prefill_rejects_prompt_logprobs_zero` (both the async and sync variants) require booting a real `LLM`/`AsyncLLM` instance. In my local sandbox this hits `AttributeError: '_OpNamespace' '_C' object has no attribute 'init_cpu_memory_env'` on **any** engine boot, including a completely vanilla one with no `kv_sharing_fast_prefill` involved — a precompiled-wheel/HEAD-source drift specific to my environment, not caused by this change. They should run normally in CI, where the build matches the commit under test. ## Model evaluation Not applicable — this is a request-validation/control-flow fix. It does not change model forward passes, sampling, or output tokens for any request that was already passing validation correctly. ## AI assistance disclosure This change was developed with AI assistance (Claude). I reviewed every changed line, ran the tests and lint listed above myself, and can defend the fix and its scope in review.", "url": "https://github.com/vllm-project/vllm/pull/51846", "createdAt": "2026-08-11T16:35:16Z", "updatedAt": "2026-08-13T15:59:23Z", "timestamp": "2026-08-13T15:59:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "frontend", "needs-rebase" ], "author": "YaseenBashaT", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:daa720e7c2806b99d1b2", "signalId": "github:vllm-project/vllm:pull_request:51208", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51208", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][AMD][Installation] add LMCache kv-connector installation and runtime packages to docker image", "text": "## Purpose Ship the LMCache KV connector in the ROCm image so it works out of the box. Note: vLLM's CUDA `docker/Dockerfile` installs LMCache when `INSTALL_KV_CONNECTORS=true`, and the release pipeline sets that for every published `vllm/vllm-openai` tag. docker/Dockerfile.rocm never referenced that arg, so vllm/vllm-openai-rocm ships without LMCache and users must build and install it themselves. This adds it so the ROCm release pipeline can enable it the same way. ### What this does - Adds a `build_lmcache` stage that compiles LMCache for HIP from source (`BUILD_WITH_HIP=1`, `CXX=hipcc`, `TORCH_DONT_CHECK_COMPILER_ABI=1`). A source build is required: LMCache publishes **no ROCm sdist on PyPI** (`>=0.4` is wheels-only), so `--no-binary` cannot force a HIP build. - Pinned to the v0.5.3 commit (140819c9d57a); LMCACHE_ROCM_ARCH defaults to gfx942;gfx950, covering MI300X/MI325X and MI350X/MI355X. - Installs the wheel with `--no-deps` plus four runtime packages. - `sortedcontainers` — hard requirement; without it `import lmcache.integration.vllm.lmcache_mp_connector` fails outright. - `opentelemetry-exporter-prometheus` — `lmcache server` calls `init_observability()` unconditionally, which imports `PrometheusMetricReader` (`mp_observability/otel_init.py:96`). Omitting it builds fine but the server dies at startup. - `cupy-rocm-7-0` — GPU cache-registration path. - `aiofile` — local-disk storage tier. - Gates everything behind build args, defaulting to off. ### Build-arg selection `INSTALL_KV_CONNECTORS` is an umbrella mirroring the CUDA Dockerfile's arg; each connector also has its own arg defaulting to it, so future connectors (like Mooncake) compose without renaming anything. | Build args | Result | | --- | --- | | *(none)* | nothing installed (default) | | `INSTALL_KV_CONNECTORS=true` | all connectors — CUDA release-pipeline parity | | `INSTALL_LMCACHE=true` | LMCache only | | `INSTALL_KV_CONNECTORS=true INSTALL_LMCACHE=false` | all except LMCache | Booleans rather than a list because the per-connector build stages are selected via `FROM _${ARG}`, which needs the value in the stage name. With a list the hipcc compile would run on every build and be discarded. ## Test Plan ```bash # (1) build docker buildx build -f docker/Dockerfile.rocm --target vllm-openai \\ --build-arg INSTALL_KV_CONNECTORS=true \\ -t vllm-rocm-lmcache:oob --load . # (2) start the LMCache server from the image, installing nothing docker run --rm --network host --ipc host \\ --device /dev/kfd --device /dev/dri --group-add video \\ -e LMCACHE_L1_SIZE_GB=906 --entrypoint bash vllm-rocm-lmcache:oob -c ' lmcache server --host 127.0.0.1 --port 5555 \\ --http-host 127.0.0.1 --http-port 8080 \\ --l1-size-gb \"$LMCACHE_L1_SIZE_GB\" --l1-init-size-gb 20 \\ --l1-read-ttl-seconds 7200 --chunk-size 1536 --max-workers 8 \\ --eviction-trigger-watermark 0.85 --eviction-ratio 0.10 \\ --eviction-policy LRU' ``` Also verified: build-arg selection matrix, and that a `false` build never runs the hipcc compile. ## Test Result Built and verified on **8x MI355X (gfx950), ROCm 7.2.3, torch 2.11**. **(1) Build** — succeeds. The in-Dockerfile smoke test passes, printing `lmcache 0.5.3`. Built wheel properties: | Check | Result | | --- | --- | | `DT_NEEDED` | `libamdhip64.so.7`, libtorch/c10 — **no CUDA libs** | | GPU archs in `c_ops` | `gfx942`, `gfx950` | | Python ABI | `cp312` | **(2) LMCache server starts out of the box**, nothing installed at runtime: ``` zmq 5555 -> accepting http 8080 -> HTTP 200 ``` Startup log shows shm capacity checked against the L1 request, L1 + L2 eviction controllers started, SessionManager cleanup thread, DeviceHostFuncDispatcher, mp-worker-reaper, and an 8-slot affinity pool matching `--max-workers 8`. **(3) No dependency downgrades** — the reason for `--no-deps`: | Package | Before | After | | --- | --- | --- | | numpy | 2.3.5 | 2.3.5 | | transformers | 5.14.1 | 5.14.1 | | opentelemetry-api / sdk | 1.44.0 | 1.44.0 | | opentelemetry-semantic-conventions | 0.65b0 | 0.65b0 | Added: `lmcache 0.5.3`, `sortedcontainers 2.4.0`, `aiofile 3.12.3`, `cupy-rocm-7-0 14.1.1`, `opentelemetry-exporter-prometheus 0.65b0`. **(4) Connectors register**: `['LMCacheConnectorV1', 'LMCacheMPConnector']`. **(5) Build-arg matrix** — all four rows behave as documented; with `INSTALL_LMCACHE=false` the `build_lmcache` stage is pruned entirely (verified by building `--target export_lmcache`: empty output, no clone, no compile). (6) Image size: 42.00 → 42.92 GB, +921 MB (+2.19%) — ~366 MB of that is CuPy, required by the GPU cache-registration path; the compiled extension is <8 MB. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51208", "createdAt": "2026-08-05T23:46:52Z", "updatedAt": "2026-08-13T15:58:15Z", "timestamp": "2026-08-13T15:58:15Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "rocm", "ready", "ci/build", "kv-connector" ], "author": "hongxiayang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3cc611a962f9a939bc1b", "signalId": "github:vllm-project/vllm:pull_request:50622", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50622", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends", "text": "# [ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends ## Summary Split overloaded `AITER_MXFP4_BF16` into distinct backends: - **`AITER_MXFP4_BF16`** → CK only (`AiterExperts`), behavior unchanged (gfx950). - **`AITER_TRITON_MXFP4_BF16`** (new) → aiter Triton `moe_gemm_a16w4` (`AiterW4A16ExpertsMonolithic`), available on **gfx942 / gfx950 / gfx1250**, selectable via `--moe-backend aiter_triton_mxfp4_bf16` and auto-selected for gpt-oss where CK is unavailable. ## Not a duplicate No open PR splits the CK vs Triton MXFP4-BF16 backends. ## Test plan / results Hardware: AMD MI300X (gfx942). Model: `openai/gpt-oss-120b`. Validated on the committed branch. **Unit test** (`tests/kernels/moe/test_ocp_mx_moe.py`): ``` pytest -s -v tests/kernels/moe/test_ocp_mx_moe.py::test_rocm_mxfp4_moe_oracle ``` → 3 passed (TRITON, TRITON_UNFUSED, AITER_TRITON_MXFP4_BF16), 2 skipped (AITER_MXFP4_BF16, AITER_MXFP4_FP8 — gfx950-only). **Accuracy** (lm_eval gsm8k, flexible-extract, TP1): `aiter_triton_mxfp4_bf16` = **0.903** — matches the pre-split path and the `--moe-backend triton` baseline (~0.906), confirming the refactor is numerically neutral. ### Performance sweep `vllm bench serve`, random dataset, input=1024 / output=1024, num_prompts = concurrency × 10. aiter Triton W4A16 (`aiter_triton_mxfp4_bf16`) vs the OAI `triton_kernels` backend (`--moe-backend triton`), output-token throughput (tok/s), mean ITL (ms), mean TTFT (ms): | TP | conc | aiter tok/s | triton tok/s | speedup | aiter ITL | triton ITL | aiter TTFT | triton TTFT | |----|------|-------------|--------------|---------|-----------|------------|------------|-------------| | 1 | 1 | 216 | 82 | 2.64x | 4.55 | 12.13 | 84 | 94 | | 1 | 8 | 740 | 467 | 1.59x | 9.74 | 16.70 | 1103 | 454 | | 1 | 32 | 1656 | 1137 | 1.46x | 17.58 | 27.46 | 1804 | 728 | | 1 | 128 | 3124 | 2651 | 1.18x | 39.19 | 47.39 | 1848 | 927 | | 2 | 1 | 276 | 107 | 2.58x | 3.56 | 9.30 | 71 | 72 | | 2 | 8 | 1086 | 706 | 1.54x | 6.74 | 10.96 | 645 | 388 | | 2 | 32 | 2484 | 1792 | 1.39x | 12.10 | 17.27 | 812 | 620 | | 2 | 128 | 4803 | 4016 | 1.20x | 25.61 | 31.14 | 1076 | 762 | | 4 | 1 | 309 | 117 | 2.65x | 3.18 | 8.52 | 54 | 58 | | 4 | 8 | 1430 | 859 | 1.67x | 5.16 | 8.92 | 445 | 408 | | 4 | 32 | 3691 | 2520 | 1.46x | 8.13 | 12.29 | 562 | 427 | | 4 | 128 | 7589 | 5845 | 1.30x | 16.21 | 21.32 | 681 | 591 | | 8 | 1 | 290 | 143 | 2.03x | 3.40 | 6.96 | 54 | 55 | | 8 | 8 | 1667 | 873 | 1.91x | 4.35 | 8.69 | 460 | 489 | | 8 | 32 | 4937 | 3003 | 1.64x | 5.93 | 10.29 | 568 | 381 | | 8 | 128 | 10794 | 7304 | 1.48x | 11.05 | 16.91 | 832 | 634 | **Mean speedup 1.73× (range 1.18–2.65×)** across all 16 points; ITL consistently lower. aiter TTFT is higher than triton at TP1 mid-concurrency (prefill scheduling) and evens out at TP4/TP8. > Known issue, tracked separately: the aiter Triton W4A16 path shows a GPQA > accuracy regression at **TP4** (a partial-scale-group shard shape); TP1/2/8 > are at parity (GPQA aiter 0.65/0.67/0.63 vs triton 0.67/0.66/0.67). This is an > independent kernel bug, not caused by the backend split, and is not a blocker > for this structural refactor. ## AI assistance This change was developed with AI assistance. Andy has reviewed every changed line and run the tests/evals reported above.", "url": "https://github.com/vllm-project/vllm/pull/50622", "createdAt": "2026-07-31T20:14:49Z", "updatedAt": "2026-08-13T15:48:35Z", "timestamp": "2026-08-13T15:48:35Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm" ], "author": "afriedri", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:e7d052f2899beda1cd1a", "signalId": "github:vllm-project/vllm:issue:52181", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52181", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: FA2 is only supported on devices with compute capability >= 8", "text": "### 🚀 The feature, motivation and pitch I am trying to use the vllm for loading [QWEN3.6 model](https://huggingface.co/Qwen/Qwen3.6-27B) Operating system= rocky linux 9 GPU = 2X Quadro RTX 8000 48 GB As soon as I try to load with the command `vllm serve ./Qwen3.6-27B` there is error below. `(EngineCore pid=1673840) ERROR 08-13 16:42:32 [fa_utils.py:273] Cannot use FA version 2 is not supported due to FA2 is only supported on devices with compute capability >= 8` No matter i use quantized model or the simple, I cannot load. This is bottle neck and this must have some solution to load via some flag or additional parameter. Kindly I need your support and help to use this model on the mentioned GPU. ### Alternatives Tell me the way how to make a quantized version of this above model with vllm which I can load and use on the cuda compute capability 7.5 as my GPU is Quadro RTX 8000 which has power less than 8.0 cuda compute capability ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52181", "createdAt": "2026-08-13T14:52:13Z", "updatedAt": "2026-08-13T15:48:26Z", "timestamp": "2026-08-13T15:48:26Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "feature request", "quantization" ], "author": "Tortoise17", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:730a6049b619ede61c7d", "signalId": "github:vllm-project/vllm:pull_request:52184", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52184", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "feat(inkling): add Eagle3 hidden state extraction support", "text": "## Summary Enable speculative decoding training for Inkling models by adding Eagle3 hidden state extraction support. This PR addresses three interconnected issues that prevented hidden state collection from working end-to-end: - **`vllm/models/inkling/nvidia/model.py`** -- Added `EagleModelMixin` to `InklingModel` and `SupportsEagle3` to `_TmlForCausalLMBase`. Modified `InklingModel.forward()` to collect auxiliary hidden states via `_maybe_add_hidden_state`. Without this, vLLM raises \"Model does not support EAGLE3 interface\" when trying to extract hidden states for speculative decoding training. - **`vllm/v1/core/kv_cache_utils.py`** -- Fixed KV cache page size handling for hidden state cache specs. The hidden state cache (e.g. 7 layers x 4096 dim x 2 bytes = 57344 bytes/token) can exceed the regular KV cache page size. `get_uniform_page_size` now returns `max(page_sizes)` instead of asserting uniformity, and hidden state alignment uses `max(common_page, actual_page)` for `page_size_padded`. - **`vllm/v1/worker/gpu_model_runner.py`** -- Fixed KV cache tensor reshape for non-uniform page sizes. In `_reshape_kv_cache_tensors`, raw tensors allocated with `max(page_sizes)` across groups couldn't always be evenly divided by each spec's native page size. Fixed by falling back to `kv_cache_config.num_blocks` and slicing oversized tensors before reshape. ## Test plan - [ ] Verify Inkling model loads with Eagle3 hidden state extraction enabled (no \"Model does not support EAGLE3 interface\" error) - [ ] Verify KV cache allocation succeeds with mixed page sizes (hidden state cache + regular KV cache) - [ ] Verify KV cache tensor reshape handles non-uniform page sizes without assertion errors - [ ] Run speculative decoding training data generation with Inkling model end-to-end 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52184", "createdAt": "2026-08-13T15:06:49Z", "updatedAt": "2026-08-13T15:48:25Z", "timestamp": "2026-08-13T15:48:25Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "documentation", "new-model", "ci/build", "qwen", "cpu", "nvidia", "quantization", "kimi", "k3" ], "author": "orestis-z", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:99adcd99026214a77476", "signalId": "github:vllm-project/vllm:pull_request:52082", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] Fix memory pinning in Qwen3_VisionTransformer for non-gpu", "text": "Fix for CI failure introduced by https://github.com/vllm-project/vllm/pull/51841", "url": "https://github.com/vllm-project/vllm/pull/52082", "createdAt": "2026-08-13T02:35:39Z", "updatedAt": "2026-08-13T15:46:52Z", "timestamp": "2026-08-13T15:46:52Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready", "needs-rebase", "qwen" ], "author": "njhill", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:08b41e689a6bcc45dab1", "signalId": "github:vllm-project/vllm:pull_request:51987", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51987", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Revert \"[Attention] Add FlashInfer XQA decode support on SM12x\" (#49718)", "text": "Reverts #49718 ([Attention] Add FlashInfer XQA decode support on SM12x). ## Why `GPQA Eval (GPT-OSS) (DGX Spark)` regressed in nightly [build 83511](https://buildkite.com/vllm/ci/builds/83511) (commit `02ac178`) and is still red on the later build 83539. - Baseline nightly 83443 (same agent `spark-0df9-1`): **passed**, score 0.5619, 9m27s, zero harmony parse errors. - Build 83511: `FlashInfer resolved query dtypes: prefill=torch.bfloat16, decode=torch.bfloat16, decode_backend=xqa, kv_cache_dtype=torch.bfloat16, arch=sm121` — the dedicated XQA decode path this PR enables for `is_device_capability_family(120)`. That log line does not exist at baseline. - gpt-oss-20b decode output becomes gibberish: **1081** `openai_harmony.HarmonyError: unexpected tokens remaining in message header` (0 at baseline), and the eval exceeds its 1800s budget — `RuntimeError: Evaluation timed out`, 33m14s vs 9m27s. - The sibling GPQA jobs on 2xB200 and 2xH100 passed, so the blast radius is confined to SM12x, matching this PR's gating. gpt-oss uses attention sinks, which this PR routes through the new SM12x XQA path (`FlashInfer supports sinks on SM12x XQA and SM100 trtllm-gen`) — that is the most likely source of the corrupt decode output. Reverting restores the previous SM12x decode selection. A re-land should gate the dedicated XQA path (at minimum for sink-enabled models) until it is validated on GB10/sm121. - Failure count linked to this PR: 1 (`GPQA Eval (GPT-OSS) (DGX Spark)`) - Build: 83511 _Auto-generated by CI failure analyzer._", "url": "https://github.com/vllm-project/vllm/pull/51987", "createdAt": "2026-08-12T12:04:41Z", "updatedAt": "2026-08-13T15:44:47Z", "timestamp": "2026-08-13T15:44:47Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "nvidia" ], "author": "vllm-agent", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8b1000388b0364701b7b", "signalId": "github:vllm-project/vllm:pull_request:52186", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52186", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][Bugfix] Fix XPU crash with speculative decoding + prefix caching in Mamba models", "text": "## Purpose Model tested: cyankiwi/Qwen3.6-27B-AWQ-INT4 HW Backend: XPU Hardware: Intel ARC Pro B70 vllm version: 0.27.1 If I only enable prefix caching, I see no error and vllm can process requests. If I only use spec decoding(MTP) that works fine too. But if I use both together I get following error: ``` ValueError: Overflow when unpacking long long File \"vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context self.state_base_addrs[idx] = state.data_ptr() ``` Root cause: On XPU, `tensor.data_ptr()` can return memory addresses >= 2^63 — exceeding the signed `int64` range. The `MambaSpecDecodeGPUContext` class stores these addresses via element-wise assignment (`tensor[idx] = value`), which fails because PyTorch's `__setitem__` converts the scalar through C-level long() before any dtype check, regardless of the tensor's declared dtype. Fix applied: Two changes, both in `vllm/v1/worker/mamba_utils.py`: 1. Change dtype from int64 to uint64 for state_base_addrs and block_table_ptrs — matching the existing pattern in block_table.py (# NOTE(woosuk): Use uint64 instead of int64 to cover all possible addresses). 2. Collect pointers into a list and bulk-create the tensor instead of element-wise assignment. torch.tensor(list, dtype=torch.uint64) correctly handles large pointer values; tensor[idx] = large_int does not (even with uint64 dtype). ## Test Plan Re-build and run the same model with both `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'` and `--enable-prefix-caching`. Send a request and it should not throw following error. Logs before: ``` vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.1.dev19912+g373592ef5.d20260813) with config: model='cyankiwi/Qwen3.6-27B-AWQ-INT4', speculative_config=SpeculativeConfig(method='mtp', model='cyankiwi/Qwen3.6-27B-AWQ-INT4', num_spec_tokens=3), tokenizer='cyankiwi/Qwen3.6-27B-AWQ-INT4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=True, dtype=torch.float16, max_seq_len=138000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=cyankiwi/Qwen3.6-27B-AWQ-INT4, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '/root/.cache/vllm/torch_compile_cache/bbdfd623a8', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': '/root/.cache/vllm/torch_compile_cache/bbdfd623a8/rank_0_0/eagle_head', 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto'), vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-bd930efe188b6bb9-b4ce8602,prompt_token_ids_len=14926,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[248044], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=16384, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=False, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([0, 0, 0, 0, 1, 2, 3, 4], [0, 0, 0, 0, 5, 6, 7, 8], [0, 0, 0, 0, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={chatcmpl-bd930efe188b6bb9-b4ce8602: 8000}, total_num_scheduled_tokens=8000, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 6], finished_req_ids=[], free_encoder_mm_hashes=[], scheduled_encoder_input_stats=null, preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null, ec_manager_metadata=null, new_block_ids_to_zero=[13, 14, 15, 16, 17, 18], kv_cache_block_copies=null, partial_tail_offloads=null, num_spec_tokens_to_schedule=3) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.17307692307692313, iteration_details=None, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=14926, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] EngineCore encountered a fatal error. vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] Traceback (most recent call last): vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1337, in run_engine_core vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] engine_core.run_busy_loop() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/fault_tolerance/engine_core_sentinel.py\", line 179, in run_with_fault_tolerance vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] busy_loop_func(self) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1381, in run_busy_loop vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] self._process_engine_step() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1434, in _process_engine_step vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] outputs, model_executed = self.step_fn() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 652, in step_with_batch_queue vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] exec_future = self.model_executor.execute_model( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 131, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] output.result() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return self.__get_result() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] raise self._exception vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 109, in collective_rpc vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] result = run_method(self.driver_worker, method, args, kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py\", line 510, in run_method vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py\", line 351, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return self.worker.execute_model(scheduler_output) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 1086, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] output = self.model_runner.execute_model( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4438, in execute_model vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] mamba_utils.preprocess_mamba( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 1190, in preprocess_mamba vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] fused.ctx.initialize_from_forward_context( vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] self.state_base_addrs[idx] = state.data_ptr() vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ~~~~~~~~~~~~~~~~~~~~~^^^^^ vllm-server | (EngineCore pid=284) ERROR 08-13 12:20:39 [core.py:1346] ValueError: Overflow when unpacking long long vllm-server | (EngineCore pid=284) INFO 08-13 12:20:39 [xpu_worker.py:167] XPUWorker shutdown: cleaning up (rank=0, local_rank=0) vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] AsyncLLM output_handler failed. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] Traceback (most recent call last): vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 690, in output_handler vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] outputs = await engine_core.get_output_async() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py\", line 1104, in get_output_async vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] raise self._format_exception(outputs) from None vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [async_llm.py:744] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] Error in chat completion stream generator. vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] Traceback (most recent call last): vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/chat_completion/serving.py\", line 488, in chat_completion_stream_generator vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] async for res in result_generator: vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 607, in generate vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] out = q.get_nowait() or await q.get() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] ^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/output_processor.py\", line 85, in get vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] raise output vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py\", line 690, in output_handler vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] outputs = await engine_core.get_output_async() vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py\", line 1104, in get_output_async vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] raise self._format_exception(outputs) from None vllm-server | (APIServer pid=1) ERROR 08-13 12:20:39 [serving.py:841] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. vllm-server | (EngineCore pid=284) INFO 08-13 12:20:40 [xpu_worker.py:177] XPUWorker shutdown: done (rank=0, local_rank=0) vllm-server | (EngineCore pid=284) Process EngineCore: vllm-server | (EngineCore pid=284) Traceback (most recent call last): vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/multiprocessing/process.py\", line 314, in _bootstrap vllm-server | (EngineCore pid=284) self.run() vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/multiprocessing/process.py\", line 108, in run vllm-server | (EngineCore pid=284) self._target(*self._args, **self._kwargs) vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1348, in run_engine_core vllm-server | (EngineCore pid=284) raise e vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1337, in run_engine_core vllm-server | (EngineCore pid=284) engine_core.run_busy_loop() vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/fault_tolerance/engine_core_sentinel.py\", line 179, in run_with_fault_tolerance vllm-server | (EngineCore pid=284) busy_loop_func(self) vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1381, in run_busy_loop vllm-server | (EngineCore pid=284) self._process_engine_step() vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 1434, in _process_engine_step vllm-server | (EngineCore pid=284) outputs, model_executed = self.step_fn() vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py\", line 652, in step_with_batch_queue vllm-server | (EngineCore pid=284) exec_future = self.model_executor.execute_model( vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 131, in execute_model vllm-server | (EngineCore pid=284) output.result() vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 449, in result vllm-server | (EngineCore pid=284) return self.__get_result() vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/usr/lib/python3.12/concurrent/futures/_base.py\", line 401, in __get_result vllm-server | (EngineCore pid=284) raise self._exception vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py\", line 109, in collective_rpc vllm-server | (EngineCore pid=284) result = run_method(self.driver_worker, method, args, kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py\", line 510, in run_method vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py\", line 351, in execute_model vllm-server | (EngineCore pid=284) return self.worker.execute_model(scheduler_output) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 1086, in execute_model vllm-server | (EngineCore pid=284) output = self.model_runner.execute_model( vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context vllm-server | (EngineCore pid=284) return func(*args, **kwargs) vllm-server | (EngineCore pid=284) ^^^^^^^^^^^^^^^^^^^^^ vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 4438, in execute_model vllm-server | (EngineCore pid=284) mamba_utils.preprocess_mamba( vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 1190, in preprocess_mamba vllm-server | (EngineCore pid=284) fused.ctx.initialize_from_forward_context( vllm-server | (EngineCore pid=284) File \"/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/mamba_utils.py\", line 767, in initialize_from_forward_context vllm-server | (EngineCore pid=284) self.state_base_addrs[idx] = state.data_ptr() vllm-server | (EngineCore pid=284) ~~~~~~~~~~~~~~~~~~~~~^^^^^ vllm-server | (EngineCore pid=284) ValueError: Overflow when unpacking long long vllm-server | (APIServer pid=1) INFO: Shutting down vllm-server | (APIServer pid=1) INFO: Waiting for application shutdown. vllm-server | (APIServer pid=1) INFO: Application shutdown complete. vllm-server | (APIServer pid=1) INFO: Finished server process [1] ``` ## Test Result Mechanism: Locally built docker image with the applied change. vllm command: ``` vllm serve cyankiwi/Qwen3.6-27B-AWQ-INT4 --port 8000 --host 0.0.0.0 --trust-remote-code --tensor-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 138000 --kv-cache-dtype fp8 --max-num-seqs 1 --trust-remote-code --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --limit-mm-per-prompt '{\"image\": 0, \"video\": 0}' --generation-config vllm --override-generation-config '{\"temperature\": 0.6, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"repetition_penalty\": 1.0}' --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' --max_num_batched_tokens 8192 --enable-prefix-caching ``` Result: No errors Sample logs: ``` vllm-server | (APIServer pid=1) INFO: 127.0.0.1:45370 - \"POST /v1/chat/completions HTTP/1.1\" 200 OK vllm-server | (APIServer pid=1) INFO 08-13 14:45:21 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% vllm-server | (APIServer pid=1) INFO 08-13 14:45:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.87, Accepted throughput: 19.20 tokens/s, Drafted throughput: 20.10 tokens/s, Accepted: 192 tokens, Drafted: 201 tokens, Per-position acceptance rate: 1.000, 0.955, 0.910, Avg Draft acceptance rate: 95.5% vllm-server | (APIServer pid=1) INFO 08-13 14:45:31 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.3%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:45:51 [loggers.py:310] Engine 000: Avg prompt throughput: 3560.5 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:45:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.55, Accepted throughput: 2.80 tokens/s, Drafted throughput: 3.30 tokens/s, Accepted: 84 tokens, Drafted: 99 tokens, Per-position acceptance rate: 0.879, 0.848, 0.818, Avg Draft acceptance rate: 84.8% vllm-server | (APIServer pid=1) INFO 08-13 14:46:01 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 59.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.9%, Prefix cache hit rate: 20.2% vllm-server | (APIServer pid=1) INFO 08-13 14:46:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.97, Accepted throughput: 44.79 tokens/s, Drafted throughput: 45.29 tokens/s, Accepted: 448 tokens, Drafted: 453 tokens, Per-position acceptance rate: 1.000, 0.993, 0.974, Avg Draft acceptance rate: 98.9% ``` AI Disclosure: Have taken assistance of same model i.e. cyankiwi/Qwen3.6-27B-AWQ-INT4 via Claude code. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52186", "createdAt": "2026-08-13T15:42:58Z", "updatedAt": "2026-08-13T15:44:32Z", "timestamp": "2026-08-13T15:44:32Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "swakhandekar", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:10e34fa41bbf8de3c05b", "signalId": "github:vllm-project/vllm:pull_request:52122", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52122", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MiniCPM-V] Fix AssertionError in get_dummy_mm_data when passing VideoDummyOptions to _get_dummy_images", "text": "**Issue**: pytest tests/lora/test_minicpmv_tp.py::test_minicpmv_lora raises AssertionError on non-CUDA platforms (e.g., XPU). **Root Cause**: Commit 9a276d6375 added a runtime assertion to _get_dummy_images in dummy_inputs.py: assert overrides is None or isinstance(overrides, ImageDummyOptions) However, MiniCPMVDummyInputsBuilder.get_dummy_mm_data in minicpmv.py had always been passing video_overrides (type VideoDummyOptions) directly to _get_dummy_images when constructing per-frame dummy images for video. This type mismatch was silently ignored before the assertion was introduced. **Why upstream CI missed it**: test_minicpmv_lora is decorated with @pytest.mark.skipif(current_platform.is_cuda_alike(), ...), so it is skipped entirely on CUDA CI. **Fix**: In minicpmv.py, convert VideoDummyOptions before passing to _get_dummy_images: - Apply num_frames override to num_video_frames - Convert width/height into an ImageDummyOptions instance", "url": "https://github.com/vllm-project/vllm/pull/52122", "createdAt": "2026-08-13T08:06:44Z", "updatedAt": "2026-08-13T15:44:04Z", "timestamp": "2026-08-13T15:44:04Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "mayuyuace", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:62d196ea8d15cf50015d", "signalId": "github:vllm-project/vllm:pull_request:52148", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52148", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention] Fix FlashInfer SM12x prefill with sinks", "text": "## Summary Use FlashInfer's sink-aware paged prefill wrapper on SM12x when attention sinks are enabled. The generic FA2 prefill path accepts a `sinks` argument but does not apply it, so #49718 can use XQA for decode while producing incorrect prefill output. The wrapper is specialized with the active dtypes, head dimensions, sliding window, and softmax scale. DCP, NVFP4, SM90/SM100, and sink-free paths are unchanged. This is not a duplicate of the automatic revert #51987: that PR removes SM12x XQA support, while this change preserves XQA and fixes the prefill path for sink-attention models on SM12x . ## Validation - ` pytest tests/v1/attention/test_attention_backends.py::test_flashinfer_native_prefill_with_sinks -q` passing - SM12x GPT-OSS model eval fixed and passing - Nemotron and Qwen models both working as expected", "url": "https://github.com/vllm-project/vllm/pull/52148", "createdAt": "2026-08-13T10:33:31Z", "updatedAt": "2026-08-13T15:44:01Z", "timestamp": "2026-08-13T15:44:01Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "ready", "nvidia" ], "author": "askliar", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:09aac7654cef3737f0fc", "signalId": "github:vllm-project/vllm:pull_request:51575", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51575", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MRV2] Respect dynamic K=0 from scheduler in AutoRegressiveSpeculator", "text": "## Problem When Dynamic Speculative Decoding (DSD) selects K=0 at higher batch sizes, `AutoRegressiveSpeculator` ignores the scheduler's runtime K and executes the configured maximum number of draft steps. Those drafts are not used, so this wastes GPU work. The scheduler already publishes the selected K in `SchedulerOutput.num_spec_tokens_to_schedule`; MRV2 did not propagate that value to proposal execution. Fixes #51510. ## Solution - Propagate `num_spec_tokens_to_schedule` through `ExecuteModelState` into `speculator.propose()`. - For K=0, run draft prefill to keep the draft KV cache synchronized, then return an empty-width draft tensor without running draft decode. - Pass only the returned draft width to `DraftTokensHandler`, preventing stale columns in the reusable draft buffer from becoming phantom draft slots. - Preserve prior behavior when the runtime K is `None` or positive. Other speculator implementations accept the optional argument but do not act on it. ## Related work / duplicate check Searches for #51510 and MRV2 dynamic-K fixes found no other PR addressing this bug. #49652 is complementary, not duplicate. It fixes CUDA graph capture-shape derivation for autoregressive draft decode under dynamic SD. This PR carries the scheduler's runtime K into proposal execution and skips draft decode when that K is zero. A deployment using FULL CUDA graphs needs #49652's capture fix as well as this runtime behavior fix. ## Tests ```bash .venv/bin/python -m pytest \\ tests/v1/worker/test_gpu_autoregressive_speculator.py \\ tests/v1/worker/test_gpu_model_runner_v2_eplb.py -q ``` Result: `23 passed`. The new regression tests verify that: - K=0 still runs draft prefill but does not prepare or execute draft decode; - the K=0 proposal has shape `(num_requests, 0)`; - the scheduler-selected K reaches `speculator.propose()`; and - stale columns in the persistent draft buffer do not reach `DraftTokensHandler`. As a red/green check, removing the K=0 early return and active-width slice made the two new tests fail with width 2 instead of 0. Restoring the fix made them pass. ```bash uvx pre-commit run --files \\ tests/v1/worker/test_gpu_autoregressive_speculator.py \\ tests/v1/worker/test_gpu_model_runner_v2_eplb.py \\ vllm/v1/worker/gpu/model_runner.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ vllm/v1/worker/gpu/spec_decode/dflash/speculator.py \\ vllm/v1/worker/gpu/spec_decode/multi_module_mtp/speculator.py \\ vllm/v1/worker/gpu/spec_decode/speculator.py ``` Result: all applicable hooks passed. `git diff --check` passed. ## Benchmark 2× RTX 5090, ThinkingCap-Qwen3.6-27B-FP8, MRV2, TP=2, `FULL_AND_PIECEWISE` CUDA graphs, and DSD `[[1,2,2],[3,16,0]]`: | Sessions | Before | After | Change | No-spec reference | |---:|---:|---:|---:|---:| | 1 | 107 t/s | 108 t/s | +1% | 107 t/s | | 4 | 193 t/s | 211 t/s | +9% | 209 t/s | | 8 | 364 t/s | 399 t/s | +10% | 393 t/s | At one session the schedule selects K=2, so no K=0 improvement is expected. At four and eight sessions, K=0 is active. Instrumentation showed that the unpatched path spent approximately 7 ms on two unused draft-decode passes per K=0 step. ## AI assistance OpenAI Codex assisted with analysis, code drafting, and test drafting. I reviewed the changes and validation results and can explain and maintain the patch.", "url": "https://github.com/vllm-project/vllm/pull/51575", "createdAt": "2026-08-09T18:49:09Z", "updatedAt": "2026-08-13T15:43:07Z", "timestamp": "2026-08-13T15:43:07Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "mrv2" ], "author": "Suppressor72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8008554759e19702acff", "signalId": "github:vllm-project/vllm:pull_request:52048", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52048", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add DeepseekV4ForCausalLM to benchmark_moe.py model param dispatch", "text": "## Purpose Fixes #52042. `benchmark_moe.py --tune` crashes for DeepSeek-V4 models (e.g. `deepseek-ai/DeepSeek-V4-Flash-0731`): ```text AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` `get_model_params()` dispatches on the architecture string. The DeepSeek-family branch reads `n_routed_experts` / `num_experts_per_tok` / `moe_intermediate_size`, but `DeepseekV4ForCausalLM` is missing from its tuple, so V4 falls through to the Mixtral default, which reads `num_local_experts` — not present on DeepSeek configs. This adds `DeepseekV4ForCausalLM` (the registered arch name, see `vllm/model_executor/models/registry.py`) to that tuple. vLLM's own V4 model reads exactly those three fields when building the MoE gate and experts (`vllm/models/deepseek_v4/nvidia/model.py`), so the branch returns the correct shapes rather than plausible-but-wrong ones. `DeepSeekV4MTPModel` / `DSparkDraftModel` are intentionally left out: the dispatch carries no MTP/draft architecture for any family, and this benchmark tunes main-model MoE shapes. **Relation to existing PRs:** #49088 generalizes this dispatch to alias-based field probing and would make this change redundant, but it has been inactive since 2026-07-19; this is a minimal targeted unblock for the filed issue, and I am happy to close it if #49088 lands first. #40546 only improves the fallback error message. #50082 adds Kimi K3 via the same one-architecture pattern. ## Test Plan The change only affects config-field dispatch, so no GPU is required. Called `get_model_params()` with a `DeepseekV4Config` carrying V4-Flash MoE fields (`n_routed_experts=256`, `num_experts_per_tok=8`, `moe_intermediate_size=2048`, `hidden_size=7168`), plus a `DeepseekV3ForCausalLM` control to confirm the existing branch is unaffected. ## Test Result - Before: `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'` — the exact error from #52042 - After: returns `(256, 8, 2048, 7168)` - V3 control unchanged: `(160, 6, 1536, 5120)` - `ruff check` and `ruff format --check` clean on the file Model evaluation: not applicable — this is a benchmarking-script fix with no effect on model output, accuracy, or serving. --- This change was developed with AI assistance; I reviewed and tested every line.", "url": "https://github.com/vllm-project/vllm/pull/52048", "createdAt": "2026-08-12T21:09:52Z", "updatedAt": "2026-08-13T15:42:07Z", "timestamp": "2026-08-13T15:42:07Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "bug", "performance", "deepseek" ], "author": "rajathpi", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:b149888be68431f77f60", "signalId": "github:vllm-project/vllm:pull_request:51633", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51633", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Platform] Add check_runner_kv_caches_multi_layer", "text": "## Purpose Add `check_runner_kv_caches_multi_layer` interface to avoid platform hardcode in bind_kv_cache. So that oot platform can override it to avoid error raising. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51633", "createdAt": "2026-08-10T07:26:32Z", "updatedAt": "2026-08-13T15:36:51Z", "timestamp": "2026-08-13T15:36:51Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "rocm", "intel-gpu", "ready", "cpu", "nvidia" ], "author": "wangxiyuan", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:32a88dac0d73a135d9ee", "signalId": "github:vllm-project/vllm:pull_request:50894", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50894", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Scale KV page size for hidden states extraction with TP", "text": "## Summary - When `extract_hidden_states` is combined with tensor parallelism, the target model's KV page size shrinks (`num_kv_heads / TP`) but the hidden-state per-token cost stays at full `hidden_size`. This causes an assertion failure in `KVCacheSpecBase.__post_init__`: `assert self.page_size_padded >= real_page_size` - Scale up all target group block sizes when the hidden-state per-token cost exceeds the common page, before aligning hidden-state specs Fixes #51016 ## Test plan - [x] Unit test: `test_hidden_states_with_tp_scales_page_size` — verifies block sizes are scaled when TP-sharded KV pages are smaller than hidden-state cost - [x] Verified end-to-end: `poolside/Laguna-S-2.1` (117.6B MoE) + `extract_hidden_states` + TP=4 launches successfully and produces hidden states for training 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/50894", "createdAt": "2026-08-03T16:38:16Z", "updatedAt": "2026-08-13T15:36:00Z", "timestamp": "2026-08-13T15:36:00Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "bug", "needs-rebase" ], "author": "orestis-z", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f3f656d9adbaac7c85a0", "signalId": "github:vllm-project/vllm:pull_request:48684", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:48684", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Misc] Remove `override_attention_dtype`", "text": "## Purpose `override_attention_dtype` is only used for V0 and has been removd from https://github.com/vllm-project/vllm/pull/25351/ long time ago. It's safe to remove it now. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48684", "createdAt": "2026-07-15T02:47:30Z", "updatedAt": "2026-08-13T15:35:29Z", "timestamp": "2026-08-13T15:35:29Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "ready" ], "author": "wangxiyuan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ffdc02e9c244dec9a7c4", "signalId": "github:vllm-project/vllm:issue:52071", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52071", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: speculative decoding under pipeline parallelism produces wrong output with --no-async-scheduling", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Reproduced at `pipeline_parallel_size` 2, 4 and 8, with two different speculative methods and two model families. Code references are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug With `--no-async-scheduling`, speculative decoding under pipeline parallelism produces wrong output — or a CUDA fault — instead of being rejected or working. Turning async scheduling back on fixes it. ### Root cause `next_decode_eligible_step` is assigned in exactly one place in the tree: ``` vllm/v1/core/sched/async_scheduler.py:49 request.next_decode_eligible_step = self.current_step + self.pp_size ``` initialised in one place: ``` vllm/v1/request.py:167 self.next_decode_eligible_step = 0 ``` and read in exactly one place: ``` vllm/v1/core/sched/scheduler.py:509 if self.current_step < request.next_decode_eligible_step: # V2+PP+async: enforce `pp_size` steps between same-req decodes # to match worker-side sampled-tokens broadcast slot ring cadence. ``` The base `Scheduler` — which is what `--no-async-scheduling` selects — reads that field and never sets it. It stays `0`, the guard never fires, and nothing keeps a request's decodes `pp_size` steps apart. The worker-side sampled-token broadcast ring is then read out of phase. ### What that produces, measured A probe at the point where a step's input ids are built (`combine_sampled_and_draft_tokens`), PP=2: ``` last_sampled=[16925] drafts=[16925, 16925, 16925] healthy last_sampled=[0] drafts=[0, 0, 0] ~25% of spec steps ``` Token id 0 is embedded as the anchor. On the model under test token 0 is `!`, and the generated text fills with `!`. The same skew has a second face — the step gets scheduled without the accepted token that its drafts hang off: ``` healthy: hidden=(4,1024) logits_indices=[0, 1, 2, 3] qlen=4 num_logits=4 ndraft=3 failing: hidden=(3,1024) logits_indices=[-1, 0, 1, 2] qlen=3 num_logits=4 ndraft=3 ``` `logits_start = query_end - num_logits = 3 - 4 = -1`, so `hidden_states[input_batch.logits_indices]` either wraps to the last row silently (torch semantics) or trips: ``` Assertion `ind >=0 && ind < ind_dim_size && \"vectorized gather kernel index out of bounds\"` failed ``` The same skew also reaches `_prepare_prefill_inputs_kernel` in `v1/worker/gpu/spec_decode/autoregressive/speculator.py`, where `query_len -= num_rejected` can reach 0 and the unmasked `tl.store(draft_input_ids_ptr + query_start - 1, ...)` writes out of bounds — Triton adds negative indices to the base pointer rather than wrapping. ### Reproduced with - `method: mtp` on GLM-4.5-Air (AWQ), PP=4 - `method: dspark` on Kimi-K3, PP=8 - PP=2 on a 16-layer synthetic target ### Controls Two controls that rule out the obvious alternatives, same tree, same session: ``` speculation OFF, async on vs async off : outputs IDENTICAL Qwen3-0.6B, PP=1 vs PP=2, no speculation : outputs IDENTICAL ``` So the pipeline path is greedy-stable on its own, and the scheduler mode alone changes nothing. The defect appears only when speculation and PP are combined with async scheduling disabled. Also worth noting: with async scheduling ON, plain decode throughput on the same config was ~1.9× better than with it off (94.6 → 48.8 ms/token TPOT, GLM-4.5-Air PP=4, concurrency 1), so the flag is expensive in its own right. ### Suggested fix Any of these beats the current behaviour, in descending order of preference: 1. Set the cadence in the base scheduler too, so the guard works regardless of scheduling mode. 2. Reject `speculative_config` + `pipeline_parallel_size > 1` + `--no-async-scheduling` at config time, with a message naming the cadence. 3. At minimum, document that speculative decoding under PP requires async scheduling. Silently emitting `!` is the worst of the available outcomes, and it is what happens today. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52071", "createdAt": "2026-08-12T23:52:42Z", "updatedAt": "2026-08-13T15:33:40Z", "timestamp": "2026-08-13T15:33:40Z", "metrics": { "reactions": 0, "comments": 5 }, "labels": [ "kimi" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:91f669bf9c3deb7bef9b", "signalId": "github:vllm-project/vllm:issue:50682", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:issue:50682", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[ROCm][AMD] Kimi-K3 Gap and Roadmap Tracking", "text": "Kimi-K3 is a very successful launch. This issue is to track vLLM upstream work for feature enablement and performance optimization on ROCm.
Day 0 features/baselines - AITER fused-moe: a16w4 (GENERAL) and a8w4 (INTERLEAVE ) were integrated for performance. Flydsl/opus a8w4 is perf is 3.5% higher than flydsl a16w4. - AITER MLA attention: non-causal mla-gluon kernel was used for 12 head. - AITER triton conv2d - AMD vendor specific KDA - DSpark speculative decoding: Target model used AITER MLA (gluon) and draft model used TRITON_MLA. Perf improvement on lower concurrency (c1-c16). - Day 0 docker: vllm/vllm-openai-rocm:kimi-k3 - Day 0 recipes: https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x&features=reasoning - vLLM upstream PRs for day 0 K3 code: #50000 [#50089](https://github.com/vllm-project/vllm/pull/50089) [#50090](https://github.com/vllm-project/vllm/pull/50090)
**Feature Gaps**: - [ ] fp8 kv-cache-dtype support: Needed to avoid AgentX performance degradation. - asm-mla path: - https://github.com/vllm-project/vllm/pull/51040 - https://github.com/vllm-project/vllm/pull/51011 - gluon-mla path: https://github.com/vllm-project/vllm/pull/50619 - - [ ] gfx950/MI350 distributed inference: Required for multi-node K3 enablement and scaling. - https://github.com/vllm-project/vllm/pull/51052 - [x] DSpark: prefix-caching needs work - https://github.com/ROCm/aiter/pull/4474 - [ ] offloading performance/compatibility: Needed for offloading support and agentic workload. - cpu offloading + rocm 7.14/7.15 - [x] Dependency: 1. [ ] ROCm Base Docker: v0.1.19+ AITER upgrade - https://github.com/vllm-project/vllm/pull/49361 2. [ ] Triton 3.6 to 3.7+ upgrade - https://github.com/vllm-project/vllm/pull/50607 - Resolving issue of triton 3.7.x with gluon mla kernel on DistributedLinearLayout -- in progress * triton: https://github.com/ROCm/triton/pull/960 * vLLM to pick the new commit https://github.com/vllm-project/vllm/pull/51464 3. [ ] Triton-kernel package/naming management. - https://github.com/vllm-project/vllm/pull/50328 - [x] model runner V2 was disabled @vllmellm (Ye Hur, Pin Siang) - https://github.com/vllm-project/vllm/pull/51653 - [ ] gfx942 support and distributed inference - https://github.com/vllm-project/vllm/pull/50817 - **Performance Optimization**: - DSpark Speculative decoding: . * Goal: Agentic workflow: spec perf should be better than non-spec - fp8 kv-cache-dtype support: Dspark+fp8 kv cache+asm-mla path - minimal fix: https://github.com/vllm-project/vllm/pull/51860 - refactored: https://github.com/vllm-project/vllm/pull/51606 - qlen (K, num_speculative_tokens) tuning and support - https://github.com/vllm-project/vllm/pull/51590 - https://github.com/vllm-project/vllm/pull/51171 - - Update AITER in vLLM base docker: - https://github.com/ROCm/aiter/pull/4521 (merged) - https://github.com/ROCm/aiter/pull/4452 (merged) - https://github.com/ROCm/aiter/pull/4555 (merged) - https://github.com/ROCm/aiter/pull/4474 (merged) - https://github.com/ROCm/aiter/pull/4494 (merged) - https://github.com/ROCm/aiter/pull/4646 - https://github.com/ROCm/aiter/pull/4647 - https://github.com/ROCm/aiter/pull/4509 - - vLLM CPU offloading: Improve CPU-offloading performance (ROCm 7.14+). * https://github.com/vllm-project/vllm/pull/43018 * upgrade base docker to rocm 7.14 or above * - Agentic workload/benchmarking optimization. * profiling * max-num-seqs tuning - Mixed Attention optimization (Linear KDA 3 + MLA 1) - KDA : - Fusion: - #50654 - Prefill: - https://github.com/vllm-project/vllm/pull/51862 - https://github.com/vllm-project/vllm/pull/51540 - - MLA: - AITER Gluon MLA optimization (need new aiter release): * https://github.com/ROCm/aiter/pull/4509 (?) * https://github.com/ROCm/aiter/pull/4555 - Dispatch small head to AITER asm MLA (Done) * #50371 and #50578 and #51088 overlapped. Need consolidate to #50578 and add an opt-in env - - MoE: Flydsl a8w4 optimizations (multi-node, multi-stream) - [[FlyDSL] [MoE]: reuse stage-1(gate up) scratch buffer across layers and graph captures by xiaohuguo…](https://github.com/ROCm/aiter/pull/4647) - - Attention Res (triton and gluon), - Fusion: https://github.com/vllm-project/vllm/pull/50593 - Agentic workload (LMCache/Mooncake) - #51004 - gemm tuning - MORI-io + K3: (1) K3 KDA statement management. (2) UMBP (Unified Memory & Bandwidth Pool) to reduce cost without sacrificing performance. **Misc: Bug fixes, cleanup/refactor, documentation, recipe, packaging** - https://github.com/vllm-project/vllm/pull/50582 - #50649 - https://github.com/vllm-project/vllm/pull/51682 - - [ ] - LMCache packaging to vLLM docker https://github.com/vllm-project/vllm/pull/51208 - [ ] - Mooncake packaging to vLLM docker **PR List** https://github.com/vllm-project/vllm/pulls?q=is%3Aopen+is%3Apr+label%3Arocm+label%3Ak3 Note that: (1) There are some overlap and duplicated efforts, for example, #50371 and #50578. (2) PRs that do not meet vLLM Policy will not be reviewed, and may be closed by maintainers. ** Live Issues** https://github.com/vllm-project/vllm/issues?q=is%3Aissue%20state%3Aopen%20label%3Arocm%20label%3Ak3 --- Things are moving fast. Please comment if you have a PR or PRs for review. For each PR, please ensure you run the full gsm8k eval if for text-only. For multi-modal fixes, please run OCRBench/MMMU Pro. For performance optimization PR, in addition to ensure accuracy (eval test result is mandatory), please show end-to-end `vllm serve` command and environment variables you used, and the before/after comparison.", "url": "https://github.com/vllm-project/vllm/issues/50682", "createdAt": "2026-08-01T12:39:29Z", "updatedAt": "2026-08-13T15:25:30Z", "timestamp": "2026-08-13T15:25:30Z", "metrics": { "reactions": 0, "comments": 16 }, "labels": [ "rocm", "kimi", "k3" ], "author": "hongxiayang", "state": "open", "assignees": [ "hongxiayang" ], "change": "updated" } }, { "id": "event:cb92a93100e14d6bb2c5", "signalId": "github:vllm-project/vllm:pull_request:52124", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52124", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Renderer] Shutdown the renderer properly.", "text": "## Purpose Currently, during online serving, the renderer is shutdown twice, while it is not shutdown during offline inference. ``` offline: python examples/basic/offline_inference/embed.py this pr: INFO 08-13 16:11:10 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1340] [shutdown] EngineCore: exiting busy loop INFO 08-13 16:11:11 [base.py:305] [shutdown] BaseRenderer main: INFO 08-13 16:19:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20374) INFO 08-13 16:19:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop online: vllm serve intfloat/e5-small this pr: (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=9092) INFO 08-13 22:56:06 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=9032) INFO 08-13 22:56:06 [contextlib.py:478] [shutdown] BaseRenderer (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=9032) INFO 08-13 22:56:06 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=9032) WARNING 08-13 22:56:06 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=9032) WARNING 08-13 22:56:06 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 9092 (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=9032) INFO 08-13 22:56:06 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:141] [shutdown] API server: engine client stopped (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:144] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=9032) INFO 08-13 22:56:06 [launcher.py:165] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=9032) INFO: Shutting down (APIServer pid=9032) INFO: Waiting for application shutdown. (APIServer pid=9032) INFO: Application shutdown complete. main: (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1327] [shutdown] EngineCore: trigger received signal=SIGINT (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:114] [shutdown] API server: shutdown triggered (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:130] [shutdown] API server: stopping engine client mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1463] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1494] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=20593) INFO 08-13 16:21:49 [core.py:1340] [shutdown] EngineCore: exiting busy loop (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:689] [shutdown] MPClient: start timeout=0s (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:691] [shutdown] MPClient: stopping engine manager (APIServer pid=20533) INFO 08-13 16:21:49 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:635] [shutdown] Process manager: force killing remaining processes count=1 (APIServer pid=20533) WARNING 08-13 16:21:49 [utils.py:640] [shutdown] Process manager: force killing remaining process EngineCore pid 20593 (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:693] [shutdown] MPClient: engine manager stopped (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:694] [shutdown] MPClient: cleaning up background resources (APIServer pid=20533) INFO 08-13 16:21:49 [core_client.py:696] [shutdown] MPClient: complete (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:139] [shutdown] API server: engine client stopped (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:142] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=20533) INFO 08-13 16:21:49 [launcher.py:163] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=20533) INFO 08-13 16:21:49 [base.py:305] [shutdown] BaseRenderer (APIServer pid=20533) INFO: Shutting down (APIServer pid=20533) INFO: Waiting for application shutdown. (APIServer pid=20533) INFO: Application shutdown complete. ``` ## Best practices for shutdown? Using ExitStack and weakref.finalize provides a nearly perfect shutdown method, and running it a second time will have no effect. ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52124", "createdAt": "2026-08-13T08:22:32Z", "updatedAt": "2026-08-13T15:25:08Z", "timestamp": "2026-08-13T15:25:08Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend" ], "author": "noooop", "state": "open", "assignees": [ "sangstar" ], "change": "updated" } }, { "id": "event:bf46ef8f2c2c9fa7ebbe", "signalId": "github:vllm-project/vllm:pull_request:51855", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51855", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[K3] support recoverssm for K3", "text": " ## Purpose This PR adds ReplaySSM speculative decoding for the NVIDIA Kimi-K3 KDA path on Model Runner V2. It is intended for Kimi-K3 serving with DSpark and supports `mamba_cache_mode=align` prefix caching. Without ReplaySSM, KDA speculative decoding materializes a full recurrent state for every speculative position. ReplaySSM keeps one checkpoint and compact per-token records, then reconstructs the state at the accepted position after every verify. Under the tested TP8 Kimi-K3 configuration, this increases the effective cache capacity by 10.97% at the same 46.48 GiB cache-memory budget. ## Design ### Kimi-K3-specific V2 integration The implementation is contained in the Kimi-K3 KDA metadata, layer, and ReplaySSM operator paths. The metadata builder gives speculative rows one KDA checkpoint slot and creates a model-specific post-sampling committer. The V1 integration from the initial implementation was removed because Kimi-K3 does not run on Model Runner V1. ReplaySSM is selected through `--use-replayssm`. The regular KDA prefill and non-speculative decode paths are unchanged. ### Verify records FP32 corrections KDA verify reads the checkpoint without modifying it and computes the normal speculative outputs. For each speculative token it records: - the correction vector `u` in FP32; - the key and raw gate in the activation dtype. Recording `u` removes `v` and `beta` from the commit path and avoids recomputing the state-dependent correction. FP32 is used for `u` to preserve reconstruction accuracy; keys and gates remain BF16 in the evaluated configuration. ### Direct accepted-position reconstruction After verification determines the accepted length, one Triton kernel reconstructs the required checkpoint directly. For the KDA recurrence: ```text S_t = S_{t-1} * exp(g_t) + u_t outer k_t ``` The accepted state is: ```text S_p = S_0 * exp(sum(g_1 ... g_p)) + sum_j u_j outer (k_j * exp(sum(g_{j+1} ... g_p))) ``` The commit kernel evaluates this expression in reverse token order. It loads the original state once and does not recursively update or write intermediate states. The loop bound is the runtime accepted length rather than the complete eight-token DSpark activation window. In align mode, the same launch also reconstructs the next prefix-cache boundary when the accepted range crosses it. Convolution history is compacted to the accepted position and to the boundary using the same commit plan. ### Cache layout Each KDA page contains: ```text (convolution state, KDA checkpoint, FP32 correction records, key/gate records) ``` Replay records are transient and are not included in prefix-cache state copies. The checkpoint and convolution state remain the prefix-cache boundary state. ## Scope - NVIDIA Kimi-K3 KDA - Model Runner V2 - speculative decoding, validated with Kimi-K3-DSpark and seven draft tokens - `mamba_cache_mode=align` prefix caching This PR does not add a generic ReplaySSM interface to shared attention or model state classes, and it does not add Mamba2 support. ## Test Plan and Results ### Unit tests ```bash .venv/bin/python -m pytest tests/models/kimi_k3/test_kda.py -v ``` Result: `69 passed`. The 18 ReplaySSM cases cover: - the actual eight-token DSpark window; - partial and full acceptance lengths; - direct final-state and align-boundary reconstruction; - both convolution-state layouts; - request-index remapping; - gates with and without a lower bound; - cache shape and dtype derivation. ```bash .venv/bin/pre-commit run --files vllm/model_executor/layers/mamba/mamba_utils.py vllm/models/kimi_k3/nvidia/kda.py vllm/models/kimi_k3/nvidia/ops/replayssm.py tests/models/kimi_k3/test_kda.py ``` Result: all hooks passed. ```bash .venv/bin/pre-commit run mypy-3.12 --all-files --hook-stage manual ``` Result: passed for Python 3.12. ### End-to-end accuracy Controlled setup: - Kimi-K3 target and Kimi-K3-DSpark draft model, TP8 on two GB300 nodes; - seven greedy draft tokens and block rejection sampling; - align-mode Mamba cache, prefix caching, FP8 MLA cache, FlashInfer MLA, and fastsafetensors; - identical datasets, prompts, scorers, and serving parameters for ReplaySSM ON and OFF. | GSM8K, 1,319 samples | ReplaySSM ON | OFF | ON - OFF | |---|---:|---:|---:| | Flexible extract | 96.3609% | 96.1334% | +0.2274 pp | | Strict match | 96.0576% | 95.9060% | +0.1516 pp | The paired 95% confidence intervals were `[-0.4549, +0.9098]` pp for flexible extract and `[-0.6065, +0.9098]` pp for strict match. Both runs completed all 1,319 requests with zero request errors. Independently initialized greedy serving runs are not bitwise deterministic, so the paired score comparison is used instead of exact generated-text equality. | MRCR, 40 long prompts run cold and warm | ReplaySSM ON | OFF | |---|---:|---:| | Cold score | 0.74879 | 0.66791 | | Warm score | 0.74879 | 0.66791 | | Cold/warm exact-output consistency | 100% | 97.5% | | Warm prefix-cache token hit rate | 89.13% | 90.50% | The MRCR prompts ranged from 16,757 to 33,772 tokens and contained 991,216 prompt tokens per pass. ReplaySSM completed both passes without request errors. Total DSpark acceptance was 53.3514% with ReplaySSM and 53.4062% without it, a difference of -0.0548 percentage points. GSM8K-only acceptance was 50.0899% with ReplaySSM and 50.0336% without it. ### Memory capacity Both variants used a fixed 46.48 GiB cache-memory budget. | Metric | ReplaySSM ON | OFF | Difference | |---|---:|---:|---:| | Effective cache capacity | 2,856,170 tokens | 2,573,777 tokens | +10.97% | | Max 262k-request concurrency | 10.895x | 9.818x | +10.97% | ### Nsight Systems profile The profile used batch 32, 128 forced output tokens, and 80 captured decode steps per GPU on the same two-node TP8 setup. The aggregate results below include eight initial small-batch steps. | KDA work per decode step | Initial recurrent ReplaySSM commit | This PR | |---|---:|---:| | Verify | 3.115 ms | 3.540 ms | | Commit | 4.793 ms | 1.464 ms | | Commit auxiliaries | 0.081 ms | 0.081 ms | | Total | 7.989 ms | 5.085 ms | The direct reconstruction makes the commit kernel 3.27x faster and reduces total ReplaySSM KDA time by 36.34%. The FP32 correction write increases verify time by 13.66%, but commit time falls by 69.45%. Native non-ReplaySSM KDA took 2.772 ms per step in the corresponding OFF trace. The 72 steady batch-32 steps each verified 256 tokens. Their CUDA-graph model-forward GPU wall time was 65.208 ms with ReplaySSM and 64.873 ms without it, a 0.335 ms or 0.52% difference. Kernel residence exceeds wall time because kernels on concurrent streams overlap. | ReplaySSM ON steady forward kernel residence | Time per step | Share | |---|---:|---:| | DeepGEMM and nvJet GEMM | 38.760 ms | 56.5% | | MoE routing and packing | 8.121 ms | 11.8% | | SiTU activation | 7.682 ms | 11.2% | | TP collectives | 6.136 ms | 8.9% | | ReplaySSM verify | 3.791 ms | 5.5% | | KDA convolution and normalization | 0.938 ms | 1.4% | | MLA attention kernels | 0.519 ms | 0.8% | | Other kernels | 2.658 ms | 3.9% | | Total kernel residence | 68.603 ms | 100% | ReplaySSM verify runs inside model forward and replaces the native KDA recurrent kernel. Its steady kernel residence was 3.791 ms versus 3.009 ms for native KDA, an increase of 0.782 ms. The smaller 0.335 ms forward wall-time difference reflects kernel overlap and run-to-run variation. Commit runs after sampling determines the accepted length, so it is not part of model forward. Across the 72 steady batch-32 steps it took 1.711 ms per step and varied with the accepted length. | Post-sampling ReplaySSM commit | Time per step | Share | |---|---:|---:| | FP32 correction state reconstruction | 1.618 ms | 94.5% | | Convolution-state compaction | 0.081 ms | 4.7% | | Commit-plan preparation | 0.0069 ms | 0.4% | | Align bookkeeping | 0.0061 ms | 0.4% | | Total | 1.711 ms | 100% | Adding the steady model-forward difference and post-sampling commit gives an estimated ReplaySSM GPU critical-path cost of 2.046 ms per batch-32 decode step, before common sampling and postprocessing work. ## Duplicate-work check The following searches were run against open vLLM PRs: ```bash gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM Kimi K3\" gh pr list --repo vllm-project/vllm --state open --search \"KDA speculative decoding\" gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM in:title,body\" ``` No other open PR implements this Kimi-K3 KDA integration. Related work is materially different: - #49847 targets Mamba2; - #49887 targets Gated DeltaNet; - #50140 is a WIP FlashInfer ReplaySSM kernel; - #42792 adds generic Mamba align-prefix speculative support rather than Kimi-K3 KDA ReplaySSM. ## AI assistance AI assistance was used for implementation, tests, profiling analysis, model evaluation analysis, and this PR description. The submitting human must review every changed line, reproduce the relevant tests, and understand and defend the design end-to-end. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and exact commands. - [x] Unit, end-to-end accuracy, memory, acceptance, and performance results. - [x] Duplicate-work check and AI-assistance disclosure.
**BEFORE SUBMITTING, PLEASE READ **", "url": "https://github.com/vllm-project/vllm/pull/51855", "createdAt": "2026-08-11T17:49:54Z", "updatedAt": "2026-08-13T15:23:57Z", "timestamp": "2026-08-13T15:23:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "performance", "mrv2", "verified", "kimi", "k3" ], "author": "ZJY0516", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:38b1b2e68238d9f9aa07", "signalId": "github:vllm-project/vllm:pull_request:51338", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "title", "text", "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51338", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RFC][Bug Fix][Spec Decode] Require explicit speculative methods", "text": "This PR removes speculative-method guessing and makes the method an explicit contract. ## Problem On current `main`, vLLM guesses the speculative method from model path/name substrings such as `eagle3`, `dflash`, and `dspark`. A DFlash or DSpark checkpoint at a neutral training path can therefore be treated as the generic autoregressive `draft_model`, while renaming or moving the same checkpoint changes serving behavior. For example: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15}' ``` Previously, this could silently select `draft_model`. In the original end-to-end Qwen3-8B + DFlash reproduction, the misdetected path produced acceptance length 1.02; adding explicit `\"method\": \"dflash\"` produced 3.88. Serving a specialized checkpoint through the wrong proposer is worse than requiring one extra field, so this PR now fails closed. ## Design - A user-provided method is required for every explicit speculative configuration. It can be supplied as `\"method\"` in `--speculative-config` or with `--spec-method`. - The user-provided method always takes precedence. - vLLM no longer infers a method from a model path/name, checkpoint architecture, or structural `model_type`. - Known checkpoint schemas that explicitly declare their algorithm may populate the method before `SpeculativeConfig` is constructed. The speculators format uses `speculators_model_type`; PEagle is normalized to `method=\"eagle3\"` with parallel drafting. - `num_speculative_tokens` remains optional when the checkpoint declares a safe proposal depth, such as speculators `speculative_tokens` or `n_predict`. Otherwise the existing clear error remains. - Structural config normalization remains where it is required to load the selected implementation (including embedded DeepSeek-V4 DSpark), but it does not select the method. - DFlash and DSpark parallel-drafting behavior is derived from the explicit method. Explicit shorthand values are folded into the config before checkpoint-declared defaults, so `--spec-method`, `--spec-model`, and `--spec-tokens` follow the same precedence rule as JSON fields. ## Breaking change Configurations that relied on implicit selection must add a method. Before: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` After: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"method\": \"dflash\", \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` For a generic autoregressive draft model, use `\"method\": \"draft_model\"`. Legacy EAGLE/EAGLE3, Medusa, MLP-speculator, n-gram, and custom-proposer configurations must likewise state their method. A speculators-format checkpoint used as the top-level model still works without an explicit `--speculative-config`, because its schema declares both the algorithm and proposal depth. ## Validation - End-to-end reproduction recorded before this design change: - misdetected `draft_model`: acceptance length 1.02 - explicit `dflash`: acceptance length 3.88 - This PR now rejects the missing-method configuration instead of silently serving it autoregressively. - Focused explicit-method/schema/token tests: 10 passed. - Local config, DSpark, and engine-argument suites: 19 passed. - Scheduler and structured-output suites: 21 passed. - Full `tests/test_config.py`: 179 passed; 2 failed only because this environment cannot access gated `meta-llama/Meta-Llama-3-8B-Instruct` (HTTP 403). - All pre-commit hooks passed, including Ruff, mypy, Markdown lint, SPDX, DCO sign-off, and configuration validation. - GPU model evaluation was not rerun for this follow-up; the serving change is fail-fast configuration behavior, and the earlier DFlash measurement above is retained as the motivating model result.
Commands run ```bash .venv/bin/python -m pytest -q tests/test_config.py \\ -k 'speculative_config_requires_method or speculators_format or deepseek_v4_dspark_normalized_before_mtp or draft_sample_method' .venv/bin/python -m pytest -q \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/transformers_utils/test_dspark_mla_config.py \\ tests/v1/test_oracle.py .venv/bin/python -m pytest -q \\ tests/v1/core/test_scheduler.py -k 'priority or speculative' \\ tests/v1/spec_decode/test_mtp_structured_output.py \\ tests/v1/structured_output/test_backend_guidance.py .venv/bin/python -m pytest -q tests/test_config.py .venv/bin/pre-commit run --files ```
## Contribution notes - This is not a duplicate of #47490: that PR preserves auto-detection and validates explicit-method mismatches; this PR removes automatic method selection and fails when an explicit configuration omits the method. - This is not a duplicate of #44566: that PR only propagates token depth from a speculators-format draft; this PR changes method-selection semantics while preserving declared token defaults. - AI assistance was used. I reviewed every changed line and ran the validation listed above.", "url": "https://github.com/vllm-project/vllm/pull/51338", "createdAt": "2026-08-07T00:53:10Z", "updatedAt": "2026-08-13T15:23:00Z", "timestamp": "2026-08-13T15:23:00Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "documentation", "new-model", "structured-output", "speculative-decoding", "needs-rebase", "rust" ], "author": "WindChimeRan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f82c9f0f1b7072204294", "signalId": "github:vllm-project/vllm:pull_request:49209", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49209", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU", "text": "## Purpose This is the second (2/2) PR introducing batch invariance to Intel XPU devices. You can find the first one [here](https://github.com/vllm-project/vllm/pull/41934). This PR registers the missing matmul and linear kernels. It also adds a new matmul_kernel_descriptor_persistent Triton kernel for better matmul performance on XPU. ## Test Plan Tested with unit tests covering batch invariance on XPU. ## Test Result All tests pass. ### Note The code was co-developed with GitHub Copilot. ---
Matmul and linear kernels were skipped in the previous PR, because they required more work. I tested batch invariance in Reinforcement Learning context and there I hit an error, because backward kernels were not registered for the \"XPU\" dispatch key. Also, I wanted to introduce a new Triton matmul kernel based on Tensor Derscriptors, because that implementation has shown ~3x speedup on XPU vs. the one based on pointers.
", "url": "https://github.com/vllm-project/vllm/pull/49209", "createdAt": "2026-07-20T14:45:22Z", "updatedAt": "2026-08-13T15:22:06Z", "timestamp": "2026-08-13T15:22:06Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "documentation", "intel-gpu", "ready", "v1" ], "author": "tzielinski-habana", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:799939fc305f98914968", "signalId": "github:vllm-project/vllm:pull_request:50983", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50983", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve the audio duration-limit error through the PyAV fallback", "text": "## Purpose load_audio wraps every failure on its PyAV fallback path as \"Invalid or unsupported audio file.\" including the rejection raised by the VLLM_MAX_AUDIO_DECODE_DURATION_S guard itself. For containers soundfile cannot probe (like WebM), audio over the duration limit therefore reports a misleading invalid-file error, while soundfile-readable formats (mp3, ogg, flac) report the actionable duration message. This PR types the guard's rejection as AudioDurationExceededError and re-raises it ahead of the generic wrap, so only genuine decode failures are normalized. No behavior change for callers catching ValueError. ## Test Plan ``` pytest tests/multimodal/media/test_audio.py ``` New tests synthesize an opus WebM in memory to exercise the PyAV fallback while a garbage-bytes control verifies genuine decode failures keep the generic error. ## Test Result All 12 tests pass (Linux x86_64, CUDA torch). Without the fix, both new rejection tests fail: the duration error is masked as \"Invalid or unsupported audio file.\". With it, WebM over the limit reports the same actionable message as mp3.", "url": "https://github.com/vllm-project/vllm/pull/50983", "createdAt": "2026-08-04T06:33:26Z", "updatedAt": "2026-08-13T15:21:15Z", "timestamp": "2026-08-13T15:21:15Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "multi-modality" ], "author": "ieaves", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e11ae20ce99b18846666", "signalId": "github:vllm-project/vllm:pull_request:51403", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51403", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Route non-divisor small-head MLA decode to Gluon on gfx950", "text": "Fixes #51232. Kimi-K3 has 96 MLA query heads; at TP8 each rank gets 12 local heads. 12 is not a divisor of 16, so `use_gluon_decode` returned `False` in \"auto\" mode, routing single-token decode to the padded ASM persistent path. That path pads 12 heads to 16 and asks AITER for a `{bf16, fp8, gqa=16, qseqlen=1}` kernel, which does not exist on gfx950 — every rank crashes with `cannot get heuristic kernel` before the server starts. The Gluon MLA decode kernel handles any head count natively (no padding), so the fix is to prefer Gluon for ALL small-head counts on gfx950 in \"auto\" mode, not just divisors of 16. On gfx942 (no Gluon build) the ASM path is still used for all counts — no behavior change. `VLLM_ROCM_AITER_MLA_ASM_PADDING=asm` still forces the padded ASM path if needed. ### Changes - `rocm_aiter_mla.py`: remove the `m % num_heads == 0` gate from the \"auto\" return in `use_gluon_decode` - `test_rocm_aiter_mla_head_padding.py`: update 3 assertions so non-divisor counts (12, 6) assert `is on_gfx950` instead of `not`", "url": "https://github.com/vllm-project/vllm/pull/51403", "createdAt": "2026-08-07T14:38:14Z", "updatedAt": "2026-08-13T15:20:41Z", "timestamp": "2026-08-13T15:20:41Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "zvigrinberg", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:5fdade679f14804745bd", "signalId": "github:vllm-project/vllm:pull_request:52185", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52185", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Pixtral: use packed multimodal encoder attention", "text": "## Purpose Fixes #52180 Pixtral currently concatenates all image patch sequences in an encoder batch. When xFormers is unavailable, it constructs a dense block-diagonal mask and applies SDPA to the combined sequence, causing latency and memory use to scale poorly with the number of images. This PR routes both Pixtral vision implementations through vLLM's `MMEncoderAttention` with per-image cumulative sequence lengths. It: - preserves image isolation without constructing a dense combined-sequence mask; - honors the existing multimodal encoder backend selection for FlashAttention, FlashInfer, Torch SDPA, Triton attention, and ROCm AITER; - delegates backend-specific sequence metadata handling to `MMEncoderAttention`; and - removes the remaining Pixtral-specific direct xFormers path. There are no checkpoint, preprocessing, output-shape, token-count, or user-facing API changes. ## Test Plan ```bash pre-commit run --files \\ vllm/model_executor/models/pixtral.py \\ tests/models/multimodal/generation/test_pixtral.py pytest -q tests/models/multimodal/generation/test_pixtral.py \\ -k packed_sequence_metadata ``` GPU validation additionally covered: - Torch SDPA, FlashAttention, and FlashInfer metadata and forward paths; - mixed-size image isolation; - 24-layer encoder parity against Torch SDPA; - controlled before/after encoder benchmarks; and - full Mistral3 vLLM inference with two differently sized images. ## Test Result - Pre-commit hooks, Ruff, mypy, SPDX, and repository checks: passed. - Packed metadata tests: `3 passed`. - Full Mistral3 smoke test: passed; the requested FlashAttention vision backend was selected and generation completed. ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [X] The test plan, such as providing test command. - [X] The test results, such as pasting the results comparison before and after, or e2e results - [X] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52185", "createdAt": "2026-08-13T15:15:43Z", "updatedAt": "2026-08-13T15:20:02Z", "timestamp": "2026-08-13T15:20:02Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "multi-modality", "mistral" ], "author": "oliverholworthy", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:d7a21b3913ef90ce0d88", "signalId": "github:vllm-project/vllm:pull_request:51704", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec", "text": "## Purpose Part of the KV-cache layout standardization series (RFC #42082). **Stacked on #51612** — the diff shown includes it until that lands and this retargets `main`. Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants in MLA spec overrides. This PR makes specs plain data and moves each packed format to the backend/component that owns it: - `AttentionSpec` gains two optional packing fields `num_head_slots` (H in standard layout) and`state_content_bytes` (C in standard layout). Page sizes derive uniformly as `num_heads × storage_block_size × content`. > NOTE: i dont love the num_head_slots name but this should hopefully be temporary until we can [fully refactor the backends](https://github.com/vllm-project/vllm/issues/42449); i.e. separate allocation from connector and kv-cache manager concerns - `AttentionBackend.customize_spec(spec)` is a temporary hook until we can move to having the backend return the spec instead of the layer (see: https://github.com/vllm-project/vllm/issues/42449) Prepares the ground for the layout-standardization PR ([[6/N]](https://github.com/vllm-project/vllm/pull/51718)), where these fields describe the standardized `[B, H, N, C]` page and the AITER backends publish separate K/V head groups through the same hook. --- AI-assisted (Claude); reviewed by submitter", "url": "https://github.com/vllm-project/vllm/pull/51704", "createdAt": "2026-08-10T16:20:40Z", "updatedAt": "2026-08-13T15:19:37Z", "timestamp": "2026-08-13T15:19:37Z", "metrics": { "reactions": 2, "comments": 21 }, "labels": [ "rocm", "ready", "nvidia", "ready-run-all-tests", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0bd390239ffbeb51feae", "signalId": "github:vllm-project/vllm:issue:50699", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:50699", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: DiffusionGemma: runtime CUDA OOM under concurrent decode (fp32 canvas×vocab temporaries)", "text": "### Summary Block-diffusion decode produces `canvas_length` (default 256) logit rows **per request per step**, and `_compiled_sample_step` in `vllm/model_executor/models/diffusion_gemma.py` materializes ~6 fp32 temporaries of shape `[num_decode, canvas_length, vocab_size=262144]`. At `--max-num-seqs 8` each of these tensors is 2.1 GiB (>12 GiB total transient). Startup/warmup passes (KV pool still empty), but at runtime the KV pool grows to fill the memory budget, leaving <1 GiB physical headroom → the engine **dies with CUDA OOM on the first full batch**. This is diffusion-specific: autoregressive decode has 1 logit row per request, so this transient never scales with concurrency there. ### Environment - vLLM `0.22.1rc1.dev357+g74b5964f0` (docker `vllm/vllm-openai:gemma`); the same pattern is present in current `main` - GPU: RTX 5090 32GB, driver 610.43.02, torch 2.11.0+cu130, CUDA 13.3 - Model: `RedHatAI/diffusiongemma-26B-A4B-it-NVFP4` (~17 GiB weights) ### Reproduction ```bash vllm serve RedHatAI/diffusiongemma-26B-A4B-it-NVFP4 \\ --max-num-seqs 8 --max-model-len 131072 \\ --gpu-memory-utilization 0.75 --kv-cache-dtype fp8 \\ --generation-config vllm \\ --hf-overrides '{\"diffusion_sampler\":\"entropy_bound\",\"diffusion_entropy_bound\":0.1}' \\ --diffusion-config '{\"canvas_length\":256}' ``` Then send 8 concurrent chat requests (`max_tokens=512`). ### Observed ``` torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 31.40 GiB of which 1.01 GiB is free. File \".../vllm/model_executor/models/diffusion_gemma.py\", line 525, in _compiled_sample_step ``` Memory math at `num_decode=8, CL=256, vocab=262144`: one fp32 tensor = `8×256×262144×4 B` = 2.1 GiB; the compiled step keeps several alive simultaneously (`logits_3d.float()`, `scaled`, gumbel noise, `log_probs`, `probs`, entropy product, plus the fp32 `scaled` it returns). ### Local workaround (validated) Chunking decode requests through `_compiled_sample_step` (e.g. 2 requests/call) fixes it: all state tensors are slot-indexed, so per-chunk in-place updates are equivalent to one call; only the zeroing of `sampled`/`num_sampled` has to move out of the per-chunk call. Combined with row-chunked `compute_logits` (preallocated output buffer), `--max-num-seqs 16` survives repeated 16-way concurrent load tests (0 OOM), with unchanged outputs. Happy to clean this up into a PR if the approach sounds reasonable. --- *Disclosure: the investigation, patch and this write-up were done with AI assistance (a local coding agent), verified with real load tests on the hardware above.*", "url": "https://github.com/vllm-project/vllm/issues/50699", "createdAt": "2026-08-01T20:27:40Z", "updatedAt": "2026-08-13T15:19:02Z", "timestamp": "2026-08-13T15:19:02Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [], "author": "pdos95", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:22d865e2f2ade93b1b1b", "signalId": "github:vllm-project/vllm:pull_request:49171", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Skip logits and sampling for unfinished prefills", "text": "## Purpose Model Runner V2 currently produces one sampling-logit row for every request in the common non-speculative path, including chunked-prefill requests whose prefill does not finish in the current step. Although these sampled tokens are discarded later by setting `num_sampled` to zero, the model has already performed the vocabulary projection and the sampler has already applied logits processors and sampling kernels. This introduces unnecessary GPU work, especially for workloads with multiple concurrent long prompts and a small `max_num_batched_tokens`. This PR avoids that work by: - Determining whether each request finishes prefill before constructing the logits batch. - Assigning zero logits rows to unfinished chunked-prefill requests. - Compacting mixed prefill/decode batches so output projection and sampling run only for requests that can produce an output token. - Short-circuiting `compute_logits` and sampling when the entire batch consists of unfinished prefill chunks. - Expanding compact sampled-token and NaN-count results back to the original per-request layout with a small Triton kernel, preserving existing downstream interfaces. - Preserving the existing pure-decode fast path. The change is limited to Model Runner V2's common non-speculative path. Speculative decoding remains unchanged. LoRA batches retain the existing dense logits layout because the per-request mapping is reused by prompt-logprob projection. This should reduce unnecessary vocabulary-projection and sampling work for chunked-prefill-heavy serving workloads without changing generated outputs. ## Test ### start server ``` VLLM_USE_V2_MODEL_RUNNER=1 \\ vllm serve Qwen3.5-4B \\ --served-model-name Qwen3.5-4B \\ --host 0.0.0.0 \\ --port 8000 \\ --enable-chunked-prefill \\ --max-num-batched-tokens 2048 \\ --max-num-seqs 32 \\ --gpu-memory-utilization 0.90 \\ --disable-log-stats ``` ### vllm bench ``` vllm bench serve \\ --backend vllm \\ --host 127.0.0.1 \\ --port 8000 \\ --endpoint /v1/completions \\ --model Qwen3.5-4B \\ --tokenizer Qwen3.5-4B \\ --dataset-name random \\ --random-input-len 8192 \\ --random-output-len 32 \\ --random-range-ratio 0 \\ --ignore-eos \\ --request-rate inf \\ --max-concurrency 32 \\ --num-warmups 16 \\ --num-prompts 256 \\ --seed 0 \\ --percentile-metrics ttft,tpot,itl,e2el \\ --metric-percentiles 50,90,99 ``` ## Test Result ### before \"IMG_3103\" ### after \"IMG_3104\" - Output token throughput increased from **174.72 tok/s** to **191.97 tok/s**, an improvement of **9.87%**. - Mean TTFT decreased from **4311.56 ms** to **3904.83 ms**, a reduction of **9.43%**. - Mean TPOT decreased from **43.97 ms** to **40.15 ms**, a reduction of **8.69%**. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49171", "createdAt": "2026-07-20T08:40:18Z", "updatedAt": "2026-08-13T15:16:32Z", "timestamp": "2026-08-13T15:16:32Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "multi-modality", "nvidia", "mrv2", "rust" ], "author": "jesse996", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a305afdec0ef25c55cd9", "signalId": "github:vllm-project/vllm:pull_request:50685", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50685", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Keep Qwen3Next layer boundaries sequence parallel", "text": " ## Purpose Fixes #50681. Qwen3.6-35B-A3B produces corrupted output during single-token decode when expert parallelism and MoE sequence parallelism are enabled with DP2 and TP2. The Qwen3Next model path infers the hidden-state layout from the first tensor dimension, but during TP2 single-token decode the full input and each padded local shard can all have one row. Shape alone therefore cannot determine whether attention must first all-gather the input. This PR gives supported all-MoE Qwen3Next/Qwen3.5 models a fixed model-wide sequence-parallel contract, following the same structure used by Kimi K3: - shard once before entering the decoder stack; - all-gather before attention and reduce-scatter its output in every layer; - keep every decoder-layer boundary sequence parallel; and - gather once after the final norm. There is no runtime layout state to infer or pass through decoder-layer arguments. Qwen3Next configurations that mix dense and MoE layers retain the existing layer-local MoE sequence-parallel path instead of switching layouts inside the model loop. Qwen3.5 MoE inherits the shared model forward and decoder forward implementations, so no duplicate change is needed in `qwen3_5.py`. The MTP implementations call decoder layers directly and bypass the shared model forward. They therefore shard before that direct call and gather after the MTP norm to honor the decoder's fixed contract. This is compatibility with the new contract, not a claim that MTP already exhibited #50681 on `main`. No open PR references #50681, and searches for Qwen3Next single-token sequence-parallel all-gather fixes found no duplicate work. I used AI assistance during the analysis and implementation of this change. I reviewed every changed line and verified the results reported below. ## Test Plan Use `Qwen/Qwen3.6-35B-A3B` in BF16 with DP2, TP2, EP4, eager execution, greedy decoding, and the default `allgather_reducescatter` all-to-all backend: ```bash vllm serve Qwen/Qwen3.6-35B-A3B \\ --tensor-parallel-size 2 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --enforce-eager \\ --max-model-len 512 \\ --max-num-seqs 4 \\ --gpu-memory-utilization 0.80 ``` Send a deterministic 64-token chat-completions request for `Explain quantum computing in simple terms:` with thinking disabled. Run code checks only on the three changed files: ```bash pre-commit run --files \\ vllm/model_executor/models/qwen3_next.py \\ vllm/model_executor/models/qwen3_next_mtp.py \\ vllm/model_executor/models/qwen3_5_mtp.py git diff --check ``` ## Test Result The four-GPU run confirmed the intended topology and backend: ```text world_size=4 [EP Rank 0/4] Expert parallelism is enabled. Using AgRsAll2AllManager all2all manager. ``` On current `main` (`89c8401c8a`), the non-MTP request reproduced corrupted and repetitive output: ```text To **E****E****E******** **Quant2** **Quant2** Quant2020** ``` With this change, the same request returned coherent output: ```text To understand quantum computing, it helps to first look at how our current computers work, and then see how quantum computers break the rules. ``` For scope validation, current `main` was also run with real Qwen3.6 MTP using one and two speculative tokens. Both produced coherent output. With two speculative tokens, 34 of 58 draft tokens were accepted, confirming that the draft path was active rather than entirely masked by target rejection. The MTP file changes in this PR are therefore contract adaptation, not a separate pre-existing output-corruption fix. All pre-commit hooks for the three changed files passed, including Ruff and mypy, and `git diff --check` passed. No test files were added. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] Before-and-after E2E results. - [x] AI assistance disclosure.
", "url": "https://github.com/vllm-project/vllm/pull/50685", "createdAt": "2026-08-01T13:18:59Z", "updatedAt": "2026-08-13T15:16:15Z", "timestamp": "2026-08-13T15:16:15Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "bug", "ready", "qwen" ], "author": "kzwrime", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:42703e9b413ece95a33b", "signalId": "github:vllm-project/vllm:issue:47691", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:47691", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: `--data-parallel-start-rank 0` is silently treated as unset due to truthiness checks in `create_engine_config`", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... uv is set ============================== System Info ============================== OS : Linux Mint 22.1 (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu128 Is debug build : False CUDA used to build PyTorch : 12.8 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime) Python platform : Linux-6.8.0-134-generic-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 12.0.140 GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3050 Laptop GPU Nvidia driver version : 580.159.03 ============================== vLLM Info ============================== vLLM Version : 0.1.dev18413+g6569df6a3 (git sha: 6569df6a3) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled ```
Note: this bug is in pure Python config-derivation logic (`vllm/engine/arg_utils.py`) and is independent of GPU/driver/topology. It affects any multi-node **hybrid data-parallel load balancing** deployment (`docs/serving/data_parallel_deployment.md`, \"Hybrid Load Balancing\" section), regardless of hardware. ### 🐛 Describe the bug **Summary:** When a node explicitly passes `--data-parallel-start-rank 0`, `EngineArgs.create_engine_config` treats it identically to \"not specified,\" because the field is checked with Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None`. Since `0` is a perfectly valid and meaningful starting rank (the node owning the first slice of ranks), this silently corrupts the derived `data_parallel_hybrid_lb` flag for exactly that node. #### The two buggy lines `vllm/engine/arg_utils.py`, inside `create_engine_config`: ```python # line 1997 if self.data_parallel_start_rank and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True ``` ```python # line 2016 self.data_parallel_rank = ( self.data_parallel_start_rank or inferred_data_parallel_rank ) ``` In both cases, `self.data_parallel_start_rank == 0` is falsy, so: - Line 1997: the condition is `0 and not headless` → `False`, so `self.data_parallel_hybrid_lb` is **never inferred `True`** for this node, even though the deployment doc explicitly requires hybrid LB nodes to pass `--data-parallel-start-rank` so \"each node knows which ranks it owns\" (`docs/serving/data_parallel_deployment.md`, lines 87-91) — including the node that owns rank 0. - Line 2016: `0 or inferred_data_parallel_rank` evaluates to `inferred_data_parallel_rank`. This usually happens to still resolve to `0` in the common single-process-per-node case, but it means the explicit user-supplied value is discarded in favor of a derived one, which is fragile and wrong in intent even when the numeric result coincides. #### The correct pattern already exists 27 lines above, for the sibling field ```python # line 1969 — data_parallel_rank, handled correctly data_parallel_external_lb = ( self.data_parallel_external_lb or self.data_parallel_rank is not None ) ``` This is the established, correct convention in this exact function for exactly this class of field (`int | None`, where `0` is a valid value). The two other places in the codebase that read `data_parallel_start_rank` also use `is not None` correctly: - `vllm/entrypoints/cli/serve.py:86`: `is_hybrid_lb = args.data_parallel_start_rank is not None` - `vllm/entrypoints/openai/dp_supervisor.py:39-40`: `if start_rank is not None: return start_rank` Neither of those call sites compensates for the bug, though — `serve.py`'s `is_hybrid_lb` is a local variable used only for `--api-server-count` defaulting and LB-mode mutual-exclusivity checks; it is never written back to `args.data_parallel_hybrid_lb`. `dp_supervisor.py` is a separate code path (multi-port external LB) that never touches `data_parallel_hybrid_lb`. So `arg_utils.py:1997`/`2016` are the only place that derive `self.data_parallel_hybrid_lb` from this flag, and they get it wrong specifically for rank 0. #### Downstream impact chain The corrupted `data_parallel_hybrid_lb` flag propagates into `ParallelConfig.data_parallel_hybrid_lb` and is consumed in multiple places: 1. `vllm/config/parallel.py:536`, `ParallelConfig.local_engines_only`: ```python return self.data_parallel_external_lb or self.data_parallel_hybrid_lb ``` For the rank-0 node in a hybrid-LB deployment, this wrongly evaluates to `False` instead of `True`. 2. `vllm/v1/engine/utils.py:1244`, `wait_for_engine_startup`: ```python remote_should_be_headless = ( not parallel_config.data_parallel_hybrid_lb and not parallel_config.data_parallel_external_lb ) ``` Wrongly evaluates to `True` for that node, misclassifying the expected startup behavior of remote engines. 3. `vllm/entrypoints/cli/serve.py`, in the Rust-frontend startup path added by #45805 (merged 2026-06-17): ```python if parallel_config.local_engines_only: expected_engine_start_index = parallel_config.data_parallel_rank expected_engine_count = parallel_config.data_parallel_size_local else: expected_engine_start_index = 0 expected_engine_count = parallel_config.data_parallel_size ``` Because `local_engines_only` is wrongly `False` for the rank-0 node, the Rust frontend computes it as owning **all** global engines (`0`, `data_parallel_size`) instead of just its local slice (`data_parallel_rank`, `data_parallel_size_local`). None of these downstream consumers compensate for or work around the truthiness bug — they all trust `data_parallel_hybrid_lb` as computed in `arg_utils.py`. #### Steps to reproduce Set up a 2-node hybrid load-balancing deployment per `docs/serving/data_parallel_deployment.md` (\"Hybrid Load Balancing\"): ```bash # Node 0 (owns global ranks 0-1, i.e. the first slice) — non-headless, exposes its own API endpoint vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \\ --data-parallel-start-rank 0 \\ --data-parallel-address 192.0.2.1 --data-parallel-rpc-port 13345 # Node 1 (owns global ranks 2-3) vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 \\ --data-parallel-start-rank 2 \\ --data-parallel-address 192.0.2.1 --data-parallel-rpc-port 13345 ``` Node 0 (rank 0) ends up misclassified as not participating in hybrid LB while every other node in the same deployment is classified correctly — see the minimal reproduction below for the exact config values this produces. #### Minimal reproduction (no cluster required) The same underlying bug can be triggered directly against `EngineArgs.create_engine_config`, without standing up an actual multi-node deployment: ```python from vllm.engine.arg_utils import EngineArgs from vllm.usage.usage_lib import UsageContext engine_args = EngineArgs( model=\"facebook/opt-125m\", data_parallel_size=4, data_parallel_size_local=2, data_parallel_start_rank=0, ) vllm_config = engine_args.create_engine_config(UsageContext.OPENAI_API_SERVER) print(\"data_parallel_hybrid_lb:\", vllm_config.parallel_config.data_parallel_hybrid_lb) print(\"data_parallel_rank:\", vllm_config.parallel_config.data_parallel_rank) ``` Actual output on current `main` (commit `6569df6a3`): ``` data_parallel_hybrid_lb: False data_parallel_rank: 0 ``` Expected output (and what this snippet produces once `data_parallel_start_rank` is checked with `is not None` instead of truthiness): ``` data_parallel_hybrid_lb: True data_parallel_rank: 0 ``` ### Proposed fix Use `is not None` instead of truthiness at both sites, matching the existing convention used for the sibling field `data_parallel_rank` (line 1969) and the other two call sites of this same field (`serve.py:86`, `dp_supervisor.py:40`): ```python # line 1997 if self.data_parallel_start_rank is not None and not headless: self.data_parallel_hybrid_lb = True ``` ```python # line 2016 self.data_parallel_rank = ( self.data_parallel_start_rank if self.data_parallel_start_rank is not None else inferred_data_parallel_rank ) ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/47691", "createdAt": "2026-07-06T05:39:54Z", "updatedAt": "2026-08-13T15:12:28Z", "timestamp": "2026-08-13T15:12:28Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "bug" ], "author": "syedalijaseem", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:f30a679cd6702e3aa5c2", "signalId": "github:vllm-project/vllm:pull_request:47692", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:47692", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix `--data-parallel-start-rank 0` being treated as unset in `create_engine_config`", "text": "## Summary `EngineArgs.create_engine_config` uses Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None` to detect whether `--data-parallel-start-rank` was explicitly set. Since `0` is a valid, meaningful starting rank (the node owning the first slice of global DP ranks), an explicit `--data-parallel-start-rank 0` is silently treated identically to \"not specified.\" This causes `data_parallel_hybrid_lb` to never be inferred `True` for that node, even though the rest of the codebase (including the sibling field `data_parallel_rank`, and the other two call sites of this same field) already use `is not None` correctly for exactly this class of field. The corrupted flag then propagates into `ParallelConfig.local_engines_only`, `wait_for_engine_startup`'s `remote_should_be_headless` computation, and the Rust-frontend engine-indexing logic added in #45805 — misclassifying the rank-0 node's role in a hybrid-LB deployment. The fix switches both truthiness checks to `is not None`, matching the established convention already used elsewhere in the same function. ## Duplicate-work check Searched for existing/overlapping work before starting: - `gh issue list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh issue list --repo vllm-project/vllm --state open --search \"data parallel rank\"` — 30 hits, none related (skimmed all titles: Ray deadlocks, MTP, PP tool-calling, RPC port robustness, etc.) - `gh issue list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank\"` — 0 relevant hits - `gh pr list --repo vllm-project/vllm --state open --search \"data_parallel_start_rank\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state open --search \"DP rank 0 hybrid\"` — 26 hits, none touch this logic - `gh pr list --repo vllm-project/vllm --state closed --search \"data_parallel_start_rank closed:>=2026-06-05\"` — 0 hits - `gh pr list --repo vllm-project/vllm --state closed --search \"hybrid_lb closed:>=2026-06-05\"` — surfaced PR #45805 (merged 2026-06-17), which _consumes_ the affected `data_parallel_hybrid_lb`/`local_engines_only` flag in `serve.py` for Rust-frontend engine indexing but does not touch or fix `arg_utils.py:1997`/`2016` Zero overlapping in-flight work found across open issues, open PRs, and PRs closed in the last 30 days. ## Changes ### `vllm/engine/arg_utils.py` Line 1997, inside `create_engine_config`: ```python # Before if self.data_parallel_start_rank and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True # After if self.data_parallel_start_rank is not None and not headless: # Infer hybrid LB mode. self.data_parallel_hybrid_lb = True ``` Line 2016: ```python # Before self.data_parallel_rank = ( self.data_parallel_start_rank or inferred_data_parallel_rank ) # After self.data_parallel_rank = ( self.data_parallel_start_rank if self.data_parallel_start_rank is not None else inferred_data_parallel_rank ) ``` ### `tests/v1/engine/test_engine_args.py` Added `test_data_parallel_start_rank_zero_infers_hybrid_lb`, asserting that `data_parallel_start_rank=0` with `data_parallel_size_local=2`, `data_parallel_size=4` (headless defaults to `False`) yields `parallel_config.data_parallel_hybrid_lb is True` and `parallel_config.data_parallel_rank == 0`. Confirmed this test **fails** against the pre-fix code (verified via `git stash` on `vllm/engine/arg_utils.py` and re-running just this test) and **passes** with the fix applied — it is a genuine regression test, not a tautology. ## Test commands run and results ```bash python -m pytest tests/v1/engine/test_engine_args.py -v # 5 passed (including the new test) python -m pytest tests/engine/test_arg_utils.py tests/entrypoints/openai/test_dp_supervisor.py -v # 111 passed, no regressions pre-commit run --files vllm/engine/arg_utils.py tests/v1/engine/test_engine_args.py # ruff check: Passed # ruff format: Passed # typos: Passed # mypy (3.10): Passed # Check SPDX headers: Passed # Check root lazy imports: Passed # Check for forbidden imports: Passed # Validate configuration has default values / docstrings: Passed # (all other applicable hooks: Passed; unrelated hooks skipped — no matching files) ``` ## AI assistance disclosure This PR includes AI-assisted contributions. Claude Code was used to trace the control flow of `create_engine_config`, identify the truthiness-vs-`is not None` inconsistency, draft the fix, and draft the regression test. The human contributor independently: - Read the full `create_engine_config` function end-to-end (not just the two changed lines) to confirm no earlier code path coerces or transforms `data_parallel_start_rank` before it reaches the affected checks - Verified the sibling-field convention (`data_parallel_rank`, handled correctly with `is not None`) and the two other call sites of `data_parallel_start_rank` elsewhere in the codebase (`serve.py`, `dp_supervisor.py`), confirming both already use the correct pattern - Traced the full downstream consumer chain (`ParallelConfig.local_engines_only` → `wait_for_engine_startup`'s `remote_should_be_headless` → the Rust-frontend engine-indexing logic added in #45805) to confirm the bug has live, current consequences - Confirmed the added regression test fails on the pre-fix code and passes on the fix by manually stashing/restoring the source change and re-running the test - Ran the full test suite and pre-commit hooks locally and reviewed all output above Closes #47691", "url": "https://github.com/vllm-project/vllm/pull/47692", "createdAt": "2026-07-06T05:45:20Z", "updatedAt": "2026-08-13T15:12:27Z", "timestamp": "2026-08-13T15:12:27Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "bug", "ready", "v1" ], "author": "syedalijaseem", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:35978be42e7656e2389a", "signalId": "github:vllm-project/vllm:pull_request:52131", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52131", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Move api_server.py out openai folder", "text": "## Purpose Following https://github.com/vllm-project/vllm/pull/41907 > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. We'd better move api_server.py out openai folder ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52131", "createdAt": "2026-08-13T08:39:15Z", "updatedAt": "2026-08-13T15:12:24Z", "timestamp": "2026-08-13T15:12:24Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "documentation", "performance", "frontend", "multi-modality" ], "author": "noooop", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f351d2c79d24dea7dff7", "signalId": "github:vllm-project/vllm:pull_request:49688", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49688", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs", "text": "## Purpose CPU GDN attention (Qwen3.5) selects its causal-conv1d implementation via `torch.cpu._is_amx_tile_supported()`: AMX CPUs (Intel GNR) use the C++ kernels (`causal_conv1d_fwd_cpu` / `causal_conv1d_update_cpu`), everything else falls back to `causal_conv1d_fn_cpu` / `causal_conv1d_update_torch`, a pure-PyTorch per-sequence loop. The C++ conv kernels (`csrc/cpu/sgl-kernels/conv.cpp`) do not use AMX tiles but `is_amx` gate blocks it from running on non-AMX platforms that support AVX-512BF16. The float32 SSM state(currently gated by `is_amx` is required for entire CPU GDN path, not just AMX This PR fixes this gating logic. ## Test plan Extended `tests/kernels/mamba/cpu/test_cpu_gdn_ops.py` (all gated on `_is_avx512_bf16_supported()`, so they run on non-AMX CPUs too). Expected Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. ```bash python -m pytest tests/kernels/mamba/cpu/test_cpu_gdn_ops.py -v ``` Qwen-3.5 workload run before and after the patch on both Intel GNR and AMD EPYC Turin ``` VLLM_CPU_OMP_THREADS_BIND=0-63 VLLM_CPU_KVCACHE_SPACE=40 vllm bench throughput \\ --model Qwen/Qwen3.5-9B \\ --trust-remote-code \\ --dataset-name random \\ --input-len 1024 \\ --output-len 1024 \\ --num-prompts 128 \\ --max-num-seqs 32 \\ --dtype bfloat16 ``` Lmeval ``` VLLM_CPU_OMP_THREADS_BIND=0-63 \\ VLLM_CPU_KVCACHE_SPACE=40 \\ lm_eval \\ --model vllm \\ --model_args \"pretrained=Qwen/Qwen3.5-9B,dtype=bfloat16,trust_remote_code=True,max_model_len=4096\" \\ --tasks gsm8k \\ --num_fewshot 5 \\ --batch_size auto ``` ## Test result Python GDN test Result: 105 passed, 0 skipped on both Intel GNR and AMD Turin. Workload Qwen3.5-9B, BF16, ISL/OSL 1024/1024, 64-core instance (median of 3 runs) Intel GNR is unchanged (it takes the identical path both ways, so no regression). AMD EPYC Turin gains about 9-10% on this memory-bound, decode-heavy shape, which is the hardest case for a conv speedup to show up. ## Correctness `lm_eval` gsm8k, 5-shot, full set (n=1319), Qwen3.5-9B BF16, before vs after: | Platform | Metric | Baseline | After | |----------|--------|---------:|------:| | Intel GNR (AMX) | exact_match | identical | identical | | AMD Turin (Zen5) | flexible-extract | 0.8696 +/- 0.0093 | 0.8749 +/- 0.0091 | | AMD Turin (Zen5) | strict-match | 0.8734 +/- 0.0092 | 0.8779 +/- 0.0090 | GNR is bit-identical (same code path both ways). On Turin, before/after differ by less than one standard error, so there is no accuracy regression. The C++ conv accumulates in fp32 and is numerically no less accurate than the torch fallback(both round to bf16 at store), which the unit tests also assert against an fp32 oracle. ## Duplicate Check Searched open PRs/issues for GDN / causal_conv1d / Zen / AVX-512BF16 CPU conv work; found none addressing the non-AMX conv fallback. This PR Fixes #49640", "url": "https://github.com/vllm-project/vllm/pull/49688", "createdAt": "2026-07-24T07:35:29Z", "updatedAt": "2026-08-13T15:10:57Z", "timestamp": "2026-08-13T15:10:57Z", "metrics": { "reactions": 3, "comments": 6 }, "labels": [ "bug", "cpu" ], "author": "dineshchitlangia", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9571250c9441f89d2fc7", "signalId": "github:vllm-project/vllm:pull_request:50174", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50174", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup", "text": "## Description This PR extends the shared JIT warmup infrastructure with provider registration and centralized orchestration. It builds on #49315 and the contract described in #47456. For more details, see parent (draft) PR: https://github.com/vllm-project/vllm/pull/49627 and tracking list issue https://github.com/vllm-project/vllm/issues/49349 ``` JIT kernel warmup (5 compile keys): 100%|██████████████████████████████████| 1/1 [00:00<00:00, 259.93kernel/s, ComputeSlotMappingKernel (5 keys)] (Worker_TP0 pid=425015) INFO 08-11 15:43:58 [kernel_warmup.py:120] JIT kernel warmup finished in 0.01s. ``` ## Motivation - Discover warmup providers from the model and backend objects actually instantiated. - Avoid model-name-based lists and warming kernels for inactive backends. - Run compilation under `kernel_warmup()` logging, ordering, progress reporting, and exception handling. - Respect `enable_jit_warmup`. - Keep model construction and runtime execution separate from startup compilation. ## What Changed - Added a scoped registry used while model and KV-cache components are constructed. - Added centralized provider deduplication and warmup execution. - Added kernel and compile-key progress reporting. - Added shared Triton helpers and an initial block-table migration. - Added contributor documentation for the warmup contract. - Added registry, orchestration, and AST-expansion tests.", "url": "https://github.com/vllm-project/vllm/pull/50174", "createdAt": "2026-07-28T19:24:41Z", "updatedAt": "2026-08-13T15:10:36Z", "timestamp": "2026-08-13T15:10:36Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "v1", "cpu", "mrv2" ], "author": "LopezCastroRoberto", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f73add1175db8dc653cb", "signalId": "github:vllm-project/vllm:pull_request:52147", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52147", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Standardise weight tying on `ParallelLMHead.tie_weights`", "text": "## Purpose vLLM expresses tied word embeddings in three different ways. Only one of them, `self.lm_head = self.lm_head.tie_weights(embed_tokens)`, dispatches through `quant_method.tie_weights`. This PR converts the other two so tying is expressed one way everywhere. - `self.lm_head.weight = embed_tokens.weight` (33 sites) bypassed the quant method entirely, so it was wrong for quant methods that repack. The `ParallelLMHead` already exists at that point, so this is a one-line swap with no structural change. - `self.lm_head = embed_tokens` (23 sites) built no `ParallelLMHead` at all in the tied branch. These now always build one and tie it. Two consequences of the second bullet are worth review: 1. A transient vocab x hidden allocation before the head is tied. 2. The quant config is now consulted for the `lm_head` prefix where it previously never was. Five models hardcode tying with `assert config.tie_word_embeddings` (`gemma`, `gemma2`, `commandr`, `cohere2_moe`, `mpt`) and have no real `lm_head` to build. They are left alone here. ## Relationship to #51665 This is not a duplicate. It is the tying-standardisation half of my own #51665, split out so it can be reviewed on its own. Everything else in that PR is deliberately excluded: - `ModelConfig.maybe_untie_word_embeddings` and the checkpoint-metadata plumbing. - The `AutoWeightsLoader` aliased-parameter skip and the ~50 hand-rolled `skip_prefixes=[\"lm_head.\"]` removals that depend on it. **Every existing skip is left untouched here**, so weight-loading behaviour is unchanged. - The `assert config.tie_word_embeddings` removals, which only exist because #51665 can untie at config time. I searched open PRs for `tie_weights`, `tie_word_embeddings` and `weight tying`; nothing else overlaps. ## Test Plan Structural change with no new behaviour, so this leans on construction and load-path checks plus existing suites. 1. Construct tied models on CPU and assert `lm_head` is a `ParallelLMHead` whose `weight` is the same `nn.Parameter` object as the embedding. Covers both conversion forms. 2. Load a real checkpoint through `load_weights` and confirm the tied head ends up populated. 3. `pre-commit run --all-files` on the changed files (ruff, ruff-format, mypy). ## Test Result Models constructed on CPU (Bloom, OPT and Mamba are all conversions of the second, structural form): ``` Qwen/Qwen3-0.6B: tie=True ParallelLMHead=True shared=True -> OK Qwen/Qwen2-0.5B: tie=True ParallelLMHead=True shared=True -> OK facebook/opt-125m: tie=True ParallelLMHead=True shared=True -> OK state-spaces/mamba-130m-hf: tie=True ParallelLMHead=True shared=True -> OK bigscience/bloom-560m: tie=True ParallelLMHead=True shared=True -> OK ``` End-to-end `load_weights` with the real `Qwen/Qwen3-0.6B` checkpoint: ``` checkpoint has lm_head.weight: True num loaded params: 226 lm_head.weight in loaded: False # still skipped by the existing skip_prefixes shared storage: True matches checkpoint: True head is nonzero: True ``` Existing suite touching the quantised tied-head path: ``` $ pytest tests/model_executor/test_qwen3_5_quantization.py -q 2 passed, 14 warnings in 0.65s ``` Linters: ``` $ pre-commit run --all-files # on the changed files ruff check ... Passed ruff format ... Passed Run mypy for Python 3.10 ... Passed ``` No model evals yet, they need a GPU box. Happy to run whatever the reviewers consider load-bearing here. ## Note for reviewers In `ernie45_moe.py` and `ernie45_vl_moe.py` the tie runs outside the `is_last_rank` check, so with PP > 1 `self.lm_head` is a `PPMissingLayer`, which has no `tie_weights`. That path already raised `AttributeError` on `PPMissingLayer.weight` before this change, so it is not a regression, but the failure mode changes. Worth fixing separately. --- AI assistance was used to produce this change. I have reviewed every changed line and stand behind it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)", "url": "https://github.com/vllm-project/vllm/pull/52147", "createdAt": "2026-08-13T10:12:28Z", "updatedAt": "2026-08-13T15:10:11Z", "timestamp": "2026-08-13T15:10:11Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "qwen", "mistral" ], "author": "hmellor", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:0da0f4f1dba4bbb0d7b2", "signalId": "github:vllm-project/vllm:issue:52167", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52167", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: Extended online quantization roadmap", "text": "### Motivation. Continuation of https://github.com/vllm-project/vllm/issues/32029. Online quantization API is now relatively stable at https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online and documented at https://docs.vllm.ai/en/stable/features/quantization/online/. This RFC summarizes and motivates proposed features that would be helpful to make online quantization to be more flexible and more widely compatible with models, including pre-quantized ones. ### Proposed Change. ### 1. Support activation quant key override Implemented in https://github.com/vllm-project/vllm/pull/51808. This is necessary to dispatch on specific linear/moe backends that can be picked only through the `activation_quant_key` (see e.g. https://github.com/vllm-project/vllm/blob/c4e969294ecab9ffefb995b734303f13f62b723f/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L441-L445) ### 2. Targeted online quantization configuration based on user patterns Implemented in https://github.com/vllm-project/vllm/pull/51285, RFC in https://github.com/vllm-project/vllm/issues/50281. The goal is simply to extend the UI to support e.g. `--quantization-config.targets '{\"re:.*o_proj.*\":\"mxfp8\"}'`, e.g. to target a single layer similar to e.g. https://github.com/vllm-project/vllm/blob/dc9f845ddc54c1df38fdbce5afe03f9fd15813bd/vllm/model_executor/layers/attention/mla_attention.py#L935-L948 (that is currently hard-coded and may be causing accuracy issues if done silently https://github.com/vllm-project/vllm/pull/48051), or e.g. `--quantization-config.targets '{\"re:.*shared_expert.*\":\"fp8_per_block\"}'` to enable fused shared expert. ### 3. Support online quantization with partially pre-quantized checkpoints Implemented in https://github.com/vllm-project/vllm/pull/51392. This is necessary to combine online quantization with already partially quantized checkpoints, which is notoriously the case recently. This is e.g. useful to test the accuracy/speed tradeoff of quantizing certain additional layers, or useful to e.g. release a single model compatible with fused shared expert / unfused shared expert. This has e.g. been requested in separate PRs: - https://github.com/vllm-project/vllm/pull/48983 - https://github.com/vllm-project/vllm/pull/49313 ### 4. Requantize already-quantized layers to different precision Not implemented yet, closest prototype is https://github.com/vllm-project/vllm/pull/48427. This is similar to https://github.com/sgl-project/sglang/pull/28291 & https://github.com/sgl-project/sglang/pull/29328 on sglang side. This idea here would be to allow loading through a certain quantization method, dequantize and override the quantization method on request. This has e.g. been requested in: - https://github.com/vllm-project/vllm/pull/48427 - https://github.com/vllm-project/vllm/pull/51274 ### Feedback Period. One month ### CC List. @BowenBao @mgoin @kylesayrs @vkuzo @maeehart @tanpinsiang @fanxingran @access2rohit ### Any Other Things. _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52167", "createdAt": "2026-08-13T13:25:24Z", "updatedAt": "2026-08-13T15:09:22Z", "timestamp": "2026-08-13T15:09:22Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "RFC", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4434b2d0208e868110f5", "signalId": "github:vllm-project/vllm:pull_request:51159", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Defer `tilelang` import through its import `from vllm.tilelang_utils import tilelang` and relaxed `has_tilelang`", "text": "## Purpose Fixes https://github.com/vllm-project/vllm/issues/51151 This PR introduces `vllm.tilelang_utils` and prevents direct `tilelang` imports. Avoid importing TileLang during ROCm module import, because importing TileLang can load **wrongful/bugged** TVM and HIP stub symbols into the global process scope before AITER loads its JIT modules. This changes `_tilelang_jit` so ROCm applies `tilelang.jit` lazily on first kernel call, while non-ROCm keeps eager JIT decoration. This PR is AI-assisted. Proper fixes in followups need to be: * a PR in `tilelang` or TVM to fix the wrongful symbol load. * a PR in vLLM so that `_has_module` does not import modules https://github.com/vllm-project/vllm/issues/51162 ## Test Plan - `python tools/pre_commit/check_forbidden_imports.py --test-regex` - `pytest tests/kernels/test_mhc_tilelang_jit.py -v` Added new test to verify that CUDA applies `tilelang.jit` eagerly, ROCm defers `tilelang.jit` until first call, and the ROCm wrapper caches the compiled kernel. ```python VLLM_ROCM_USE_AITER=1 HIP_VISIBLE_DEVICES=1 pytest tests/quantization/test_online.py::test_online_quantization[True-fp8_per_tensor-None-Fp8PerTensorOnlineLinearMethod-Fp8PerTensorOnlineMoEMethod] -s -vvvvv ``` that is failing on `main` following https://github.com/vllm-project/vllm/pull/50879, and which surfaces the issue https://github.com/vllm-project/vllm/issues/51151 ## Test Result `test_mhc_tilelang_jit.py` pass on MI350. `tests/quantization/test_online.py` as well on MI300 (previous failing case) `python tools/pre_commit/check_forbidden_imports.py --test-regex` passes as well.", "url": "https://github.com/vllm-project/vllm/pull/51159", "createdAt": "2026-08-05T12:25:31Z", "updatedAt": "2026-08-13T15:09:15Z", "timestamp": "2026-08-13T15:09:15Z", "metrics": { "reactions": 2, "comments": 25 }, "labels": [ "documentation", "rocm", "ready" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c3c71bfbbdde8deb1a9a", "signalId": "github:vllm-project/vllm:pull_request:51906", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51906", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add routed-experts prompt offset", "text": "## Summary - Add `routed_experts_prompt_start` to OpenAI chat/completion requests and `SamplingParams`, allowing clients to omit an already-known prompt prefix from returned R3. - Centralize NumPy-to-base64 serialization used by existing R3 responses and document the `int32` expert-ID representation. - Keep OpenAI streaming behavior unchanged: R3 remains supported only on existing non-streaming responses. ## Why this is not duplicate work No open PR implements the R3 prompt offset or shared R3 response serialization as a frontend change. #49555 returns sparse-attention indexer top-k, which is a distinct feature; this PR only factors shared R3-facing plumbing. ## Validation - `uvx pre-commit run` — passed. - `PYTHONPATH=. /home/aoshen/vllm/.venv/bin/python -m pytest tests/test_sampling_params.py tests/entrypoints/openai/test_stop_token_ids.py tests/utils_/test_serial_utils.py -q` — 49 passed. - R3 OpenAI integration coverage remains in `tests/entrypoints/openai/test_return_routed_experts.py`. Model evaluation was not run: this changes request/response transport and validation only; it does not change routing computation or generated model outputs. AI assistance was used to implement and validate this PR.", "url": "https://github.com/vllm-project/vllm/pull/51906", "createdAt": "2026-08-12T02:00:29Z", "updatedAt": "2026-08-13T15:09:00Z", "timestamp": "2026-08-13T15:09:00Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "frontend", "ready" ], "author": "aoshen02", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:28415da195058c46af19", "signalId": "github:vllm-project/vllm:issue:52180", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:52180", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Improve Pixtral vision attention scaling for batched images", "text": "### Proposal to improve performance Pixtral concatenates the patch sequences for all images in an encoder batch, while attention remains semantically independent per image. On the default installation, where xFormers is unavailable, the model constructs a dense block-diagonal mask and applies SDPA to the combined sequence. The resulting attention work and mask storage scale poorly as more images are batched. This issue proposes routing both Pixtral vision implementations through vLLM's `MMEncoderAttention`, using cumulative sequence lengths to preserve per-image boundaries. This would: - use vLLM's existing multimodal attention backend selection, including FlashAttention, FlashInfer, Torch SDPA, Triton attention, and ROCm AITER; - avoid materializing the dense combined-sequence mask; - remove Pixtral's model-specific optional xFormers path; and - preserve checkpoint compatibility, preprocessing, output ordering, and token counts. Backend-specific packed metadata still needs to be respected. In particular, FlashInfer uses padded sequence-length metadata and transformed cumulative offsets, so the implementation delegates metadata construction to `MMEncoderAttention` rather than assuming the FlashAttention layout. ### Report of performance regression Controlled local benchmark on an NVIDIA DGX Spark (GB10), using BF16, the 24-layer Pixtral HF vision encoder, 840 x 840 images, three warmup iterations, and ten measured iterations. Both variants used the same branch parent, container, model configuration, and identically seeded random weights. | Encoder batch | Unpatched base | Packed FlashAttention | Speedup | | ---: | ---: | ---: | ---: | | 1 | 119.4 ms / 8.37 images/s | 95.9 ms / 10.43 images/s | 1.25x | | 2 | 371.1 ms / 5.39 images/s | 213.4 ms / 9.37 images/s | 1.74x | | 4 | 1187.4 ms / 3.37 images/s | 430.9 ms / 9.28 images/s | 2.76x | Numerical validation against Torch SDPA used the same 24-layer, identically initialized encoder on mixed-size images: | Backend | Max absolute difference | Cosine similarity | | --- | ---: | ---: | | FlashAttention | 0.00220 | 0.999985 | | FlashInfer | 0.00195 | 0.999980 | A two-layer pretrained-weight check produced identical features when the same image was encoded alone or alongside a differently sized image. A full Mistral3 vLLM smoke test also loaded a real checkpoint, selected the requested FlashAttention vision backend, processed two differently sized images, and generated successfully. ### Misc discussion on performance ## Related work and design implications - #40703 migrates Pixtral's current xFormers imports to `mslk` while intentionally preserving the existing attention design. This proposal is materially different and would supersede the need for that model-specific path if accepted. - #38175 tracks full ViT CUDA graph support, and Pixtral/Mistral3 work has recently been claimed there. This proposal does not add encoder CUDA graph support, but metadata preparation may eventually move outside `forward` as part of that work. - #43052 addresses a separate Pixtral/Mistral3 graph-mode failure. Would maintainers prefer Pixtral to adopt the standard `MMEncoderAttention` path, or retain a model-specific `mslk` path? If `MMEncoderAttention` is preferred, should graph-friendly metadata preparation be included here or remain scoped to the CUDA graph work in #38175? ### Your current environment (if you think it is necessary) ```text The output of `python collect_env.py` ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52180", "createdAt": "2026-08-13T14:51:49Z", "updatedAt": "2026-08-13T15:07:31Z", "timestamp": "2026-08-13T15:07:31Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "performance" ], "author": "oliverholworthy", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f74762f6563101f5fafb", "signalId": "github:vllm-project/vllm:pull_request:52169", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52169", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Docs] Note pooled aggregation semantics for mean_acceptance_length", "text": "## Purpose `SpecDecodingLogging._log` and the PromQL recipe in `SpecDecodingProm.__doc__` compute the `mean_acceptance_length` as a **pooled mean** (weighted by draft count, equivalent to `total_generated / total_verify_calls`), not an unweighted mean of per-request means. Some external benchmarks report the latter — SpecForge's `average_acceptance_length`, for example, is `statistics.fmean(acceptance_lengths)` over per-request lengths. On variable-length workloads the two aggregations can differ by several points in either direction, depending on the sign of the per-request-length ↔ acceptance-rate correlation. This has confused at least one report already (issue #42508: a 55.08%-vs-44% pooled-vs-mean-of-means gap for EAGLE3 on GSM8K, and 66.57%-vs-72% in the opposite direction for the standalone drafter — same numerator, different aggregation). ## Changes Docstring/comment only. Adds a short aggregation note - inline where `mean_acceptance_length` is computed, and - inside `SpecDecodingProm.__doc__` below the existing PromQL recipe. Both points cross-reference #42508 for the reproduction. The metric definitions and the CLI/Prometheus output are byte-identical to before. ## Test Plan None — comment/docstring only, no runtime behavior changed. `ruff format --check` clean; pre-commit hooks pass locally. ## Test Result n/a ## Documentation Update n/a (this PR *is* the documentation update). ## Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [x] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1MBOTQeE4aOFIL8UkuGRyGXe6qYtsIt1FGWyxTvvIvtM/edit?tab=t.0).", "url": "https://github.com/vllm-project/vllm/pull/52169", "createdAt": "2026-08-13T13:28:14Z", "updatedAt": "2026-08-13T15:06:02Z", "timestamp": "2026-08-13T15:06:02Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "speculative-decoding" ], "author": "arrdel", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7dfee226b386c143856b", "signalId": "github:vllm-project/vllm:pull_request:52182", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52182", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Remove VLLM_TEST_FORCE_FP8_MARLIN to replace with linear_backend/moe_backend", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52182", "createdAt": "2026-08-13T14:53:00Z", "updatedAt": "2026-08-13T15:05:14Z", "timestamp": "2026-08-13T15:05:14Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "mgoin", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:3c8c496f6b410f67566d", "signalId": "github:vllm-project/vllm:issue:51914", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:51914", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark", "text": "## Summary With **DeepSeek-V4-Flash-0731** on **vLLM v0.27.1** with **DSpark enabled**, we have intermittently observed malformed DSML tool-call output where the opening wrapper is corrupted from: ```text <｜DSML｜tool_calls> ``` to something like: ```text <｜DSML｜toolcalls> ``` while the rest of the DSML block remains structurally close to valid and the closing wrapper is still emitted as: ```text ``` This causes the tool-call parser to miss the call and surface the DSML as assistant content instead of producing `tool_calls`. This report intentionally uses a **synthetic example** and does not include the original production prompt, tool names, arguments, paths, or other private information. ## Environment - vLLM: **v0.27.1** - Model: **deepseek-ai/DeepSeek-V4-Flash-0731** - Speculative decoding: **DSpark enabled** - Tool-call parser: DeepSeek V4 / DSML path - Observed intermittently rather than deterministically ## Synthetic example Expected model output: ```text <｜DSML｜tool_calls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` Malformed output observed in the same class of failure: ```text <｜DSML｜toolcalls> <｜DSML｜invoke name=\"get_weather\"> <｜DSML｜parameter name=\"city\" string=\"true\">Seoul ``` The difference is only the opening wrapper (`tool_calls` -> `toolcalls`), but that is enough for strict DSML parsing to fail. ## Expected behavior If the model emits an otherwise complete DSML invoke block for a tool that is actually declared in the request, vLLM should ideally either: 1. recover the tool call safely, or 2. at minimum detect the malformed DSML as a protocol error instead of silently returning the whole block as ordinary assistant content. Any recovery should remain conservative and validate the recovered tool name against the tools declared in the request. ## Relation to existing issues This appears related to #48931 and #49117, but it is a **different malformed-output variant**. - #48931 covers the case where the `<｜DSML｜tool_calls>` opening wrapper is **completely omitted**. - This case has an opening wrapper, but it is **corrupted/misspelled** (`tool_calls` -> `toolcalls`). - Therefore, recovery logic that only handles a fully missing wrapper may not cover this case as-is. It would be useful to add parser regression coverage for malformed opening wrappers in both streaming and non-streaming paths, especially with DeepSeek-V4-Flash-0731. ## DSpark note This was observed with DSpark enabled. I do **not** have enough evidence yet to claim that DSpark is the cause. A DSpark ON/OFF comparison would be useful because speculative decoding should not materially change protocol correctness if rejected draft tokens are handled correctly. The main issue here is that a small model-side DSML formatting error currently turns into a complete tool-call failure at the serving layer.", "url": "https://github.com/vllm-project/vllm/issues/51914", "createdAt": "2026-08-12T02:54:35Z", "updatedAt": "2026-08-13T15:03:07Z", "timestamp": "2026-08-13T15:03:07Z", "metrics": { "reactions": 1, "comments": 3 }, "labels": [], "author": "jinbagi", "state": "open", "assignees": [ "chaunceyjiang" ], "change": "updated" } }, { "id": "event:4410882116bfeba9460a", "signalId": "github:vllm-project/vllm:pull_request:51793", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Remove dead `QuantizationConfig.is_mxfp4_quant`", "text": "## Purpose This is dead code following https://github.com/vllm-project/vllm/pull/37128. This was originally added in https://github.com/vllm-project/vllm/pull/29008 that supported padding for gpt-oss / certain MXFP4 backends, see: https://github.com/xuebwang-amd/vllm/blob/c62f664e97977ee54ab1d1c77604ebb45081bc06/vllm/model_executor/layers/fused_moe/layer.py#L259-L277 This is now handled in: https://github.com/vllm-project/vllm/blob/490259c1f63faf025b8050504db63d81c817d781/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L640 ## Test Plan N/A ## Test Result N/A", "url": "https://github.com/vllm-project/vllm/pull/51793", "createdAt": "2026-08-11T09:46:17Z", "updatedAt": "2026-08-13T15:02:07Z", "timestamp": "2026-08-13T15:02:07Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "quantization" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3137e73f9482c5c05745", "signalId": "github:vllm-project/vllm:issue:46967", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:46967", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]:[New Model] Gemma4UnifiedForConditionalGeneration (google/gemma-4-12B-it)", "text": "### The model to consider. `google/gemma-4-12B-it` — `model_type: gemma4_unified`, architecture `Gemma4UnifiedForConditionalGeneration` (Google's unified text+vision+audio \"omni\" Gemma 4). Public repo; ships no remote code (no `auto_map` / modeling `.py`). ### The closest model vLLM already supports. vLLM registers `Gemma4ForCausalLM` and `Gemma4ForConditionalGeneration`, but **not** the `Unified` variant. The supported-models docs list only `Gemma4ForCausalLM`. ### What happens today. Serving fails at model load, before any request: ValueError: The checkpoint you are trying to load has model type gemma4_unified but Transformers does not recognize this architecture. Verified in-image: - `Gemma4UnifiedForConditionalGeneration` is **absent** from `ModelRegistry.get_supported_archs()`. - `gemma4_unified` is **absent** from `transformers` `CONFIG_MAPPING_NAMES`. - No remote code in the repo, so `--trust-remote-code` does not help. ### Request. Add `Gemma4UnifiedForConditionalGeneration` to vLLM's model registry, and document the minimum `transformers` version that provides the `gemma4_unified` config.", "url": "https://github.com/vllm-project/vllm/issues/46967", "createdAt": "2026-06-28T17:16:35Z", "updatedAt": "2026-08-13T15:00:20Z", "timestamp": "2026-08-13T15:00:20Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "feature request" ], "author": "adityasingh0510", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:a6dfc9c4609b1f0c656d", "signalId": "github:vllm-project/vllm:pull_request:52183", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52183", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel)", "text": "## Purpose Update the PyTorch ecosystem to the **2.14.0** release candidate, resolving wheels from the PyTorch **test channel** (`download.pytorch.org/whl/test/...`): - `torch`: → **2.14.0** - `torchvision`: → **0.29.0** - `triton`: → **3.8.0** 2.14.0 is published on `download.pytorch.org/whl/test/` but not yet on the release index / PyPI, so the index URLs point at the test channel. This mirrors the 2.13.0 test PR #45731. ### Verified availability (test channel) before pinning - `torch-2.14.0+cu130`: present - `torchvision-0.29.0`: present - `triton==3.8.0`: present, and is exactly what `pytorch` `release/2.14` pins in `.ci/docker/triton_version.txt` - `torchaudio`: newest published is still **2.11.0** on every channel, so that pin is unchanged ### ROCm `requirements/build/rocm.txt` moves from the `rocm7.1` index to `test/rocm7.2`. `rocm7.1` has no torch 2.14.0 wheels — 2.14 is built for `rocm7.2` and `rocm7.14` only. That file was also still on torch 2.11.0 / triton 3.6.0, so this brings it in line with `CMakeLists.txt`'s `TORCH_SUPPORTED_VERSION_ROCM`. ### Ray dependency-compatibility check `.buildkite/scripts/check-ray-compatibility.sh` now offers `uv` both the stable and test channels (it already runs with `--index-strategy unsafe-best-match`), so an RC `torch` pin coming from a transitive dep resolves during the RC window instead of failing against a stable-only index. Same fix as #45731. ### Deliberately *not* carried over from #45731 That PR also carried fixes reacting to breakages found during 2.12/2.13 CI. Those are not copied here, since 2.14 should get whatever it actually needs once CI reports: - xfails for pytorch#184431, pytorch#187735, pytorch#187727 - the ROCm source-build override + `install_flash_attn_rocm.sh` (needed because ROCm/pytorch had no `release/2.13` branch) - `verify_torch_wheel.py` / `UV_NO_VERIFY_HASHES` RC-respin handling `requirements/build/tpu.txt` is also left alone — TPU torch is bumped separately (#50412). ## Test Plan Full CI (apply the `ready` label to trigger the Buildkite run). ## BC-breaking? No — test-channel version bump only. *Authored with AI assistance; modeled on #45731.*", "url": "https://github.com/vllm-project/vllm/pull/52183", "createdAt": "2026-08-13T14:56:49Z", "updatedAt": "2026-08-13T15:00:13Z", "timestamp": "2026-08-13T15:00:13Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "ci/build", "cpu", "nvidia" ], "author": "atalman", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:330e5246f95b56ce4ad4", "signalId": "github:vllm-project/vllm:pull_request:47562", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47562", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Drop incomplete tool-call markup in non-streaming to match streaming", "text": "## Purpose Fixes #47137 When generation terminates (`max_tokens`/`length` or a `stop` string) inside a `` opener that has not been promoted to a tool call, the streaming path drops the buffered markup but the non-streaming path returned it as `content` — the two paths disagreed for identical model output. Contrary to the hope in the issue thread that #46875 had fixed this, I re-ran the issue's no-server unit reproduction on current main (`978de83`) and **the divergence still reproduces** (confirming @TheDuyIT's July 1 retest): ``` 2 '\\n' stream= '' nonstream= '\\n' DIVERGE 3 '\\n<' stream= '' nonstream= '\\n<' DIVERGE 4 '\\n ruff check / ruff format / typos / mypy: Passed ``` Also re-ran the issue's part-D reproduction with the fix: all three truncation points now MATCH between streaming and non-streaming. ## Test Result All new and existing parser tests pass on macOS CPU (source build, `VLLM_TARGET_DEVICE=cpu`). --- This PR was authored with AI assistance (Claude Code); the diff was reviewed and the tests were run locally as described above. cc @bbrowning @TheDuyIT", "url": "https://github.com/vllm-project/vllm/pull/47562", "createdAt": "2026-07-03T15:21:25Z", "updatedAt": "2026-08-13T15:00:01Z", "timestamp": "2026-08-13T15:00:01Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "bug", "tool-calling" ], "author": "JaynouOliver", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:93d4e61fe88e87bad981", "signalId": "github:vllm-project/vllm:pull_request:49544", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] gfx942: use FlyDSL fp8 MQA logits kernel (ROCm/aiter#3913)", "text": "On gfx942, replace the vendored Triton `fp8_mqa_logits` with `aiter.ops.flydsl.flydsl_fp8_mqa_logits` from ROCm/aiter#3913. Drop-in replacement (identical args/semantics), gated behind `_ON_GFX942`. gfx950 and other paths are untouched. ## Results: GLM-5.2-FP8, 8× MI325X TP8 ### ISL=128K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 42,839 | 21,871 | **−49%** | | Median TPOT (ms) | 109.11 | 68.76 | **−37%** | | Output tok/s | 52.44 | 85.34 | **+63%** | ### ISL=8K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |--------|-------------------|--------|-------| | Median TTFT (ms) | 918 | 863 | −6% | | Median TPOT (ms) | 19.79 | 19.68 | −0.6% | | Output tok/s | 382.2 | 386.3 | +1.1% | No regression at short context; gains scale with input length as seen above ## Accuracy | Test | Filter | Result | |------|--------|--------| | GSM8K (5-shot, full 1319 samples) | flexible-extract | 0.9416 ± 0.0065 | | | strict-match | 0.9424 ± 0.0064 | | NIAH (128K tokens, 10 needles, 10 trials) | — | 8/10 perfect, 2/10 miss one needle |
Serve command 
export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export HSA_NO_SCRATCH_RECLAIM=1 vllm serve zai-org/GLM-5.2-FP8 --block-size 1 --compilation-config '{\"pass_config\": {\"fuse_norm_quant\": true, \"fuse_act_quant\": true}, \"custom_ops\": [\"+rms_norm\", \"+silu_and_mul\"]}' --disable-log-stats --enable-auto-tool-choice --no-enable-expert-parallel --gpu-memory-utilization 0.80 --hf-overrides '{\"use_index_cache\": true, \"index_topk_freq\": 8}' --host 0.0.0.0 --kv-cache-dtype fp8 --max-model-len 263168 --max-num-batched-tokens 16384 --max-num-seqs 128 --no-enable-prefix-caching --port 8000 --reasoning-parser glm45 --tensor-parallel-size 8 --tool-call-parser glm47 --trust-remote-code
Bench command 
vllm bench serve --backend vllm --model zai-org/GLM-5.2-FP8 --host localhost --port 8000 --dataset-name random --random-input-len 131072 --random-output-len 1024 --max-concurrency 8 --num-prompts 20 --num-warmups 2 --seed 4 --ignore-eos --trust-remote-code
", "url": "https://github.com/vllm-project/vllm/pull/49544", "createdAt": "2026-07-23T07:20:08Z", "updatedAt": "2026-08-13T14:57:49Z", "timestamp": "2026-08-13T14:57:49Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "v1" ], "author": "akii96", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6d0f5cb9f253e6fc7082", "signalId": "github:vllm-project/vllm:pull_request:47392", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:47392", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MoE] Plumb swigluoai activation into FlashInfer b12x MoE", "text": "## Purpose Models with a clamped SwiGLU-OAI MoE activation (`swigluoai_uninterleave`, e.g. [nvidia/MiniMax-M3-NVFP4](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)) cannot use the FlashInfer b12x MoE backend: `FlashInferB12xExperts` handles only silu and relu2. This matters on SM120/SM121: FlashInfer TRT-LLM MoE requires SM100-family GPUs and FlashInfer CUTLASS lacks this activation, so such models have no FlashInfer MoE path there. FlashInfer supports `swigluoai_uninterleave` on this kernel since flashinfer-ai/flashinfer#3744; this PR adds it to `FlashInferB12xExperts` and passes `swiglu_alpha/beta/limit` to the `B12xMoEWrapper`, gated on the installed FlashInfer supporting them. Behavior with the current pin (0.6.13, which predates the kernel support) is unchanged; a later pin bump enables these models on b12x with no further vLLM changes. The unit test now drives the production weight-processing path instead of hand-preparing weights, covers silu and swigluoai, and uses a relative-error check that actually fails on a wrong gate/up ordering or activation (the old tolerances passed under any kernel behavior). This guards the whole b12x weight pipeline, not just the new activation. Not a duplicate of #47001, which widens b12x `_supports_activation` without plumbing the activation into the kernel call. ## Test Plan ``` pytest tests/kernels/moe/test_flashinfer_b12x_moe.py ``` Needs SM120 and FlashInfer at/past flashinfer-ai/flashinfer#3744 for the swigluoai cases; they skip on the current pin. ## Test Result RTX 5080, FlashInfer at the #3744 merge commit: 48 passed with the fix. Without the fix, all 24 swigluoai cases fail (relative error 0.60 vs the 0.45 bound) and the 24 silu cases pass. On pre-#3744 FlashInfer the swigluoai cases skip and silu passes. Developed with AI assistance (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/47392", "createdAt": "2026-07-02T05:02:35Z", "updatedAt": "2026-08-13T14:57:49Z", "timestamp": "2026-08-13T14:57:49Z", "metrics": { "reactions": 1, "comments": 4 }, "labels": [ "bug", "needs-rebase", "nvidia" ], "author": "yichengj0", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:a8dc54b66d088ce21f98", "signalId": "github:vllm-project/vllm:pull_request:46340", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46340", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] TD operand loads for batched MoE GEMM (moe_mmk) on XPU", "text": "### Purpose Add a Tensor-Descriptor (TD) operand-load path to `moe_mmk` / `expert_triton_kernel` (the batched MoE expert GEMM) and enable it on XPU. On XPU, masked `tl.load` feeding `tl.dot` bypasses the Xe XMX 2D-block-read path; loading both operands via `tl.make_tensor_descriptor` restores it. Output is bit-identical to the non-TD path. The batched kernel is the one used by **low-latency Expert-Parallel** dispatch (the batched activation format `[E_local, max_tokens, K]`), so its e2e effect must be measured **in that EP context**, not against the single-GPU fused MoE (a different path, rightly faster when a model fits on one GPU). This addresses the earlier review feedback: the correct comparison is **TD-on vs TD-off on the batched EP path**, which is what this PR touches. (Earlier single-GPU GraniteMoE numbers were misleading and are superseded below.) Gated by `VLLM_TRITON_USE_TD` (auto-on for XPU, opt-in on CUDA sm90+). Stacks under the portable low-latency all-to-all EP backend (#46871). ### Test Plan - Kernel unit tests: `tests/kernels/moe/test_batched_moe.py` (TD vs plain bit-exact, zero-expert handling, device enablement, backend selection). - E2E via `vllm serve` + `vllm bench serve` on **2× Intel Arc B70**, DeepSeek-V2-Lite, batched Triton EP path (`--all2all-backend naive_low_latency`), bf16, `--enforce-eager`, seed 42. Isolates TD by toggling only `VLLM_TRITON_USE_TD`. (`VLLM_MLA_DISABLE=1` — DeepSeek MLA on XPU has a separate pre-existing numerics bug unrelated to this kernel.) - Accuracy: gsm8k on GraniteMoE (standard attention → clean XPU read) confirms TD is neutral. ### Test Result ### Per-kernel microbench - Arc B70 (E=8, K=N=2048, bf16): TD −89…−91% latency across tokens/expert. - RTX 5060 / H200: TD −68…−70% (CUDA), gains scale with expert GEMM size; bit-exact. ### E2E on the batched low-latency EP path — 2× Arc B70, DeepSeek-V2-Lite (TD off vs on) **Saturation (ShareGPT, 500 prompts):** | metric | TD0 | TD1 | Δ TD1 vs TD0 | |---|---|---|---| | output tok/s | 134.2 | 290.6 | **+116%** | | median TTFT | 27.1 s | 12.0 s | **−56%** | | median TPOT | 797 ms | 334 ms | **−58%** | **Single-stream (`--max-concurrency 1`, realistic low-latency regime):** | metric | TD0 | TD1 | Δ | no-EP default | |---|---|---|---|---| | median TTFT | 1033 ms | 600 ms | −42% | 427 ms | | median TPOT | 353 ms | 270 ms | −24% | 124 ms | ### CUDA cross-check (DeepEP low-latency, Qwen3-30B-A3B, 4× H200) +72% req/s, −53% TPOT with TD on; per-kernel 1.3–6.1× (scales with K/N); bit-identical. (Logs: `cuda_td_deepep_ll_logs.zip`, attached below in a comment) ### Accuracy (TD is a bit-exact operand-load change) GraniteMoE gsm8k (8-shot, 200): default 0.500 / TD0 0.480 / TD1 0.470 — flat within noise, matching the model's published ~0.49. Takeaway: TD is a large, consistent win on the batched EP path it actually touches (+116% throughput, −58% TPOT on 2× Arc B70; +72% req/s on H200), bit-exact, accuracy-neutral, never a regression. Full XPU logs: [pr46340_bench_logs.zip](https://github.com/user-attachments/files/30139331/pr46340_bench_logs.zip)", "url": "https://github.com/vllm-project/vllm/pull/46340", "createdAt": "2026-06-22T07:08:49Z", "updatedAt": "2026-08-13T14:54:11Z", "timestamp": "2026-08-13T14:54:11Z", "metrics": { "reactions": 2, "comments": 10 }, "labels": [ "intel-gpu", "ready", "v1", "verified" ], "author": "oonyshch", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:d1c1d27fa2683063a1c6", "signalId": "github:vllm-project/vllm:pull_request:51291", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51291", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Bugfix] Restore AITER-compatible Triton 3.7 pin", "text": "## Purpose Restore an AITER-compatible ROCm Triton 3.7 pin. [vLLM #50607](https://github.com/vllm-project/vllm/pull/50607) moved the ROCm image to Triton `0263a6a` from `release/internal/3.7.x`. That revision rejects the `DistributedLinearLayout` offsets produced by AITER's CDNA4 Gluon MLA kernel: ```text expected offsets type layout to be BlockedLayout or SliceLayout ``` The Kimi-K3 DSpark K=7 stress reproducer reaches this path during piecewise graph capture because the draft verifies eight query tokens across 12 local heads. Pinning ROCm Triton to `89002410` restores the distributed-layout lowering used by AITER without changing vLLM or AITER runtime code. The production InferenceX configuration uses K=2 and is tested separately below. This restores compilation and execution of AITER's actual Gluon MLA kernel; it does not disable Gluon or route the workload to a different attention backend. At the package-version level, this is a compatibility rollback from Triton 3.7.1 to the validated 3.7.0 revision. The commits are on divergent branches, so this is not a linear Git revert. A corrected internal 3.7.x revision can replace this pin after the complete layout support is backported and validated. [AITER v0.1.19's default installer](https://github.com/ROCm/aiter/blob/v0.1.19/.github/scripts/install_triton.sh#L77-L83) pins `triton==3.7.0`. On AITER's ROCm 7.2 package index, that resolves to `triton-3.7.0+amd.rocm7.2.0.git89002410`, so this change matches AITER's production wheel exactly. AITER's older `756afc06` pin is used only by its opt-in source-build CI path. I searched open vLLM, AITER, and ROCm Triton issues and pull requests for the error, the two Triton revisions, and the AITER MLA layout. I found no existing fix. [Triton #9513](https://github.com/triton-lang/triton/pull/9513) is related: cherry-picking only its relaxed `DistributedLayout` type check onto ROCm Triton's `release/internal/3.7.x` branch at the exact vLLM-pinned base commit [`0263a6a6203cf27c441c57a6c808ea87ffb8f654`](https://github.com/ROCm/triton/commit/0263a6a6203cf27c441c57a6c808ea87ffb8f654) made the exact Kimi-K3 shape compile, but the default two-split path still produced nondeterministic incorrect output. In a deterministic seed sweep from 0 through 15, seeds 7, 10, 14, and 15 failed at `atol=rtol=0.01`; seeds 7 and 10 had about 10.3% mismatching elements with maximum absolute errors of 3.67 and 3.84. Repeated seed-7 runs used identical input and reference hashes but produced different GPU output hashes, with mismatches confined to query position 2 across all 12 heads. Forcing one split on the same inputs passed with maximum absolute error `0.00390625`, isolating the remaining problem to the multi-split Stage-1/Stage-2 dependency rather than the frontend layout check. Therefore the isolated #9513 backport is not sufficient; the validated `89002410` revision is used instead. AI assistance was used for root-cause investigation and drafting. The submitter reviewed the Dockerfile changes and test evidence before opening this pull request. ## Test Plan ### 1. Docker build Build the same stage changed by this PR: ```bash docker build \\ -f docker/Dockerfile.rocm_base \\ --target build_triton \\ -t vllm-rocm-build-triton:89002410 . ``` The stage must fetch the full immutable commit, complete the Triton build, and produce a `triton-3.7.0+git89002410` wheel. ### 2. AITER MLA operator reproducer Compare the immutable ROCm nightly image `sha256:726321b39e1dbbe24d2735160be54d679496ef5d2a8479a264394f7296908617` with only its Triton package changed from `0263a6a` to `89002410`. Run the AITER MLA operator at the Kimi-K3 DSpark verification shape: ```bash python3 op_tests/test_mla.py -c 128 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal ``` Also run context length 256 while forcing the one- and two-split reduction paths. ### 3. TP8 endpoint smoke test Build a new ROCm base image directly from this PR's `docker/Dockerfile.rocm_base`. Then build the runnable vLLM server image from the same PR checkout using that base. Do not install or copy Triton into an existing nightly image. ```bash git rev-parse HEAD # 2d1bb40fb4a07e65706749b77c1701eed771695a DOCKER_BUILDKIT=1 docker build \\ --progress=plain \\ -f docker/Dockerfile.rocm_base \\ -t rocm/vllm-dev:pr51291-base . DOCKER_BUILDKIT=1 docker build \\ --progress=plain \\ -f docker/Dockerfile.rocm \\ --build-arg BASE_IMAGE=rocm/vllm-dev:pr51291-base \\ -t vllm/vllm-openai-rocm:pr51291-source . ``` Verify that the runnable image contains the PR source revision and the Triton wheel produced by the base-image build: ```bash docker run --rm --entrypoint /bin/bash \\ vllm/vllm-openai-rocm:pr51291-source -lc ' cat /app/versions.txt python3 - <<\"PY\" from importlib.metadata import version import triton import vllm print(\"vLLM:\", vllm.__version__) print(\"AITER:\", version(\"amd-aiter\")) print(\"Triton:\", triton.__version__) PY ' ``` `/app/versions.txt` must contain `TRITON_BRANCH: 89002410dee054a1207a3d88cfbe1748dacfd819`, and the installed Triton version must be `3.7.0+git89002410`. Start Kimi-K3 and DSpark on 8x MI355X. This uses InferenceX's production DSpark level 2 and its token-aligned capture sizes: with K=2, each sequence has a decode width of three tokens, so the capture list is `3,6,...,48` for `max_num_seqs=16`. This validation fixes both the model and KV cache to BF16. The command uses `--kv-cache-dtype bfloat16` explicitly rather than relying on dtype inference. ```bash docker run -d --name vllm-kimi-k3-pr51291-source-k2-ix \\ --network host --ipc host \\ --device /dev/kfd --device /dev/dri --group-add video \\ --cap-add SYS_PTRACE \\ --security-opt seccomp=unconfined --security-opt label=disable \\ -v /data/huggingface/Kimi-K3:/model:ro \\ -v /data/huggingface/Kimi-K3-DSpark:/draft:ro \\ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\ -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\ -e AITER_ROCM_ARCH=gfx950 \\ -e VLLM_ROCM_USE_AITER=1 \\ -e VLLM_ROCM_USE_AITER_FP4BMM=1 \\ -e SAFETENSORS_FAST_GPU=1 \\ -e AITER_SITUV2_A8W4=1 \\ -e AITER_BF16_FP8_MOE_BOUND=0 \\ -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \\ --entrypoint vllm \\ vllm/vllm-openai-rocm:pr51291-source \\ serve /model \\ --served-model-name moonshotai/Kimi-K3 \\ --host 0.0.0.0 --port 8000 \\ --tensor-parallel-size 8 \\ --distributed-executor-backend mp \\ --trust-remote-code \\ --load-format auto --moe-backend auto \\ --gpu-memory-utilization 0.95 \\ --kv-cache-memory-bytes 51539607552 \\ --kv-cache-dtype bfloat16 \\ --mm-encoder-tp-mode data \\ --max-num-seqs 16 \\ --max-num-batched-tokens 4096 \\ --max-model-len 1048576 \\ --enable-prefix-caching \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"],\"cudagraph_capture_sizes\":[3,6,9,12,15,18,21,24,27,30,33,36,39,42,45,48]}' \\ --speculative-config '{\"method\":\"dspark\",\"model\":\"/draft\",\"num_speculative_tokens\":2,\"attention_backend\":\"TRITON_MLA\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' ``` Wait for readiness and send a chat smoke request: ```bash until curl --fail --silent http://127.0.0.1:8000/health; do test \"$(docker inspect -f '{{.State.Status}}' vllm-kimi-k3-pr51291-source-k2-ix 2>/dev/null)\" = running || { docker logs vllm-kimi-k3-pr51291-source-k2-ix exit 1 } sleep 30 done curl --fail --silent --show-error \\ http://127.0.0.1:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ -d '{\"model\":\"moonshotai/Kimi-K3\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with one short word.\"}],\"temperature\":0,\"max_tokens\":16}' ``` ### 4. Repository checks ```bash pre-commit run --files docker/Dockerfile.rocm_base ``` ## Test Result ### Docker build The `build_triton` target fetched and checked out `89002410dee054a1207a3d88cfbe1748dacfd819`, completed all 503 compilation steps, and generated the `triton-3.7.0+git89002410` wheel. ### Operator result MI355X, TP8 system, BF16 inputs and KV cache: ```bash # Context 128, automatic split selection. python3 op_tests/test_mla.py -c 128 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal # Context 256, force the single-split path. python3 op_tests/test_mla.py -c 256 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal --gluon-num-kv-splits 1 # Context 256, force the two-split Stage-1/Stage-2 path. python3 op_tests/test_mla.py -c 256 -b 1 -n 12,8 \\ -k 512 -qn 512 -qr 64 -vh 512 -blk 1 \\ -d bf16 -kvd bf16 --causal --gluon-num-kv-splits 2 ``` | Test | `0263a6a` | `89002410` | |---|---|---| | MLA, context 128, automatic two splits | compile error | pass, 10.30 us | | MLA, context 256, forced one split | compile error | pass, 10.64 us | | MLA, context 256, forced two splits | compile error | pass, 10.80 us | All passing operator cases matched the PyTorch reference with maximum absolute error `0.00390625` at `atol=rtol=0.01` and zero failing elements. These tests executed `aiter.ops.triton.gluon.mla_gluon`, including the Kimi-K3 12-head, eight-query-token verification shape. ### TP8 endpoint result The image build and endpoint smoke were run with images built from this PR's source tree and Dockerfiles; no package was replaced in an existing nightly image. No other vLLM pull request or AITER overlay was included. Explicit BF16 KV startup was reconfirmed on 2026-08-07. - Base image: `rocm/vllm-dev:pr51291-base` (`sha256:a53d6c7dae565b2c7df3e530c2eeec496b259d22195e3115ed4ca82737aa4a3b`) - Runnable image: `vllm/vllm-openai-rocm:pr51291-source` (`sha256:4212a65ac86027e370b18ba2ad8acbb53e0f9d72c6d8a64ddf611bf75abf061b`) - Runtime versions: vLLM `0.1.dev19627+g2d1bb40fb.d20260806`, Triton distribution `3.7.0+git89002410`, AITER `0.1.19`, PyTorch `2.12.0+git6bbd260` - Hardware: 8x MI355X (`gfx950`), TP8 - KV cache: BF16, selected explicitly with `--kv-cache-dtype bfloat16` - Target backend: `ROCM_AITER_MLA`; DSpark draft backend: `TRITON_MLA` - With K=2 and the explicit InferenceX token-aligned capture list, piecewise graph capture completed for all 16 capture sizes on all eight ranks in 66 seconds. - `/health` returned HTTP 200. A chat-completions smoke request returned HTTP 200 with 16 completion tokens. - Speculative decoding was active: the K=2 smoke metrics recorded 11 drafts, 22 draft tokens, and four accepted tokens; both draft positions recorded at least one accepted token. The chat request is an endpoint/runtime smoke test, not an accuracy evaluation. All pre-commit hooks applicable to `docker/Dockerfile.rocm_base` passed.", "url": "https://github.com/vllm-project/vllm/pull/51291", "createdAt": "2026-08-06T17:37:39Z", "updatedAt": "2026-08-13T14:50:27Z", "timestamp": "2026-08-13T14:50:27Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm", "needs-rebase", "ci/build" ], "author": "JohnQinAMD", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:b394bf9db2ab33295b10", "signalId": "github:vllm-project/vllm:pull_request:51674", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51674", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN", "text": "## Purpose Speed up Qwen3.5 (GDN linear attention) MTP speculative decode on Blackwell. During MTP decode, the Triton path launches a chain of small kernels per step (gating, delta-rule recurrence, state rewind/update, gated RMSNorm), which leaves the GPU latency-bound at decode batch sizes. This PR adds a single fused CUDA kernel, `fused_gdn_decode_post_conv_mtp`, that consumes the post-convolution `mixed_qkv` and performs the entire GDN MTP decode step in one launch: - SiLU activation on q/k/v and gating (`softplus`/`sigmoid` on `a`, `b`, `A_log`, `dt_bias`) - the gated delta-rule recurrence over up to 8 speculative tokens per request, with rewind to the last accepted token via `num_accepted_tokens` - recurrent state update in place (BF16 or FP32 state) - gated RMSNorm epilogue, with output precision matched to the Triton path ## Test Plan New/extended tests (all require SM100): ```bash pytest tests/kernels/mamba/test_gdn_fused_mtp.py -v pytest tests/kernels/test_fused_gdn_post_conv.py -v pytest tests/v1/attention/test_gdn_metadata_builder.py -v pytest tests/test_envs.py -k \"gdn\" pytest tests/compile/test_config.py ``` `test_gdn_fused_mtp.py` covers the model-path dispatch for pure spec-decode, mixed, prefill, and regular-decode batches against the Triton reference. `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` covers the kernel directly for BF16/FP32 state, ragged acceptance patterns, and TP head counts (tp4/tp16). ## Test Result On 1x B200 (SM100), CUDA 13.1, torch 2.13.0+cu130: - `tests/kernels/mamba/test_gdn_fused_mtp.py`: **6 passed** - `tests/kernels/test_fused_gdn_post_conv.py`: **70 passed** - `tests/v1/attention/test_gdn_metadata_builder.py`: **10 passed** - `tests/test_envs.py -k gdn`: **1 passed** - `tests/compile/test_config.py`: **44 passed** ## Microbenchmark 1x B200, CUDA 13.1, torch 2.13.0+cu130. Both variants measured over the **same kernel boundary**: post-conv `mixed_qkv` in, gated+normed `core_attn_out` out, recurrent state updated in place. - Triton path = `fused_sigmoid_gating_delta_rule_update` + gated RMSNorm (`rmsnorm_fn`), exactly as dispatched by `_forward_core_fused_norm` today. - Fused path = single `fused_gdn_decode_post_conv_mtp` launch. - Timed by capturing 20 iterations in a CUDA graph and replaying 50x with CUDA events (production MTP decode runs under full CUDA graphs, so graph-replay GPU time is the relevant metric; CPU launch overhead excluded for both). - **cold** rotates input/state buffer sets so the working set exceeds 4x L2 (126 MB), i.e. state comes from HBM as in real serving where other layers evict L2 between calls; **warm** reuses one buffer set (L2-hot upper bound). - K=V=128, 4 MTP tokens/request, `num_accepted_tokens=2`. Context length does not affect either variant (fixed-size recurrent state), so results apply to any seqlen (e.g. 8192). Cold-cache speedup is **1.17x-2.20x**, largest in the latency-sensitive BS=4-32 decode regime. **TP=4 (H=4, HV=32), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 20.31 | 12.19 | 1.67x | 16.49 | 11.57 | 1.42x | | 4 | 27.91 | 12.99 | 2.15x | 20.65 | 11.73 | 1.76x | | 8 | 35.69 | 17.89 | 2.00x | 27.26 | 16.59 | 1.64x | | 16 | 46.66 | 32.32 | 1.44x | 41.44 | 30.60 | 1.35x | | 32 | 71.85 | 51.75 | 1.39x | 71.99 | 51.62 | 1.39x | | 64 | 120.65 | 97.12 | 1.24x | 120.61 | 96.93 | 1.24x | | 128 | 221.13 | 188.16 | 1.18x | 221.11 | 188.02 | 1.18x | **TP=4 (H=4, HV=32), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 23.50 | 13.73 | 1.71x | 17.74 | 12.70 | 1.40x | | 4 | 32.04 | 14.57 | 2.20x | 24.12 | 12.82 | 1.88x | | 8 | 42.69 | 20.78 | 2.05x | 33.78 | 18.68 | 1.81x | | 16 | 60.27 | 38.14 | 1.58x | 60.01 | 37.70 | 1.59x | | 32 | 96.82 | 65.27 | 1.48x | 97.08 | 65.15 | 1.49x | | 64 | 167.63 | 130.97 | 1.28x | 167.27 | 126.25 | 1.32x | | 128 | 311.52 | 267.23 | 1.17x | 314.21 | 266.75 | 1.18x | **TP=16 (H=1, HV=8), bfloat16 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 15.87 | 11.52 | 1.38x | 15.92 | 11.57 | 1.38x | | 4 | 19.64 | 12.32 | 1.59x | 16.46 | 11.57 | 1.42x | | 8 | 23.52 | 12.67 | 1.86x | 17.92 | 11.61 | 1.54x | | 16 | 27.56 | 12.92 | 2.13x | 20.64 | 11.69 | 1.77x | | 32 | 35.39 | 17.82 | 1.99x | 27.16 | 16.60 | 1.64x | | 64 | 46.45 | 32.27 | 1.44x | 40.72 | 30.57 | 1.33x | | 128 | 73.57 | 51.77 | 1.42x | 73.69 | 51.61 | 1.43x | **TP=16 (H=1, HV=8), float32 recurrent state** | BS | Triton cold (us) | Fused cold (us) | Speedup | Triton warm (us) | Fused warm (us) | Speedup | |---:|---:|---:|---:|---:|---:|---:| | 1 | 17.44 | 12.59 | 1.38x | 17.71 | 12.61 | 1.41x | | 4 | 23.30 | 13.71 | 1.70x | 17.92 | 12.68 | 1.41x | | 8 | 26.18 | 13.93 | 1.88x | 20.08 | 12.71 | 1.58x | | 16 | 31.77 | 14.50 | 2.19x | 23.99 | 12.81 | 1.87x | | 32 | 42.81 | 20.84 | 2.05x | 34.81 | 19.05 | 1.83x | | 64 | 61.15 | 38.45 | 1.59x | 60.99 | 37.89 | 1.61x | | 128 | 98.89 | 66.26 | 1.49x | 99.21 | 66.21 | 1.50x | --- ## Model Evaluation An lm_eval score comparison was not run: the fused path requires the `num_v_heads == 8 * num_k_heads` GDN head layout, and no public Qwen3.5 checkpoint ships that config (0.8B is 16/16, 35B-A3B is 16/32, 397B-A17B is 16/64) — on those models the dispatch check never selects the fused kernel, so default behavior is unchanged. Correctness of the fused kernel against the Triton path is covered by the numerical-equivalence tests above: `test_fused_gdn_post_conv.py::test_fused_gdn_decode_post_conv_mtp_ratio8` checks kernel outputs and updated recurrent state against `fused_sigmoid_gating_delta_rule_update` + `rmsnorm_fn` across BF16/FP32 state, ragged acceptance patterns, and TP head counts, and `test_gdn_fused_mtp.py` checks the model-path dispatch (pure / mixed / prefill / decode batches) end-to-end at the layer level.
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51674", "createdAt": "2026-08-10T12:49:25Z", "updatedAt": "2026-08-13T14:50:17Z", "timestamp": "2026-08-13T14:50:17Z", "metrics": { "reactions": 5, "comments": 9 }, "labels": [ "ready", "ci/build", "qwen", "nvidia" ], "author": "Jie-Fang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:530577a7d6288a020347", "signalId": "github:vllm-project/vllm:pull_request:42436", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42436", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "fused_moe: add VLLM_TRITON_USE_TD tensor-descriptor path", "text": "Working on perf optimizations for the Triton MoE kernels. Adds an opt-in `VLLM_TRITON_USE_TD` env var that switches the fused MoE kernel onto a tensor-descriptor based load/store path, mirroring `VLLM_TRITON_ATTN_USE_TD` (PR #40327). Auto-on for XPU; off by default on CUDA/ROCm (opt-in on Blackwell). The changes in this PR are scoped to the fused MoE kernel (`fused_moe_kernel`) only. The env var name follows the single-flag design proposed in the TD adoption strategy RFC (#42545): one `VLLM_TRITON_USE_TD` switch gates the tensor-descriptor path across Triton kernels. Note the auto-select policy is per-subsystem: with the env unset, both the attention dispatcher (`triton_attn.py`) and this MoE path auto-enable on XPU only. Blackwell CUDA can run the path but must opt in explicitly (see below). This PR renames the previously-proposed `VLLM_TRITON_MOE_USE_TD` to the general form for consistency with the RFC. ## Quantization The TD gather branch is validated for non-quantized (bf16) weights only. Under any quantization (fp8 w8a8, int8 w8a8, int8 w8a16, int4 w4a16) the launch site falls back to the pointer path — `use_td = resolve_moe_use_td() and not is_quantized` — so quantized MoE is byte-identical to `main`. When `VLLM_TRITON_USE_TD` is set explicitly but ineffective (non-Triton backend, or quantized weights), a one-shot warning fires against the *selected* MoE backend. Extending the TD path to quantized MoE is a known gap and is being addressed as a separate follow-up under the same RFC (#42545); that PR is almost ready for publication. ## Hardware gating (CUDA) The A gather uses `tensor_descriptor.gather`, which lowers to the PTX `tile::gather4` instruction (part of the `tcgen05`/Tensor Memory family introduced with Blackwell). `ptxas` rejects it on Hopper (sm90) and earlier — `Feature '.tile::gather4 ...' requires .target sm_100 or higher`. `moe_use_td_hw_supported()` reports whether a device can *compile* the path: `True` on XPU, `has_device_capability(100)` on CUDA (gather4 spans the whole sm100+ range incl. consumer Blackwell sm120/121, unlike `scatter4` — see triton-lang/triton#8498), `False` otherwise. Auto-select does not follow compile-capability: with the env unset the path is XPU-only. Blackwell CUDA can compile it but is opt-in (`VLLM_TRITON_USE_TD=1`) pending its own accuracy/perf validation — it currently has none, and this PR deliberately does not enable it by default. An explicit `VLLM_TRITON_USE_TD=1` on hardware that cannot compile the path still forces it (useful for A/B), so the parametrized test carries a matching hardware skip-guard, and the launch registers the Triton scratch allocator (`make_tensor_descriptor` needs it on CUDA). A `tl.static_assert` enforces that `USE_TD` and `SWAP_AB` are never both enabled. ## Testing / CI coverage `test_fused_moe` is parametrized over `use_td`. The `use_td=True` cell skips on any device that cannot compile the TD path (Hopper/Ampere, and Triton < 3.6) — i.e. it does not execute on the default upstream kernel-CI fleet. The TD branch is exercised only on XPU / Blackwell CI, so the validation below was run by hand on the target hardware. [pr-42436-multiplatform-validation.zip](https://github.com/user-attachments/files/30166388/pr-42436-multiplatform-validation.zip) ## Results - multi-platform production-readiness validation (2026-07-17/18) Extended the validation above to a full unit → accuracy → E2E-perf → quantized-smoke → kernel-level pass on one card each of Intel Arc Pro B70 (XPU), NVIDIA H200 (Hopper, sm90), NVIDIA B200 (Blackwell, sm100). Pinned commit: `92a58914f9ee6a39bc884c278a2fc6367aea7c3a`. Full methodology, kernel-level benchmark results, dispatch-log verification of the TD auto-select resolver on every platform, hardware-gate verification on H200, quantization no-op verification, an unresolved B70 host-to-host perf disagreement, and a data-correctness correction applied to some `sharegpt` numbers below (documented in the report itself), are in the attached campaign report (`TRITONXPU-181_production_readiness_campaign_2026-07-18.md`). Full raw logs attached per-platform (`b70/`, `h200/`, `b200/`). Summary: correctness (K-alignment guard fix, TD/pointer parity, quantization no-op fallback, hardware-gate enforcement on Hopper) is well-evidenced across all 3 platforms via dispatch-log-level verification, not just output matching. Perf is shape/platform-dependent with no universal TD win — reported honestly below, not as a performance pitch. Variant naming used below: `default` = no `--moe-backend` flag, no `VLLM_TRITON_USE_TD` (platform's true native default — SYCL on B70, FlashInfer TRTLLM on B200, Triton on H200 per vLLM's own Hopper-specific backend priority); `td0`/`td1` = `--moe-backend triton` with `VLLM_TRITON_USE_TD=0`/`=1`. ### Accuracy — GSM8K, full 1319-question test set, 5-shot, greedy decode (temperature=0.0), seed=42 | Platform | td0 (pointer) | td1 (TD) | Δ | |---|---:|---:|---:| | B70 | 46.85% | 46.85% | 0.00pp (bit-identical) | | H200 | 48.60% | N/A (TD uncompilable on sm90) | — | | B200 | 48.07% | 48.07% | 0.00pp (bit-identical) | TD does not change accuracy anywhere it runs — bit-identical to the pointer path token-for-token on both B70 and B200 (confirms the K-alignment correctness fix). Raw per-platform results below.
B70 ```json --- sycl (default) --- { \"accuracy\": 0.47763457164518575, \"invalid_rate\": 0.001516300227445034, \"latency\": 43.434272416867316, \"questions_per_second\": 30.367724071458788, \"total_output_tokens\": 178930, \"tokens_per_second\": 4119.5578984883405, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.46853677028051555, \"invalid_rate\": 0.002274450341167551, \"latency\": 103.7776588909328, \"questions_per_second\": 12.70986466736766, \"total_output_tokens\": 179153, \"tokens_per_second\": 1726.3156821477774, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } --- td1 --- { \"accuracy\": 0.46853677028051555, \"invalid_rate\": 0.002274450341167551, \"latency\": 74.65302416798659, \"questions_per_second\": 17.66840680200637, \"total_output_tokens\": 179153, \"tokens_per_second\": 2399.8090097042054, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"1\", \"seed\": 42 } ```
H200 ```json --- default (= td0, dispatch-confirmed identical Triton code path) --- { \"accuracy\": 0.48597422289613346, \"invalid_rate\": 0.0, \"latency\": 9.003313882742077, \"questions_per_second\": 146.5016123150292, \"total_output_tokens\": 179112, \"tokens_per_second\": 19894.00817662586, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.48597422289613346, \"invalid_rate\": 0.0, \"latency\": 10.623563874978572, \"questions_per_second\": 124.15795824474773, \"total_output_tokens\": 179112, \"tokens_per_second\": 16859.878860601406, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } ``` (No `td1` row: TD cannot compile on Hopper/sm90 — `ptxas` rejects `tile::gather4` as requiring sm_100+. Attempted and captured as an expected-failure artifact, see the attached logs.)
B200 ```json --- default (FlashInfer TRTLLM, real distinct backend) --- { \"accuracy\": 0.4836997725549659, \"invalid_rate\": 0.000758150113722517, \"latency\": 10.077878876123577, \"questions_per_second\": 130.8807156955382, \"total_output_tokens\": 178460, \"tokens_per_second\": 17708.09137454568, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": null, \"vllm_triton_use_td_env\": null, \"seed\": 42 } --- td0 --- { \"accuracy\": 0.4806671721000758, \"invalid_rate\": 0.002274450341167551, \"latency\": 8.92053499398753, \"questions_per_second\": 147.8610869066721, \"total_output_tokens\": 179372, \"tokens_per_second\": 20107.762608509165, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"0\", \"seed\": 42 } --- td1 --- { \"accuracy\": 0.4806671721000758, \"invalid_rate\": 0.002274450341167551, \"latency\": 9.130568650085479, \"questions_per_second\": 144.45978673931242, \"total_output_tokens\": 179372, \"tokens_per_second\": 19645.21673010155, \"num_questions\": 1319, \"num_shots\": 5, \"max_tokens\": 256, \"moe_backend_arg\": \"triton\", \"vllm_triton_use_td_env\": \"1\", \"seed\": 42 } ```
`td0`/`td1` are bit-identical on both B70 and B200 (confirms the K-alignment fix preserves output correctness under TD). Full accuracy methodology caveat (N=1 has a ~1-2pp run-to-run noise floor on this class of backend) is in the attached report §5. ### Perf — `vllm bench sweep serve`, N=3 per (variant, dataset), median reported, seed=42, `--no-enable-prefix-caching` Datasets: `sharegpt` (500 prompts, concurrency 64), `prefill_heavy` (random, in=2048/out=512, concurrency 32), `decode_heavy` (random, in=512/out=2048, concurrency 8). Model: `ibm-granite/granite-3.0-3b-a800m-instruct` (bf16). **td0 vs td1, mean output tok/s over N=3 runs** (mean used here instead of median — with only 3 samples per cell, one dataset's median flips sign depending on which single run is treated as the outlier; mean is more stable for this specific comparison. Raw per-run values are below, unchanged, so this is fully reproducible from the same numbers): | Platform | Dataset | td0 (pointer) | td1 (TD) | Δ | |---|---|---:|---:|---:| | B70 | sharegpt | 719.5 | 762.6 | **+6.0%** | | B70 | prefill_heavy | 420.1 | 417.6 | -0.6% | | B70 | decode_heavy | 134.0 | 133.3 | -0.5% | | B200 | sharegpt | 8181.1 | 8057.0 | -1.5% | | B200 | prefill_heavy | 6264.2 | 6094.0 | -2.7% | | B200 | decode_heavy | 2228.7 | 2144.7 | -3.8% | B70: a real throughput gain on `sharegpt` (mixed realistic prompt/response lengths, BMG's most representative workload), flat on the two synthetic datasets. B200: small negative deltas on prefill/decode-heavy, consistent with this hardware's already-documented TD-neutral-to-slightly- regressive pattern — this PR does not enable TD by default on CUDA, and this data supports that choice.
B70 ```json { \"sycl (default)\": { \"sharegpt\": [982.13, 994.13, 1065.42], \"prefill_heavy\": [640.0, 638.36, 633.31], \"decode_heavy\": [158.47, 158.87, 159.72] }, \"td0\": { \"sharegpt\": [599.93, 774.98, 783.72], \"prefill_heavy\": [418.45, 421.03, 420.96], \"decode_heavy\": [133.91, 134.13, 133.88] }, \"td1\": { \"sharegpt\": [685.9, 768.48, 833.28], \"prefill_heavy\": [418.19, 419.06, 415.58], \"decode_heavy\": [133.4, 133.4, 133.14] } } ``` Medians: sycl sharegpt=994.1, prefill_heavy=638.4, decode_heavy=158.9 | td0 sharegpt=775.0, prefill_heavy=421.0, decode_heavy=133.9 | td1 sharegpt=768.5, prefill_heavy=418.2, decode_heavy=133.4 tok/s.
H200 ```json { \"default\": { \"sharegpt\": [8003.47, 7249.66, 7635.22], \"prefill_heavy\": [5049.66, 4989.72, 4961.28], \"decode_heavy\": [2007.62, 1979.14, 1977.29] }, \"td0\": { \"sharegpt\": [7266.54, 7919.86, 7713.62], \"prefill_heavy\": [5140.22, 5040.31, 5046.53], \"decode_heavy\": [2009.16, 1969.85, 2007.99] } } ``` Medians: default sharegpt=7635.2, prefill_heavy=4989.7, decode_heavy=1979.1 | td0 sharegpt=7713.6, prefill_heavy=5046.5, decode_heavy=2008.0 tok/s. No `td1` (uncompilable on sm90).
B200 ```json { \"default (env-unset, still --moe-backend triton, kept for reference)\": { \"sharegpt\": [7351.53, 8372.63, 8720.87], \"prefill_heavy\": [6207.44, 6315.79, 6205.99], \"decode_heavy\": [2187.75, 2230.64, 2213.36] }, \"default_native (true no-flag default, FlashInfer TRTLLM)\": { \"sharegpt\": [9115.8, 9278.24, 9218.8], \"prefill_heavy\": [7542.21, 7465.44, 7468.28], \"decode_heavy\": [2981.51, 2979.89, 2979.25] }, \"td0\": { \"sharegpt\": [7803.92, 7989.08, 8750.16], \"prefill_heavy\": [6224.09, 6285.98, 6282.4], \"decode_heavy\": [2242.47, 2216.71, 2226.98] }, \"td1\": { \"sharegpt\": [7640.3, 7953.87, 8576.79], \"prefill_heavy\": [6092.19, 6106.03, 6083.66], \"decode_heavy\": [2139.16, 2157.93, 2137.13] } } ``` Medians: `default_native` (FlashInfer, true platform default) sharegpt=9218.8, prefill_heavy=7468.3, decode_heavy=2979.9 | td0 sharegpt=7989.1, prefill_heavy=6282.4, decode_heavy=2227.0 | td1 sharegpt=7953.9, prefill_heavy=6092.2, decode_heavy=2139.2 tok/s. Collected with `--no-enable-flashinfer-autotune` (see attached report §4.5 for an unrelated FlashInfer autotuner crash found and root-caused during this measurement — not a PR#42436 issue).
### Attachments [pr-42436-multiplatform-validation.zip](https://github.com/user-attachments/files/30166403/pr-42436-multiplatform-validation.zip) Attached as a single zip (`pr-42436-multiplatform-validation.zip`) containing: - `TRITONXPU-181_production_readiness_campaign_2026-07-18.md` — full campaign report. - Per-platform raw logs (`b70/`, `h200/`, `b200/`), each split into `accuracy/`, `perf/`, `quant_smoke/`, `kernel_bench/`, `result_json/` — every command's full stdout/stderr/meta.json, no empty directories.", "url": "https://github.com/vllm-project/vllm/pull/42436", "createdAt": "2026-05-12T16:15:09Z", "updatedAt": "2026-08-13T14:49:09Z", "timestamp": "2026-08-13T14:49:09Z", "metrics": { "reactions": 4, "comments": 10 }, "labels": [ "ready" ], "author": "afierka-intel", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:8a909ad734f7d6b6a9c9", "signalId": "github:vllm-project/vllm:pull_request:52179", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52179", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix PP speculative decode cadence with sync scheduling", "text": "## Purpose Fixes #52071. When using the V2 model runner with pipeline parallelism and speculative decoding, the base `Scheduler` can schedule a request again before the sampled token from the previous PP step has been relayed back. The scheduler already checks `request.next_decode_eligible_step`, but this value was only being updated in `AsyncScheduler`. With async scheduling disabled, speculative draft tokens can increase `num_tokens_with_spec` and make the request runnable again while the sampled-token state is still stale. This PR updates `next_decode_eligible_step` in the base scheduler when: - the V2 model runner is enabled; - pipeline parallelism is enabled; - speculative decoding is enabled; - the request is no longer in a prefill chunk. The next eligible step is set to: ```text current_step + pipeline_parallel_size ``` This keeps non-final chunked prefill scheduling unchanged, while decode waits for the PP sampled-token relay before the request can be scheduled again. The existing `AsyncScheduler` handling is unchanged. Thanks @nickus for the detailed investigation in #52071. The reproduction, measurements, and analysis around the PP sampled-token relay and stale sampled tokens were especially helpful in narrowing this down to the scheduler cadence. ## Regression test Added: ```text tests/v1/core/test_scheduler.py::test_v2_pp_spec_decode_waits_for_sampled_token_relay ``` The test uses the base scheduler with: - V2 model runner - PP=2 - speculative decoding enabled - async scheduling disabled It simulates the final prefill step, injects speculative draft tokens, and checks that the request is not scheduled on the next step while the sampled token is still being relayed. After the sampled token is settled, the request becomes eligible at the expected PP step and schedules the sampled-token anchor together with the draft tokens. ## Test results Targeted regression: ```text tests/v1/core/test_scheduler.py::test_v2_pp_spec_decode_waits_for_sampled_token_relay 1 passed ``` Related scheduler tests: ```text tests/v1/core/test_async_scheduler.py 16 passed tests/v1/core/test_scheduler.py -k \"spec\" 17 passed ``` `git diff --check` also passes. ## GPU E2E I also ran a before/after GPU repro on 2x RTX 4090. To keep the repro small while still exercising the real PP + MTP path, I used the GLM-4.7-Flash architecture with `load_format=\"dummy\"` and reduced the model through `hf_overrides`. The test configuration was: ```text V2 model runner async_scheduling=False MTP speculative decoding num_speculative_tokens=1 4 transformer layers 2 routed experts greedy decoding 16 output tokens ``` The same setup was tested with PP=1 and PP=2. ### Before the fix | Configuration | Result | | --- | --- | | PP=1 + V2 + MTP + async=False | PASS | | PP=2 + V2 + MTP + async=False | FAIL — CUDA illegal memory access | The PP=1 run completed normally and produced 15 speculative draft steps. The PP=2 run did not produce an output and failed in the worker with: ```text torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` ### After the fix | Configuration | Result | | --- | --- | | PP=1 + V2 + MTP + async=False | PASS | | PP=2 + V2 + MTP + async=False | PASS | Both runs generated all 16 requested tokens and speculative decoding remained active: ```text PP=1: output_length = 16 num_drafts = 15 PP=2: output_length = 16 num_drafts = 15 ``` The PP=1 and PP=2 runs also produced identical token IDs: ```text [55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495, 55582, 18495] ``` So with the same model setup and scheduler configuration: | | Before fix | After fix | | --- | --- | --- | | PP=1 | PASS | PASS | | PP=2 | CUDA illegal memory access | PASS |", "url": "https://github.com/vllm-project/vllm/pull/52179", "createdAt": "2026-08-13T14:33:36Z", "updatedAt": "2026-08-13T14:47:42Z", "timestamp": "2026-08-13T14:47:42Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "icarus0adios-netizen", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:3f5f3180cd205474dc0c", "signalId": "github:vllm-project/vllm:pull_request:48673", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:48673", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "test(quantization): cover tied lm_head/embed_tokens when lm_head excluded from ModelOpt", "text": "### Summary Adds regression coverage for tying an excluded `lm_head` to `embed_tokens` through the quantization method path (`QuantizeMethodBase.tie_weights`), the behavior that fixed the original NVFP4 crash on main. Test only. Related to #48238", "url": "https://github.com/vllm-project/vllm/pull/48673", "createdAt": "2026-07-14T23:59:09Z", "updatedAt": "2026-08-13T14:47:20Z", "timestamp": "2026-08-13T14:47:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "quantization" ], "author": "pjdurden", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:72c518eccbd970d02e0c", "signalId": "github:vllm-project/vllm:pull_request:51959", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51959", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] DeepGEMM pin has no SM120 kernels: family-12 Blackwell cannot run hyperconnections", "text": "## The problem DeepGEMM dispatches the TF32 hyperconnection pre-norm GEMM on the device arch major, in `csrc/apis/hyperconnection.hpp`. At the revision this repository pins, `vllm-project/DeepGEMM` `e21c821`, that dispatch handles `arch_major` 9 and 10 only: ```cpp const auto arch_major = device_runtime->get_arch_major(); if (arch_major == 9) { sm90_tf32_hc_prenorm_gemm(...); } else if (arch_major == 10) { sm100_tf32_hc_prenorm_gemm(...); } else { DG_HOST_UNREACHABLE(\"Unsupported architecture\"); } ``` That revision ships no `sm120_*` implementation at all. `csrc/jit_kernels/impls/` contains `sm90_*` and `sm100_*` only. `tools/install_deepgemm.sh` currently comments that the pin targets nv-dev \"due to sm120 support\", but the pinned tree does not carry it. So every SM120/SM121 device aborts as soon as a model exercises hyperconnections. DeepSeek-V4 Flash is one: it is configured with `hc_mult=4`, so the path is hit on the first forward pass. ## The change Points both pins at `deepseek-ai/DeepGEMM` `a6b593d`, which carries `sm120_bf16_gemm`, `sm120_bmk_bnk_mn`, `sm120_fp8_fp4_gemm_1d1d` and `sm120_tf32_hc_prenorm_gemm`, and whose dispatch has the `arch_major == 12` branch. `cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh` both document that they must stay in sync, so both are updated. To be explicit: only the cmake path is exercised by my build; the `install_deepgemm.sh` edit is the same two values kept in sync, but I have not run that script. ## Testing With this change, DeepSeek-V4-Flash-0731 builds and serves on 2× NVIDIA GB10 (SM121, aarch64, CUDA 13.2, torch 2.13.0+cu132) with `--tensor-parallel-size 2` across two nodes. That is the configuration currently running here, and the fetched tree does contain the four `sm120_*` kernels. The failure itself does not really need a reproduction, since it is visible in the pinned tree: `csrc/apis/hyperconnection.hpp` at `e21c821` has no `arch_major == 12` branch, and `csrc/jit_kernels/impls/sm120_tf32_hc_prenorm_gemm.hpp` does not exist at that revision. For the record, we did hit `DG_HOST_UNREACHABLE(\"Unsupported architecture\")` on the old pin before switching, but I no longer have that log and am not claiming a fresh reproduction. Note it is a runtime abort rather than a build error: DeepGEMM JIT-compiles its kernels, so the build succeeds and the process dies on the first forward pass reaching the dispatch. Recipe, scripts and raw measurements from that setup: https://github.com/Mirrdhyn/dsv4-flash-dgx-spark ## Caveat, and a question for maintainers This is the minimal change that demonstrably fixes SM12x, but it may not be the change you want. The two hyperconnection kernels shared by both trees are byte-identical (`sm90_tf32_hc_prenorm_gemm.hpp` `c17d1b5`, `sm100_tf32_hc_prenorm_gemm.hpp` `0071e2c`), so the hyperconnection path itself is unaffected on SM90 and SM100. Other kernels have diverged, however: `sm90_bf16_gemm.hpp` differs between the two revisions, which suggests the fork carries vLLM-specific work that repointing would drop. I have no SM90 or SM100 hardware and have not tested those architectures. If the fork does carry changes worth keeping, the better fix is to port the `sm120_*` kernels into `vllm-project/DeepGEMM`, or to rebase the fork onto a newer upstream, and leave the pin where it is. I am happy to redo the patch that way. Either path is fine by me; the point of this PR is that SM12x is currently broken and the fix is small.", "url": "https://github.com/vllm-project/vllm/pull/51959", "createdAt": "2026-08-12T08:40:00Z", "updatedAt": "2026-08-13T14:47:11Z", "timestamp": "2026-08-13T14:47:11Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "needs-rebase", "ci/build" ], "author": "Mirrdhyn", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:60011011240a9740133b", "signalId": "github:vllm-project/vllm:pull_request:52132", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52132", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Enhance engine snapshot management and API lifecycle", "text": "## Purpose Add an opt-in Engine Snapshot deep-sleep path that keeps the API frontend, HTTP listener, and launcher alive while snapshotting and exiting the `EngineCoreProc` subtree. On wake, vLLM validates the snapshot and runtime environment, restores the EngineCore and CUDA state, adopts the restored process, reconnects the frontend, and reopens request admission only after health and semantic checks pass. Design, roadmap, compatibility constraints, and complete experimental results are documented in [RFC #52125: vLLM Native Reusable Engine Snapshot Sleep (L3)](https://github.com/vllm-project/vllm/issues/52125). This work overlaps with #51360 in its use of CRIU and CUDA Checkpoint/Restore for an initialized engine. The snapshot boundary and user-facing lifecycle differ: - #51360 captures the complete initialized service process tree before the HTTP listener is bound and restores it during instance startup. - This PR implements runtime deep sleep. The frontend and HTTP listener remain available for status and wake requests while only the EngineCore subtree is captured, terminated, restored, and reattached. ## Change Summary - Add the Engine Snapshot lifecycle: `READY → DRAINING → PREPARING → SNAPSHOTTING → HIBERNATED` and `HIBERNATED → RESTORING → ATTACHING → VERIFYING → READY`. - Add idle-only request admission control and readiness handling around snapshot transitions. - Add EngineCore detach, expected-exit handling, restored-process adoption, and frontend reconnection. - Add `fake` and `criu_cuda` snapshot providers. - Add atomic snapshot artifact commit, manifest validation, diagnostics, rollback, and cleanup. - Expose two L3 resource policies: - `full`: preserve weights, KV cache, and runtime state in the CUDA image. - `minimized`: discard weights and KV cache before capture, then reload weights from model files and rebuild KV cache after restore. - Add development-mode CLI configuration and the Level 3 sleep, wake, and status endpoints. - Add allocator, worker, process-lifecycle, provider, manager, API, middleware, CLI, and serving tests. - Document prerequisites, supported scope, endpoints, snapshot contents, and deployment restrictions. ## Current Scope The initial implementation supports: - Same-machine restore. - One host and one GPU. - `TP=1`, `PP=1`, and `DP=1`. - One Python API server using `UniProcExecutor`. - Idle-only capture with no in-flight request preservation. - `fake` and CRIU/CUDA providers. - Development-mode enablement through `VLLM_SERVER_DEV_MODE=1`. The feature is opt-in and does not change the default `vllm serve` process topology or request path. Cross-machine restore, 1→N restore, multi-GPU snapshot groups, distributed topologies, and in-flight request preservation remain follow-up work described in the RFC. ## Test Plan - Run Ruff on the changed Python files. - Run Python 3.10 mypy on the changed typed modules. - Run the snapshot unit-test suite: `pytest -q tests/snapshot` - Run the Engine Snapshot CLI parser tests: `pytest -q tests/entrypoints/openai/test_cli_args.py -k engine_snapshot` - Exercise the lifecycle and failure paths with the `fake` provider. - Run repeated real CRIU/CUDA capture and restore cycles for the `full` and `minimized` policies. - Verify restored inference output, process cleanup, physical HBM release, snapshot artifacts, and phase-level timing. ## Test Result Test environment: - NVIDIA H20, single GPU. - Qwen3-8B BF16. - `TP=1`, `PP=1`, `DP=1`, one API server. - Non-eager execution. - Model, compilation cache, CRIU images, and snapshots stored on tmpfs. - Six runs per group: one warmup followed by five measured runs. - `minimized` was repeated across three groups for 15 measured runs. Median results: | L3 policy | Measured runs | Checkpoint time | Restore time | Artifact size | | --- | ---: | ---: | ---: | ---: | | `minimized` | 15 | 2.837 s | 5.948 s | 4.251 GiB | | `full` | 5 | 27.242 s | 22.990 s | 49.659 GiB | Restore time is measured from the `/wake_up` request to the first non-empty token from a fixed streaming request, with the final response content also checked. Correctness and resource results: - All reported `full` and `minimized` runs restored successfully and passed the fixed semantic checks. - Every pre-restore model page-residency check reported 100%. - Engine `read_bytes` delta was 0 in all reported runs. - Every reported `HIBERNATED` check showed 0 MiB for both project-process HBM and full-GPU HBM. - Ruff passed. - Python 3.10 mypy passed. - Snapshot tests: `75 passed`. - CLI parser tests: `3 passed, 30 deselected`. These validation results predate the final public-policy simplification commit and are not presented as a rerun of the current tree. ## AI Assistance AI assistance was used for implementation, testing, analysis, and drafting. The submitter reviewed the changes and owns the final submission.", "url": "https://github.com/vllm-project/vllm/pull/52132", "createdAt": "2026-08-13T08:52:39Z", "updatedAt": "2026-08-13T14:43:13Z", "timestamp": "2026-08-13T14:43:13Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "frontend" ], "author": "fuhefei", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:c19f8ae51f9253e529e8", "signalId": "github:vllm-project/vllm:pull_request:51901", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51901", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI/Build] Add warning for unsupported global PTX architecture requests in...", "text": "... CMake configuration. ## Purpose Implements one item from #9129: Warn that PTX builds are not currently supported (post [CI/Build] Per file CUDA Archs (improve wheel size and dev build times) #8845), currently if there is a +PTX in TORCH_CUDA_ARCH_LIST this will be ignored. We should warn when this is the case Users can request PTX through `TORCH_CUDA_ARCH_LIST` values such as `8.0+PTX`. vLLM strips the Torch-provided global `-gencode` flags and rebuilds per-source gencode flags, so the global PTX request is not preserved. This PR surfaces that behavior during CMake configuration instead of silently dropping the request. This branch builds on PR #42375 by: 1. Renaming `clear_cuda_arches` to `clear_cuda_gencode_flags` to clarify that it clears `-gencode` flags from `CMAKE_CUDA_FLAGS`, not the CUDA target architectures list itself. 2. Adding documentation to [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) to explain the CUDA architecture and PTX flags normalization behavior. ## Changes - **Refactor**: Renamed `clear_cuda_arches()` macro to `clear_cuda_gencode_flags()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) for clarity. - **Feat**: Added `warn_if_ptx_arch_requested()` in [cmake/utils.cmake](file:///home/swidanagama/vllm/cmake/utils.cmake) and called it in [CMakeLists.txt](file:///home/swidanagama/vllm/CMakeLists.txt) to warn users when PTX code generation is requested via global CUDA architecture flags. - **Tests**: Added pytest unit test in [tests/test_cmake_utils.py](file:///home/swidanagama/vllm/tests/test_cmake_utils.py) for `clear_cuda_gencode_flags()`. - **Docs**: Added a note in [docs/getting_started/installation/gpu.cuda.inc.md](file:///home/swidanagama/vllm/docs/getting_started/installation/gpu.cuda.inc.md) explaining CUDA architecture and PTX flags normalization. ## Test Plan Tested behavior by: 1. Running pytest unit tests in [tests/test_cmake_utils.py](tests/test_cmake_utils.py): ```bash .venv/bin/python -m pytest tests/test_cmake_utils.py -v ``` 2. Running a local install command to verify that the build environment compiles and registers: ```bash VLLM_USE_PRECOMPILED=1 uv pip install -e . ``` ## Test Result ### 1. CMake Utility Pytest Verification Output ```text $ .venv/bin/python -m pytest tests/test_cmake_utils.py -v ============================= test session starts ============================== platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/swidanagama/vllm/.venv/bin/python cachedir: .pytest_cache rootdir: /home/swidanagama/vllm configfile: pyproject.toml plugins: asyncio-1.4.0, anyio-4.14.2 asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 3 items tests/test_cmake_utils.py::test_exact_family_arch_precedes_generic_family_fallback PASSED [ 33%] tests/test_cmake_utils.py::test_extract_archs_prefers_sass_target_over_corrupted_virtual_arch PASSED [ 66%] tests/test_cmake_utils.py::test_clear_cuda_gencode_flags PASSED [100%] ======================== 3 passed in 0.95s ======================== ``` ### 2. Local Install Result ```text Uninstalled 4 packages in 19ms Installed 164 packages in 939ms ... + vllm==0.26.1rc1.dev652+g066f84253.precompiled (from file:///home/swidanagama/vllm) ``` ## AI assistance disclosure This PR description and code changes were prepared with AI assistance. The submitting contributor has reviewed every changed line and run the relevant validation tests.", "url": "https://github.com/vllm-project/vllm/pull/51901", "createdAt": "2026-08-12T00:11:47Z", "updatedAt": "2026-08-13T14:40:47Z", "timestamp": "2026-08-13T14:40:47Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "documentation", "ready", "ci/build", "nvidia" ], "author": "shanewidanagama", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4415e84d4d0a09b73465", "signalId": "github:vllm-project/vllm:pull_request:51924", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51924", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[MoE] Refine FlashInfer one-sided All2All integration", "text": "## Purpose Refine the FlashInfer NVLink one-sided All2All integration for DeepSeek Blockwise FP8 MoE and sequence parallelism. - Describe one-sided activation payloads explicitly in bytes for BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8. - Dispatch E4M3 activations with FP32 1x128 scales and feed the received layout directly to the FlashInfer TRT-LLM DeepSeekFp8/BlockMajorK kernel. - Validate activation and scale shapes before converting scales from `[M, K/128]` to the TRT-LLM `[K/128, M]` layout. - Enable MoE sequence parallelism for `flashinfer_nvlink_one_sided`, avoiding routing and compute on TP-replicated post-attention tokens. - Document and exercise DeepSeek Blockwise FP8 support in the existing MoE and MNNVL test matrices. No kernel-warmup changes are included because current upstream `main` already provides generalized distributed FlashInfer autotuning and persistent-cache synchronization. ### Duplicate-work check No issue number was supplied, so an issue-specific lookup was not applicable. I checked open PRs with: ```bash gh pr list --repo vllm-project/vllm --state open --search 'FlashInfer one-sided All2All' gh pr list --repo vllm-project/vllm --state open --search 'DeepSeek Blockwise FP8 FlashInfer' gh pr list --repo vllm-project/vllm --state open --search 'sequence parallel FlashInfer one sided' ``` The related open PRs are distinct: - #47733 handles per-tensor FP8 scaling, invalid expert IDs, and communicator cleanup; it does not add E4M3 activation dispatch with FP32 1x128 scales, TRT-LLM BlockMajorK integration, or sequence parallelism. - #42034 only changes the one-sided padding sentinel for local expert maps. - #42133 handles delayed MXFP8 scale swizzling for FlashInfer CUTLASS. ## Test Plan - Run all pre-commit hooks on the changed files. - Compile every changed Python file. - Check the patch for whitespace errors. - Validate DeepSeek Blockwise FP8 payload sizing and TRT-LLM scale layout with focused assertions. - Run a reproducible model evaluation on a public checkpoint. ## Test Result Passed all configured pre-commit hooks on the changed files, including ruff check/format, typos, markdownlint, mypy, SPDX, forbidden-import, configuration, and API checks: ```bash .venv/bin/pre-commit run --files \\ docs/design/moe_kernel_features.md \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py ``` Passed: ```bash .venv/bin/python -m py_compile \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py \\ vllm/config/parallel.py \\ vllm/distributed/device_communicators/all2all.py \\ vllm/model_executor/layers/fused_moe/all2all_utils.py \\ vllm/model_executor/layers/fused_moe/experts/trtllm_fp8_moe.py \\ vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided.py git diff --check origin/main...HEAD ``` A focused Python assertion script covering BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8 dispatch byte layouts plus the TRT-LLM `[M, K/128]` to `[K/128, M]` scale conversion passed. The repository-wide collection command below could not run in this CPU development environment because the installed `xgrammar` lacks `normalize_tool_choice` and the CUDA flash-attention extension is unavailable: ```bash .venv/bin/python -m pytest -q --collect-only \\ tests/distributed/test_mnnvl_alltoall.py \\ tests/kernels/moe/test_moe_layer.py ``` ### Model evaluation Full 1,319-example GSM8K on public `Qwen/Qwen3.5-35B-A3B-FP8`, using 8 B300 GPUs with TP2 x DP4 / EP8. The runtime used this PR plus the Qwen3Next SP correctness fix from #50685; that fix is not part of this PR. Common settings: `flashinfer_trtllm`, FlashInfer autotune enabled, `max_num_batched_tokens=4096`, `max_num_seqs=64`, 5-shot greedy decoding, `max_gen_toks=16384`, and 256 concurrent requests. Only the All2All backend changed. The task used `\"\\nQuestion:\"` instead of stock lm-eval's broad `\"Question:\"` stop to avoid terminating inside model reasoning. | All-to-all backend | Flexible exact match | Strict exact match | |---|---:|---:| | `allgather_reducescatter` | 95.75% (1263/1319) | 92.49% (1220/1319) | | `flashinfer_nvlink_one_sided` | 95.00% (1253/1319) | 90.98% (1200/1319) | Both runs completed without retries or engine errors. The flexible-score gap is 10 samples and is not statistically significant (`p=0.143`). Per-sample review and a repeated AG/RS run found trajectory and formatting variation, but no systematic one-sided correctness failure. ## AI assistance AI assistance was used for code exploration, cherry-pick conflict resolution, compatibility review, validation, and PR drafting. The human submitter must review every changed line, understand the change end-to-end, and be able to defend it before merging. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and commands. - [x] The current test results and model-evaluation status. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/51924", "createdAt": "2026-08-12T04:41:12Z", "updatedAt": "2026-08-13T14:37:16Z", "timestamp": "2026-08-13T14:37:16Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "documentation", "ready", "nvidia", "verified" ], "author": "bobboli", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e35da632d5e648b83420", "signalId": "github:vllm-project/vllm:issue:51313", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:51313", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Kimi-K3 with --kv-cache-dtype fp8 is unusable on H200/Hopper — assertion demands use_prefill_query_quantization, but that flag is silently ignored on non-Blackwell devices", "text": "## Summary Serving **Kimi-K3** with `--kv-cache-dtype fp8` on **H200 (Hopper, SM 9.0)** fails at runtime with an assertion telling the user to enable `--attention-config '{\"use_prefill_query_quantization\": true}'`. **Setting that flag does not resolve the error** — the same assertion fires again. On any non-Blackwell (non-capability-100) device the flag is silently ignored, so the error is circular and unsatisfiable on Hopper. ## Environment - **vLLM:** nightly (`vllm/vllm-openai:nightly`); code line references below are from `main` @ `bc4e425883` (2026-08-06). - **Model:** `moonshotai/Kimi-K3` (nvidia MLA path, `vllm/models/kimi_k3/nvidia/mla.py`). - **Hardware:** 4 nodes × 8× **H200** (Hopper, compute capability **9.0**). - **Serve flags (relevant):** `--kv-cache-dtype fp8`, `--attention-backend FLASHMLA`, `--moe-backend humming`, `--all2all-backend deepep_v2`, `--tensor-parallel-size 8`, `--enable-expert-parallel`. ## Error ``` ERROR [multiproc_executor.py:1029] AssertionError: Kimi-K3 fp8 KV cache requires an fp8 prefill query; enable --attention-config '{\"use_prefill_query_quantization\": true}'. ``` Adding `--attention-config '{\"use_prefill_query_quantization\": true}'` to the serve command does **not** fix it; the identical assertion still fires. ## Root cause The assertion at `vllm/models/kimi_k3/nvidia/mla.py:700-704` requires `fp8_prefill` (prefill query dtype == fp8) for a plain `fp8` (non-`fp8_ds_mla`) KV cache: ```python elif is_quantized_kv_cache(self.kv_cache_dtype): assert fp8_prefill, ( \"Kimi-K3 fp8 KV cache requires an fp8 prefill query; enable \" \"--attention-config '{\\\"use_prefill_query_quantization\\\": true}'.\" ) ``` `fp8_prefill` derives from `q_data_type`, decided in `determine_prefill_query_data_type` (`vllm/model_executor/layers/attention/mla_attention.py:1807-1848`): ```python use_fp8 = ( is_quantized_kv_cache(vllm_config.cache_config.cache_dtype) # True for \"fp8\" and vllm_config.attention_config.use_prefill_query_quantization # True (flag set) and backend_supports_prefill_query_quantization() # <-- False on H200 ) if use_fp8: return fp8_dtype elif vllm_config.attention_config.use_prefill_query_quantization: logger.info_once( \"Unable to perform FP8 prefill attention when\" \" use_prefill_query_quantization is enabled. ...\" ) return model_dtype # <-- returns bf16 => fp8_prefill = False => assertion fires ``` `backend_supports_prefill_query_quantization()` (`mla_attention.py:1541-1566`) returns `False` on Hopper for **two** independent reasons: ```python if not current_platform.is_device_capability_family(100): # H200 is SM 9.0 -> False, returns early return False ... return backend_cls.get_name() in (\"FLASHINFER\", \"TRTLLM_RAGGED\", \"TOKENSPEED_MLA\") # FLASHMLA not in list ``` `is_device_capability_family(100)` is `(current_capability // 10) == (100 // 10)`, i.e. **Blackwell (SM 10.x) only**. H200 (SM 9.0 → `90 // 10 = 9 != 10`) fails immediately, before the backend check. **Net effect:** on H200 (and any non-Blackwell device / unsupported prefill backend), `use_prefill_query_quantization=true` can *never* make `q_data_type` fp8, so the Kimi-K3 plain-`fp8` KV-cache assertion is unsatisfiable. The user is instructed to enable a flag that has no effect on their hardware — a circular error with no escape path from the message itself. ## Expected behavior Any one (ideally the first) of: 1. **Fail fast with an accurate message at config-validation time**: reject `--kv-cache-dtype fp8` for Kimi-K3 when the device is non-Blackwell or the prefill backend isn't in the supported set, and point users at `fp8_ds_mla` (bf16 prefill query) or a supported backend/device — instead of a misleading runtime assertion that recommends an ineffective flag. 2. The Kimi-K3 assertion message (`mla.py:701-704`) should reflect the actual device/backend gate (Blackwell + FlashInfer / TRT-LLM Ragged / TokenSpeed), not recommend `use_prefill_query_quantization` unconditionally. 3. `determine_prefill_query_data_type` should **warn or raise** when `use_prefill_query_quantization=true` but the device/backend can't honor it, rather than silently downgrading to `model_dtype` and letting a downstream assertion fire. ## Workaround (for other users hitting this) Use `--kv-cache-dtype fp8_ds_mla` (DS-MLA fp8, 656B per-tile self-scaled) instead of `--kv-cache-dtype fp8`. That branch (`mla.py:681-699`) uses a **bf16 prefill query** (it asserts `not fp8_prefill`) and works on Hopper without any prefill query quantization. Drop `--attention-config '{\"use_prefill_query_quantization\": true}'` when switching.", "url": "https://github.com/vllm-project/vllm/issues/51313", "createdAt": "2026-08-06T21:38:39Z", "updatedAt": "2026-08-13T14:37:06Z", "timestamp": "2026-08-13T14:37:06Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "elvircrn", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:8323bd02ea1ec943fa2e", "signalId": "github:vllm-project/vllm:pull_request:41834", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:41834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes", "text": "## Summary This PR enables DeepSeek V4 Flash on SM120/SM121 Blackwell client hardware by carrying the SM12x fallback and tuning stack needed for the current vLLM V1 path. It targets RTX PRO 6000 Blackwell Workstation Edition, RTX 5090-class SM120, and GB10 / DGX Spark SM121 users who cannot use SM100-only TMEM / `tcgen05` kernels. The branch is **reconciled on top of the merged #43477** and provides the **stock-deps path**: DeepSeek V4 on SM120/121 that builds and serves on *released* FlashInfer / DeepGEMM wheels, complementing #43477's route that needs the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches. It is kept **synced onto current `upstream/main`**. **Latest validated head: tag `sm120-pr-41834-stable-preview-20260809` (`aa0d513027`), synced onto `upstream/main` as of 2026-08-09 (`f18e10a7e1`)** — see [Update 2026-08-09](#update-2026-08-09) below. **The default model runner is now V2**; `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Model / speculative-decode status.** `deepseek-ai/DeepSeek-V4-Flash-0731` is the checkpoint this branch is validated on. It **removed the MTP heads** and folded the DSpark draft into the main checkpoint, so **DSpark (`method: \"dspark\"`, `num_speculative_tokens: 5`) is the speculative path**; MTP is supported only for older checkpoints that still carry those weights. Running without speculation is also fully supported and validated. ## Change footprint — model kernels vs. core-vLLM touch points 187 files, ~+29.1k / −1.3k against `upstream/main`, of which ~10.7k added lines are tests. The branch splits cleanly into model/kernel code and a small set of core-vLLM integration points: - **DeepSeek-V4 model + SM12x kernels — the enablement itself.** Everything under `vllm/models/deepseek_v4/**` plus the SM12x sparse-MLA decode / indexer / DeepGEMM kernels that live in shared dirs (`v1/attention/backends/mla/sparse_mla_kernels.py`, `model_executor/layers/sparse_attn_indexer.py`, `v1/attention/backends/mla/{indexer,sparse_swa}.py`, `utils/deep_gemm.py`, `kernels/mhc/tilelang.py`), the new DSv4 reasoning parser / tokenizer, and device tuning JSONs. - **C128A metadata device→host sync removed** (`models/deepseek_v4/sparse_mla.py`, perf) — `_c128a_effective_topk_width` takes the max position from the CPU-side `CommonAttentionMetadata.max_seq_len` instead of a per-step `int(positions.max().item())` device sync, dropping a launch-stream stall on every C128A metadata step. Decode is identical (`max_seq_len-1 == positions.max()`); only chunked prefill sees a safe, slightly-wider 128-aligned top-k. - **Core-vLLM integration — the hooks below.** Almost all are gated by model architecture / quant config / an env flag and are inert for other models. | Subsystem | Files | What it does | | --- | --- | --- | | KV-cache core | `single_type_kv_cache_manager.py`, `kv_cache_coordinator.py`, `kv_cache_manager.py`, `sched/scheduler.py` (+1) | prefix-cache correctness for DSv4 sparse-MLA + speculative decode: an MLA cache-manager with prompt-block protection, a hybrid-coordinator `cache_blocks` tail-block-reuse rewrite | | Speculative decode | `v1/spec_decode/{dspark,dspark_sampling,llm_base_proposer,dflash}.py`, `config/speculative.py` | DSpark self-drafting proposer + sampling; DSv4 probabilistic draft sampling and per-step draft-layer routing in the shared proposer base; DSpark/MTP method detection and validation | | MoE quantization | `fused_moe.py`, `oracle/mxfp4.py`, `routed_experts.py`, `experts/flashinfer_cutlass_moe.py`, `quantization/mxfp4.py`, `oracle/nvfp4.py` | MXFP4 / NVFP4 backend selection; the one-line NVFP4 fix (FLASHINFER_CUTLASS into the SwiGLU-clamp allow-list) lets DSv4-Flash-NVFP4 serve | | FP8 / Marlin GEMM | `quantization/utils/fp8_utils.py`, `linear/scaled_mm/{cutlass,marlin}.py`, `csrc/.../marlin_moe_wna16/ops.cu` (the only C++) | SM12x e8m0→fp32 upcast + Marlin MoE SM12.0a cudagraph hardening | | cudagraph / compile / config | `config/vllm.py`, `compilation/breakable_cudagraph.py`, `passes/utility/fix_functionalization.py`, `config/compilation.py` | breakable-cudagraph auto-enable gate (MiniMax-only; DSv4 deliberately excluded), DSv4 custom-op defunctionalization + splitting-op registration | | OpenAI entrypoints / parsers | `chat_completion/protocol.py`, `serve/render/serving.py`, `tool_parsers/structural_tag_registry.py`, `chat_utils.py`, `engine/protocol.py`, `chat_completion/{serving,batch_serving}.py`, `reasoning/__init__.py` | expose DSv4 API semantics — `reasoning_content` / `thinking` param / tool-call streaming (jasl/vllm#19 instruction-following) | | Kernel warmup | `model_executor/warmup/deepseek_v4_sm12x_warmup.py` (new), `kernel_warmup.py` (+11) | DSv4 warmup passes (D512-split prefill precompile, paged-MQA rowwise, draft path) that avoid JIT-during-inference wedges. Kept in a separate module so `kernel_warmup.py` stays a two-line hook on upstream's file | | Weight loading | `weight_utils.py`, `default_loader.py` | fast-safetensors weight filter + EP-skip (lowers DSv4 load overhead on GB10) | | env / utils | `envs.py`, `utils/flashinfer.py`, `utils/import_utils.py`, `v1/worker/{gpu_model_runner,ubatch_utils}.py` | `VLLM_DEEPSEEK_V4_*` flags + `has_cutedsl` / `has_flashinfer_trtllm_sparse_mla` probes | Two notes for review: - The most invasive generic edits were **removed** in the 2026-06-21 audit cleanup: the scheduler carries a single +1-line change (the prefill-fairness heuristics were dropped) and the prefix-cache write-fence is gone. - A few hooks touch code paths shared with non-DSv4 models and are worth a closer look: the `kv_cache_coordinator` `cache_blocks` rewrite (affects hybrid-KV models; validated ≥ prior behavior), the proposer base-class change, and the OpenAI-entrypoint plumbing. Everything else (MoE oracle, fp8_utils, cudagraph gate, warmup, envs) is arch / quant / env-gated and inert for other models. ## Duplicate-work check The nearest open/merged PRs are related but not duplicates: | PR | Difference | | --- | --- | | #43477 | **Merged 2026-06-22.** Enables DeepSeek V4 + GLM-5.1 on SM120 via the FlashInfer-SM120 sparse-MLA route, but on its merged form **requires the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches** — on released/stock wheels its SM12x path raises at model construction. This PR is **reconciled on top of #43477** (merge `42657aca65`) and carries the **stock-deps** DSv4 SM120/121 path that runs on released wheels. | | #40929 | Earlier WIP Triton fallback effort. This PR is the maintained replacement branch with the broader scheduler, prefix-cache, parser, quant, warmup, and harness-validated fixes carried forward. | | #42856 | Focused workspace-bound fix that explicitly depends on / references this PR; a subset-style bugfix, not the full DeepSeek V4 SM12x enablement branch. | | #49335 | mxfp8 activation-scale swizzle after DP/EP dispatch — **carried in this branch** (unclaimed upstream). Inert at DP=1; taken for this branch's multi-node DP users. | | #50686 | Consecutive-assistant-message merging in DSv4 prompt encoding — **carried in this branch** (reproduced here before taking it). | | #50693 | B300-targeted prefill-workspace fix. **Test carried, code not needed**: this branch's `_prefill_workspace_topk_bound` returns early for `compress_ratio <= 1` and never reaches the affected buffer. | Upstream PRs whose fixes this branch previously carried as local deltas and has since **retired** in favour of upstream's own version: #48304, #48911, #48959 (via #49052). ## Fixed preview tags These tags are in `jasl/vllm` and give users stable pins while the PR is still moving: | Tag | Commit | Notes | | --- | --- | --- | | `sm120-pr-41834-stable-preview-20260809` | `aa0d513027` | **latest validated head** — 87 upstream commits incl. FlashInfer 0.6.16.post3; the V2 recall collapse root-caused as a prefix-cache ghost-block race and fixed (port of #42359); **default runner switched to V2**. See **Update 2026-08-09**. | | `sm120-pr-41834-stable-preview-20260804` | `0f59188db1` | 35 further upstream commits, four fixes from community reports (DSpark out-of-vocab draft token, eager scratch pool), two contributor PRs. Validated on **both** SM121 and SM120. See **Update 2026-08-04**. | | `sm120-pr-41834-stable-preview-20260802` | `9a94c54292` | 234 upstream commits, `DeepSeek-V4-Flash-0731` support, two DSpark config fixes, #49335 / #50686 absorbed. See **Update 2026-08-02**. | | `sm120-pr-41834-stable-preview-20260727d` | `d64074e6f0` | 209-commit upstream sync + torch 2.13 (tag `…-20260727`, `70a33886bd`); DSpark VRAM work (jasl/vllm#27) merged; bounded block-table gather in `compute_global_topk_indices_and_lens`. | | `sm120-pr-41834-stable-preview-20260721` | `832775efd1` | 79-commit upstream sync; #48911 dropped in favour of upstream's merged version; compact CPU KV offload (opt-in). | | `sm120-pr-41834-stable-preview-20260717` | `f63bfd3d7b` | 195-commit upstream sync; prefill ctx_pp +4.7% @ d8192. | | `sm120-pr-41834-stable-preview-20260711` | `b5c0d43b96` | 181-commit upstream sync; #48304 MTP unscaled-draft-rope; ~1097-line dead-kernel cleanup. | | `sm120-pr-41834-stable-preview-20260704` | `b43470e871` | @GanyX19 GB10 fixes: per-shape `constexpr`→runtime (stops the Triton recompile → unified-memory leak → hard-freeze) + fp8-einsum `tl.multiple_of(16)` (~24% decode @256K). | | `sm120-pr-41834-stable-preview-20260703` | `444fe3ac8b` | DSpark spec-decode (self-drafting block-5), V2 padded-Q OOM fix (jasl/vllm#26), exact non-cooperative `persistent_topk` for <128 KB-smem parts. | Older tags (`…-20260705` back to `…-20260612…`) remain in `jasl/vllm` for history. ## Update 2026-08-02 — `DeepSeek-V4-Flash-0731`, 234 upstream commits, two DSpark fixes Validated head **`9a94c54292`** (tag `sm120-pr-41834-stable-preview-20260802`), 234 upstream commits absorbed, level with `upstream/main` as of 2026-08-02. **What's in it** - **`DeepSeek-V4-Flash-0731` support.** The new checkpoint ships **no MTP heads** — `enorm`, `hnorm`, `e_proj`, `h_proj` and `shared_head` are absent from the weight index, and `mtp.{0,1,2}.*` now carries the DSpark-style `main_norm` / `main_proj` structure (matching `dspark_target_layer_ids: [40, 41, 42]`). DSpark is the speculative path going forward; the MTP code is retained for older checkpoints. - **`num_speculative_tokens` vs `dspark_block_size`** — *this rule was relaxed on 2026-08-04; see Update 2026-08-04. It now errors only BELOW the block size and warns above it.* The original reasoning and measurements follow. - The validator was tightened to require equality. The validator previously accepted `>=` and its error message recommended exceeding it. The drafter emits exactly one block per pass, so the extra slots are structurally unreachable — measured on a prose workload, the **7th draft position accepted 0.000 in every sample** (the 6th in all but one, 0.004 there), and `nst=7` drafts 40% more tokens per step for strictly worse acceptance: | configuration | mean acceptance length (3 samples) | avg draft acceptance rate | | --- | --- | --- | | nst=5 probabilistic | 2.15 / 2.16 / **2.19** | 22.9 / 23.2 / **23.8%** | | nst=7 probabilistic | 1.61 / 1.75 / 1.95 | 8.7 / 10.7 / 13.6% | | nst=5 greedy | 1.82 / 2.06 / **2.23** | 16.4 / 21.2 / **24.5%** | | nst=7 greedy | 1.57 / 1.66 / 1.75 | 8.2 / 9.5 / 10.8% | All samples are shown rather than a single figure: the probe reads whatever `SpecDecoding metrics` lines vLLM flushed inside its window, so a low sample means \"not much steady traffic in that slice\", not a worse drafter. Both `nst=7` runs also hit connection errors partway through, so their spread is noisier. - **`method: \"mtp\"` is no longer silently rewritten to `\"dspark\"`.** Auto-detection preserved an explicitly requested method only for `eagle`/`eagle3`/`dflash`/`dspark`. Since 0731 puts `dspark_block_size` in every DSv4 config, `method: \"mtp\"` fell through to the dspark branch, was rewritten, and then failed validation with a DSpark message the user never asked for. - **Absorbed** #49335 (mxfp8 activation-scale swizzle after DP/EP dispatch, verified non-vacuous) and #50686 (consecutive assistant messages merged in prompt encoding — it also removes a spurious mid-turn `<|end_of_sentence|>` on the default `drop_thinking=True` path). #50693's regression test is carried; its code fix is not reachable here. - **Three community reports re-tested and not reproduced** on this head: nst=7 draft-position collapse, TP=4 + expert-parallel `Check failed: num_tokens > 64`, and `FLASHMLA_SPARSE_DSV4` missing `tile_sched`. Details in [this comment](https://github.com/vllm-project/vllm/pull/41834#issuecomment-5152562124). **Validation (GB10 SM121, 2-node TP=2, `DeepSeek-V4-Flash-0731`, torch 2.13.0, FlashInfer 0.6.15.post1, nccl 2.30.7)** | | DSpark nst=5 | no speculation | | --- | --- | --- | | GSM8K 8-shot (flexible) | 0.9394 | 0.9500 | | GSM8K 8-shot (strict) | 0.9363 | 0.9484 | | instruction-following (jasl/vllm#19, JSON-only) | PASS | PASS | | long-context recall (arthur needle, c=1) | 2/2 | 2/2 | | illegal-access / assertion in serve log | 0 | 0 | | draft acceptance (prose) | mean 2.08, 21.7% | — | The GSM8K difference (1.06 pp flexible / 1.21 pp strict) is within this gate's measured single-run spread (~1.1 pp). **Resolved:** three runs per cell were collected and the arms interleave, so it was noise. `0731` is the first checkpoint where the strict and flexible extractors disagree at all; on every prior baseline they were identical. Perf — pinned llama-benchy standard (fp8 KV, prefix-cache on, `FULL_AND_PIECEWISE`, mml 49152, util 0.85; C=1, 3 runs), against the **full** recorded range of the prior MTP2 baselines. This crosses a checkpoint boundary, so read it as a sanity band rather than a controlled A/B: | metric | prior MTP2 range (n=10) | 0731 + DSpark | vs band | | --- | --- | --- | --- | | pp2048 @ d8192 | 1339.11 – 1400.81 | **1432.23** ± 11.74 | **above** | | pp2048 @ d16384 | 1308.77 – 1344.68 | **1356.56** ± 11.78 | **above** | | pp2048 @ d32768 | 1089.05 – 1226.63 | **1250.75** ± 2.18 | **above** | | ctx_pp @ d8192 | 1757.16 – 1876.01 | 1816.97 ± 5.89 | inside | | ctx_pp @ d16384 | 1769.85 – 1842.16 | 1817.43 ± 1.43 | inside | | ctx_pp @ d32768 | 1595.87 – 1756.01 | 1740.22 ± 2.87 | inside | | tg128 @ d8192 | 36.27 – 43.08 | 41.72 ± 5.09 | inside | | tg128 @ d16384 | 34.59 – 43.14 | 37.92 ± 9.92 | inside | | tg128 @ d32768 | 32.77 – 42.91 | 34.88 ± 5.78 | inside | | ctx_tg @ d8192 | 38.52 – 43.01 | 39.37 ± 2.34 | inside | | ctx_tg @ d16384 | 39.29 – 43.07 | **35.07** ± 0.67 | **below, −10.7%** | | ctx_tg @ d32768 | 38.02 – 42.73 | 40.70 ± 6.85 | inside | **Batched prefill (pp2048) is above the historical band at all three depths** (+2.2% / +0.9% / +2.0%) — the only consistent directional move here. Clearing the max of ten prior runs at all three depths says more than any single one of those margins would: +0.9% is inside this metric's own resolution, so read the consistency rather than the magnitudes. No sign of DSpark being slower than MTP2 was. One caveat reported rather than buried: **`ctx_tg @ d16384` sits 10.7% below its historical minimum**, the only metric outside its band. It is non-monotonic against our own neighbouring depths (39.37 at d8192, 40.70 at d32768, where history has d16384 ≈ d8192), which points at a single-run artifact rather than a depth-specific regression. **Resolved:** repeated on later heads and it did not recur. A measurement caveat for anyone benchmarking this branch: the `±` in a benchy row is the spread of the three runs *inside one invocation*, and it runs 5–30× smaller than the build-to-build spread. This branch's own history spans 31% on tg128 @ d32768 and ~1.3% on ctx_pp, so anything under ~15% on tg or ~2% on ctx_pp is not resolvable this way. ## Update 2026-08-04 — four fixes from community reports, 35 upstream commits, and first SM120 validation Validated head **`0f59188db1`** (tag `sm120-pr-41834-stable-preview-20260804`). **This is the first head validated on both SM121 and SM120.** Every SM120 discrete-GPU result on this PR up to now was a contributor's measurement we could not reproduce. We have since rebuilt a 2× RTX PRO 6000 Blackwell box as a first-party SM120 target. ### Fixes - **DSpark's fused Markov sampler could emit an out-of-vocab token id** (`e171c51036`). `_dspark_markov_probs_blocks_kernel` stores `vocab_size` as the filler for a block with no active lane. On a fully-masked row — every candidate `-inf`, which structured-output constraints can produce — *no* block has an active lane, so every block stores the filler and the reduce kernel returns it verbatim as the sampled token. Nothing downstream bounded it: the runner clamped `input_ids` with `min=0` only, and the DSv4 hash-MoE router indexes `tid2eid[token_id * 6 + lane]` on a `[vocab_size, 6]` table. Result is an illegal memory access on every TP rank. This is the producer on the **V1** path, which is this branch's default. @alexbi29's report traced the same class of defect to the V2 samplers (#50843) — a real defect, but a different tree. Fixed by folding out-of-range to `0` (matching `torch.argmax` on such a row, so the fused kernel stays bit-identical to the eager reference) and making the runner clamp two-sided. Worth stating plainly for anyone with similar gates: **our own gates could not have caught this**. The fused path is skipped when `all_greedy`, and both our long-context recall gate and GSM8K are greedy, so they are structurally incapable of executing that kernel. The new regression test is explicitly non-greedy. - **Adopted #50844** (`3df857ba50`) — bound `token_id` before the `tid2eid` gather. Defence in depth; `prompt_token_ids` reach that gather directly when `--skip-tokenizer-init` disables the engine's vocab check. **Not** taking #50843 (V2-tree only, inert on our default) or #50845, which has a defect reported on its own thread. - **Eager scratch pool is now OFF by default** (`d42b8d9f55`, `b1ef3033f4`), opt-in via `VLLM_DEEPSEEK_V4_EAGER_SCRATCH_POOL=1`. @tobymao bisected output corruption under concurrent mixed prefill+decode to it: pool active 7/7 rounds corrupt, disabled 0/2. We first removed the cross-template aliasing (`max()` → `sum()` sizing with per-family offsets); **they tested that commit directly and it was still corrupt in round 1**. Their diagnosis is the useful part: the pre-pool code was race-free *for free* because per-call transients go through the caching allocator, whose cross-stream reuse is event-guarded — the pool reuses memory without that machinery, so no static partitioning fixes it. Making the cross-layer reuse safe needs producer-waits-on-consumer events against the real stream graph; until then, off by default. - **Two contributor PRs merged** — jasl/vllm#37 (tuned FP8 W8A8 config for `N=4096,K=12288` on RTX PRO 6000) and jasl/vllm#38 (hoist the E8M0 block-scale upcast out of the FP8 GEMM hot path, 13,561 kernel launches removed per 25 decode steps), both from @alexbi29. - **`num_speculative_tokens` rule relaxed.** Upstream removed its own assertion in #50869 as \"invalid\". They were right that erroring *above* `dspark_block_size` is wrong — two users on this thread run `nst=7` against `block_size=5` and it demonstrably works. The two directions are not symmetric, so this branch now **errors below** the block size (that genuinely garbles output) and **warns above** it, quoting the acceptance cost. Strictly more permissive than what shipped before. ### Validation Full gate battery on **both** architectures, same branch: | gate | SM121 (2× GB10, 2-node TP=2) | SM120 (2× RTX PRO 6000, TP=2) | | --- | --- | --- | | serve, DSpark nst=5, `--block-size 256` | ✅ | ✅ | | instruction-following (jasl/vllm#19) | PASS | PASS | | long-context recall, arthur c=1 | 2/2 | 2/2 | | long-context recall, arthur c=12 | 22, 23, 22 / 24 | 22, 23 / 24 | | GSM8K 8-shot flexible | 0.9484 / 0.9507 / 0.9492 | 0.9371 | | GSM8K 8-shot strict | 0.9462 / 0.9477 / 0.9462 | 0.9303 | | tool-calling, 135 cases | 256/270 (94.8%) | — | | illegal-access / assertion lines | 0 | 0 | The ~1.1 pp GSM8K difference between architectures sits inside this gate's measured single-run spread and spans different silicon, different memory architecture and a 3× smaller KV cache (6.25 GiB vs ~18.5 GiB). We are not claiming a difference from it. **`Check failed: num_tokens > 64` does not reproduce on this branch.** @fuzzifikation reported stock 0.26.0 dying there on SM120 at `--block-size 256`, correctly tracing it to the DSv4 decode dispatch requiring `page_block_size == 64`. On our SM120 box, at the same `--block-size 256`, the serve comes up and the assertion never appears — the DSv4 packed KV cache is laid out in 64-token pages independent of vLLM's logical block size, and FlashInfer derives `page_block_size` from tensor geometry rather than the engine config. The SM120 packed decode path is confirmed engaged in the same run. Note the same assertion has **two** distinct gates (`page_block_size` and `(num_heads, topk)`, the latter being #50720 / flashinfer#3989), so patching one and still seeing it means checking the other. ### Prefill: V1 vs V2 model runner The 2026-08-02 V1-vs-V2 comparison never measured throughput. It has now been measured, blocked and pre-registered — 10 blocks, both arms inside each node pair, exact sign-flip permutation test, Holm-corrected across the six prefill cells, with the decision rule committed before any data was collected: | metric | V2 / V1 | 95% CI | exact p | | --- | --- | --- | --- | | ctx_pp @ d8192 | **+1.18%** | [+0.82, +1.53] | 0.0020 | | ctx_pp @ d16384 | **+1.11%** | [+0.35, +1.87] | 0.0137 | | ctx_pp @ d32768 | **+1.61%** | [+1.17, +2.04] | 0.0020 | | pp2048 @ d8192 | **+4.18%** | [+3.28, +5.09] | 0.0020 | | pp2048 @ d16384 | **+4.18%** | [+3.36, +5.00] | 0.0020 | | pp2048 @ d32768 | **+4.33%** | [+3.56, +5.11] | 0.0020 | All six survive Holm; both node pairs agree in direction on every cell. **Decode is not resolved in either direction** — `tg128` was declared unresolvable before the run (its within-build spread equals its entire historical range) and is reported for the record only. **V1 was the default when this was written; that was reversed on 2026-08-09 — see Update 2026-08-09.** The reasoning below was correct on the evidence available at the time, and the collapse it describes was real; it turned out not to be a property of the runner. Kept unedited because how the conclusion failed is the useful part. V2 is ahead on prefill, KV headroom (+4.70 GiB) and draft acceptance (+6.6%), but its long-context recall under concurrency is unreliable in a way that is worse than a consistent deficit: across 14 independent serves on the same build and configuration, roughly two thirds land in a state that loses most of the needles (arthur c=12 as low as 3/24), while the rest match V1 at 22–24/24. The mode is fixed at startup and stable within a serve, and nothing we have found predicts or detects it. A deployment could run clean for days and restart into the bad mode. The cause is **not identified**. Eliminated so far: the eager-scratch cross-template aliasing, the upstream merges, and the eager scratch pool as a whole (pool on 2 good / 6 bad vs pool off 3 good / 3 bad over 14 serves — no effect). The startup logs of a good and a bad serve are structurally identical, which rules out \"a different code path was taken\". Anyone opting into V2 with `VLLM_USE_V2_MODEL_RUNNER=1` should know this. ### Measurement note Two errors from our own process, since they affect how the numbers above should be read. The n=8 sampling that originally established V2's recall deficit took eight gate runs from **one serve** — it measured within-serve variance while the quantity that actually varies is across-serve. Raising n on the wrong axis. The 14-serve figures above use the inverted design: many serves, few gates each. And the `±` in a benchy row is the spread *within one invocation*; it runs 5–30× smaller than the build-to-build spread. The blocked design above exists because of that: a coarse range screen over the same 10 blocks returns \"no measurable difference\" on all six prefill cells, while the paired test finds all six. Had the screen been the decisive statistic, this section would have concluded the opposite and been wrong. ## Update 2026-08-09 — the V2 recall collapse was a prefix-cache race, not the runner; V2 becomes the default 70 upstream commits (to `643c125fab`), and the long-standing reason this branch pinned V1 is gone: it was **an unfixed upstream bug**, not a property of the V2 model runner. ### The defect `FullAttentionManager.cache_blocks()` commits prefix block hashes to the shared `BlockPool` **at scheduling time, before the forward pass writes their KV**. A request admitted later in the same step can match those hashes and read unwritten values. `MambaManager` has guarded this since #29387; no other manager does. This is **vllm-project/vllm#42359**, open and unmerged. Two more reports look like the same triple on different models — **#50188** (prefix caching + MTP spec decode + fp8 KV, byte-identical repeat requests, RTX 5090 / Qwen3.6-27B-NVFP4) and **#43559** (closed without a merged fix). Anyone on `--enable-prefix-caching` with speculative decoding is exposed; DeepSeek-V4 is not special here. What makes it hard to catch: **the damage persists**. A serve that loses the race keeps serving from the poisoned blocks for its lifetime, so a later *serial* request fails too — which is why it looked like a per-serve \"mode\" rather than a race. It is also stochastic, roughly half of cold serves. ### Evidence Same binary, `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT` the only variable, cache populated by the real gate, **4 fresh serves per arm** (a single clean serve proves nothing at ~50% incidence), 3 arthur c=12 runs each: | | serve 1 | serve 2 | serve 3 | serve 4 | mean | min | | --- | --- | --- | --- | --- | --- | --- | | guard **off** | 22/23/24 | **6/5/3** | **14/9/10** | **7/8/7** | 11.5 | 3 | | guard **on** | 23/22/21 | 20/22/20 | 22/23/23 | 23/22/23 | 22.0 | 20 | Mann-Whitney U, p = 0.0043. Every serve's runner and guard state was read back from the serve log rather than assumed. **It also fixes V1**, which was not expected: V1's arthur c=12 goes 20.7 → 23.0 with the guard on (24.0 with prefix caching disabled entirely). V1's own 2–4 needle shortfall was the same defect, not an inherent concurrency margin. ### Runner arbitration, re-run on the fixed tree Same tree, same guard mode, **runner the only variable**: | | V1 | V2 | | --- | --- | --- | | arthur c=12, 4 serves × 3 | 22.3 / 21.7 / 21.7 / 21.0 → **21.67** | 22.0 / 20.7 / 22.7 / 22.7 → **22.00** | | pp2048 d8192 / 16384 / 32768 | 1427 / 1363 / 1216 | **1472 / 1421 / 1303** (+3.1% / +4.2% / +7.1%) | | tg128 mean | 39.95 / 41.16 / 35.25 | **41.56 / 49.61 / 45.18** | | e2e TTFT | 1437 / 1506 / 1689 ms | **1393 / 1444 / 1576 ms** | | GPU KV cache | 339,194 tok | **423,752 tok** (+24.9%) | | GSM8K strict / flexible | 0.9378 / 0.9401 | 0.9401 / 0.9439 | | issue19 · multi-needle · c=1 | PASS · 48/48, 0 leaks · 2/2 | PASS · 48/48, 0 leaks · 2/2 | Recall: p = 0.697, neither side with a single-digit serve. GSM8K differs by <0.4 pp against ~1.1 pp single-run noise. **V2 is not behind anywhere and leads on throughput, latency and KV headroom, so it becomes the default.** `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Correction:** the *\"V2 +6.6% draft acceptance\"* figure in Update 2026-08-04 does **not** survive re-measurement — 2.772 (V1) vs 2.710 (V2) on the same formula and sample size, i.e. a tie. It was measured while V2 was poisoned. ### If you are running this branch **Nothing to set — the guard is on by default where it matters.** `KVCacheCoordinator` enables it whenever prefix caching and speculative decoding are both active, which is the only configuration in which a block hash can be published before its KV is written *and* a second request admitted in the same step to match it. This correction matters: an earlier revision of this update shipped V2 as the default while leaving the guard off by default, which would have handed a plain serve the exact combination measured at mean 11.5 with a 3/24 floor. Both changes looked like improvements in isolation. If you pulled `c054feedac`, take `aa0d513027` instead, or set the variable yourself. To turn it **off** (it is a real escape hatch, pinned by a test): ```bash VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0 ``` `1` is upstream's semantics, gated on `use_eagle`; on DeepSeek-V4 that covers only **2 of 5** managers and leaves the main MLA path unguarded — measured, not assumed, via a startup log line this branch adds that reports how many managers are actually guarded. `2` covers every group and is what the engine selects. Regression sweep on the merged tree: 261 passed, 1 failed, that one failing identically on the pre-merge tree `4ebd1fb698`. ### The two endpoints disagreed about the same model, twice `ResponsesRequest.reasoning` took its type from the OpenAI SDK, whose `ReasoningEffort` stops at `xhigh`, so DeepSeek's documented top tier `max` was rejected by schema validation on `/v1/responses` while `/v1/chat/completions` accepted it. Worse, and on the default path: with no thinking kwarg, `DeepSeekV4Tokenizer.apply_chat_template` defaults thinking **on** while `DeepSeekV4ReasoningParser` defaults it **off** and selects `IdentityReasoningParser`. The model reasoned and its reasoning, with a bare ``, came back inside `output_text` as though it were the answer — whenever a request omitted `reasoning`, which is exactly what a stock OpenAI SDK sends. Chat was immune only because it normalises thinking state at the protocol boundary, and its own docstring says why: *\"so the tokenizer and reasoning parser see the same effective state\"*. Responses never called that hook. The derivation now lives in `deepseek_v4_chat_kwargs` and both request types call it, so a third endpoint cannot repeat it. Measured on both checkpoints with no workaround flag set, 21/21 each: | | `DeepSeek-V4-Flash-0731` | `DeepSeek-V4-Flash` | | --- | --- | --- | | silence: reasoning in its own field | PASS | PASS | | silence: no `` in the answer | PASS | PASS | | `effort: none` disables thinking | PASS | PASS | | six spellings × two endpoints | PASS | PASS | | `high` reasons deeper than `low` | +85% / +110% | +19% / +35% | 26 unit cases accompany it, 11 of which fail on the unpatched tree. `tests/reasoning` 440 passed, `tests/tokenizers_/test_deepseek_v4.py` 45 passed. ### Acceptance on the exact published SHA Everything above was re-measured on `d44e224ab9` — the commit this tag points at, after a second upstream sync (17 further commits, FlashInfer 0.6.16.post3) — not on an ancestor assumed to be equivalent. 17 of 18 checks pass: | check | result | | --- | --- | | four nodes clean at the SHA, FlashInfer 0.6.16.post3 | PASS | | `tests/v1/core` | 509 passed, 1 pre-existing failure | | default serve with **nothing set**: boots, runner V2, guard 5/5, no NameError | PASS | | arthur c=12 ×3 / c=1 | 22 / 20 / 23 · 2/2 | | GSM8K strict · issue19 · multi-needle | 0.9363 · PASS · 48/48, 0 leaks | | pp2048 d8192 | 1480.77 (arbitration V2 arm 1471.8) | | `VLLM_USE_V2_MODEL_RUNNER=0` → V1, guard still 5/5, c=1 2/2 | PASS | | `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0` → guard 0/5, c=1 2/2 | PASS | **The one non-pass, and what it turned out to be.** `tests/v1/spec_decode` does not complete on this hardware — it wedges under a 30-minute bound on this head and on `4ebd1fb698` alike. Narrowed to `test_max_len.py` and measured both ways: | how it is run | result | | --- | --- | | whole file, one pytest process | wedges after ~7 min, 5 of 11 done | | each case in its own process | **11 of 11 pass**, free memory steady at 117 GiB | No individual case is broken. Each stands up a full engine, and repeated create/tear-down inside one process does not release resources fast enough on a **single-GPU** unified-memory node. That also explains why both trees wedge and why they stop at different points. It remains **unverified coverage rather than a pass**; running one process per case produces a verdict instead of a hang. `test_async_scheduling_pp_allows_rescheduling_with_output_placeholders` is the same class: it builds `pipeline_parallel_size=2`, and a GB10 node has one GPU, so it fails at config construction. It is the only case in `tests/v1/core` that needs more than one GPU; the other 509 pass. ### What this arbitration does and does not cover Everything above was measured on **one configuration**: 2-node TP=2, `DeepSeek-V4-Flash-0731`, DSpark `num_speculative_tokens: 5`, fp8 KV, `max_model_len` 131072, prefix caching on, GB10 (SM121). The default now applies to **every** DSpark config, including shapes not measured here — TP=4, other context lengths, the NVFP4 checkpoint, single-node setups. The reasoning generalises better than the numbers do: the race is in block publication and is not specific to a model shape, and V2's advantage comes from KV headroom and scheduling rather than anything config-specific. But if you run a materially different shape and see something worse, `VLLM_USE_V2_MODEL_RUNNER=0` returns you to V1 and a report would be welcome — that is a gap in our coverage, not a claim we have ruled out. ## Running DSpark DSpark is DeepSeek's self-drafting speculative-decode variant; on `0731` the draft weights are carried in the main checkpoint, so no separate `--speculative-model` is needed. ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --max-model-len 49152 \\ --max-num-seqs 64 \\ --max-num-batched-tokens 8192 \\ --gpu-memory-utilization 0.85 \\ --enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\"}' ``` - `num_speculative_tokens` **must equal** the checkpoint's `dspark_block_size` (5). Larger values are rejected: they are never accepted and only waste draft compute. - `--kv-cache-dtype fp8` is mandatory — DSv4's `fp8_ds_mla` attention asserts an fp8 KV layout, so the default `auto` fails at model construction. Not DSpark-specific. - Runs on the **V1 runner by default** (correct long-context recall). `VLLM_USE_V2_MODEL_RUNNER=1` opts into the V2 DSpark speculator; V2's long-context recall is correct after the jasl/vllm#26 padded-Q fix. - If you measure draft acceptance yourself, **use prose**. On counting or repeated text the Markov head alone reaches 68–100% acceptance even with the neural draft path degraded, which hides real regressions entirely. ## Dependencies (stock-deps path) Pins on the current head: torch 2.13.0 (triton 3.7.1) · `flashinfer-python` / `flashinfer-cubin` 0.6.15.post1 · tilelang 0.1.12 · `nvidia-cutlass-dsl[cu13]` 4.6.0 · `quack-kernels>=0.6.1` · nvidia-nccl-cu13 2.30.7 (multi-node, see below). - **FlashInfer** is pinned in `requirements/cuda.txt` (`flashinfer-python` **and** the GitHub-release `flashinfer-cubin`, which must be the same version); it ships the SM120 packed sparse-MLA kernels, so a stock build picks them up with no manual install dance. - **GB10 / multi-node**: pin `nvidia-nccl-cu13==2.30.7` on every node. A rebuild silently reverts it to torch's bundled version, and a per-node mismatch hangs the NCCL handshake. - The SM120 **decode** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_DECODE`) and **prefill** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_PREFILL`) FlashInfer sparse-MLA paths **default on**; set either `=0` to fall back to the FlashMLA / Triton path. Both are availability-gated, so stock installs without the kernel degrade gracefully rather than raising. ## Running the NVFP4 checkpoint This branch also serves `nvidia/DeepSeek-V4-Flash-NVFP4` on SM12x (RTX PRO 6000 / GB10). The NVFP4 MoE auto-selects the FlashInfer CUTLASS backend (the SwiGLU-clamp model gate accepts it), so no `--moe-backend` flag and no special FlashInfer build are required: ```bash vllm serve nvidia/DeepSeek-V4-Flash-NVFP4 \\ --trust-remote-code --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --tokenizer-mode deepseek_v4 ``` Expert-parallel off (plain TP) is the supported path. Accuracy matches MXFP4 (GSM8K 8-shot ~0.96 on both SM120 and SM121). On SM12x NVFP4 is **not** a memory or throughput win versus MXFP4: NVFP4 weights are ~4 GiB/GPU larger, leaving less KV-cache room; single-stream prefill is marginally faster and aggregate decode marginally slower. Its value here is checkpoint availability / parity with the SM100 datacenter path — MXFP4 remains the better practical choice on consumer Blackwell. ## AI assistance disclosure AI assistants, including OpenAI Codex/GPT models and Anthropic Claude models, were used for code review, refactoring support, regression-script writing, and benchmark analysis. The branch was validated through human review plus the commands and harness artifacts listed above; every performance and accuracy number quoted was measured on real SM120/SM121 hardware.", "url": "https://github.com/vllm-project/vllm/pull/41834", "createdAt": "2026-05-06T15:17:15Z", "updatedAt": "2026-08-13T14:36:24Z", "timestamp": "2026-08-13T14:36:24Z", "metrics": { "reactions": 40, "comments": 441 }, "labels": [ "new-model", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "tool-calling", "deepseek", "kv-connector", "nvidia", "quantization", "mrv2" ], "author": "jasl", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4e645d54b4e236f6e06d", "signalId": "github:vllm-project/vllm:pull_request:51653", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51653", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Enable V2 model runner for Kimi-K3 on ROCm", "text": "## Purpose Kimi-K3 on ROCm was gated from using V2 model runner. After validation using the up-to-date upstream. V2 model runner is working as expected. ## Test Plan Command to start Kimi-K3 on mi355x ``` export VLLM_ROCM_USE_AITER=1 export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # enable/disable V2 as A/B test export VLLM_USE_V2_MODEL_RUNNER=0/1 vllm serve moonshotai/Kimi-K3 \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --load-format auto \\ --gpu-memory-utilization 0.95 \\ --mm-encoder-tp-mode data \\ --max-num-batched-tokens 4096 \\ --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}' \\ --enable-auto-tool-choice \\ --tool-call-parser kimi_k3 \\ --reasoning-parser kimi_k3 \\ --max-num-seqs 128 ``` ## Test Result Tested with lm-eval: ``` lm_eval --model local-chat-completions --model_args model=moonshotai/Kimi-K3,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=256,max_retries=10,max_gen_toks=4096,max_length=16384,timeout=60000 --batch_size auto --tasks gsm8k --num_fewshot 20 --apply_chat_template ``` | runner | run 1 | run 2 | |--------|-------|-------| | V1 | 96.66% | 97.04% | | V2 | 97.57% | 96.89% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51653", "createdAt": "2026-08-10T10:03:11Z", "updatedAt": "2026-08-13T14:34:54Z", "timestamp": "2026-08-13T14:34:54Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "ready", "kimi", "k3" ], "author": "vllmellm", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:dc548c74086bc6aa8b4d", "signalId": "github:vllm-project/vllm:issue:51921", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:51921", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] v0.27.0 engine permanently stalls after ~1 min idle on 4-node TP=4 (GB10/sm_121, aarch64): shm_broadcast writer starves, requests never reach scheduler", "text": "## Bug On 4-node TP=4 (GB10/sm_121a, aarch64), a v0.27.0 engine serves its first requests normally, then **permanently stalls after roughly one minute of engine idleness**. Requests after the gap never enter the scheduler (`num_requests_running` stays 0), the API keeps answering `/v1/models` and `/tokenize`, and the head's EngineCore logs repeat: ``` [shm_broadcast.py:802] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work ``` every 60s indefinitely. All four rank containers stay alive; all TP workers' last log lines are healthy post-warmup (graph capture complete). Only a full fleet restart recovers. Reproduced 3/3 boots. Keeping the API warm (`/v1/models` polling every 45s) does NOT prevent it; the stall follows engine-level idleness, not connection idleness. Sequence, timestamped from one boot: warmup collectives complete across all ranks (so the broadcast queue works) → first 1-2 completions succeed (~1.3s each) → ~1-2 min idle → all subsequent requests hang forever. ## Environment Official `vllm/vllm-openai:v0.27.0-aarch64` + two patches required to get this far on sm_121: DeepGEMM repinned to `2fd67329` (#51758) and the `masked_mha_available` fix (#51920). GLM-5.2 (GlmMoeDsaForCausalLM), TP=4 via `--nnodes 4 --node-rank N`, mp executor, `FLASHINFER_MLA_SPARSE_SM120` + FLASH_ATTN MLA prefill, `fp8_ds_mla` KV, MTP k=2 spec decode, dual-rail RoCEv2, `VLLM_USE_BREAKABLE_CUDAGRAPH=0` set. torch 2.13.0+cu130 bundled NCCL. Related: #51758 validated v0.27.0 on 2x Spark (TP=2 and PP=2) with a 400-request soak and zero hangs — consistent with our result that the first requests DO work; the failure needs the multi-node idle gap, and possibly the 4-node width. This looks adjacent to the pre-0.27 TP-hang family that release fixed for 2-node. Full per-rank logs, exact launch argv, and timings available; the cluster reproduces this in ~20 minutes per attempt and I can run instrumented builds or targeted env matrices on request.", "url": "https://github.com/vllm-project/vllm/issues/51921", "createdAt": "2026-08-12T04:03:50Z", "updatedAt": "2026-08-13T14:34:52Z", "timestamp": "2026-08-13T14:34:52Z", "metrics": { "reactions": 0, "comments": 10 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2544baf5a0cada052424", "signalId": "github:vllm-project/vllm:pull_request:51968", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51968", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[XPU][Tests] Make tests device-agnostic", "text": "## Purpose Two kernel tests hardcoded a CUDA device, so they gave no coverage on XPU even though the kernels they exercise are available there. This makes both resolve the device via `current_platform` instead. **`test_fused_recurrent_packed_decode.py`** — gated on `torch.cuda.is_available()`, so it skipped on every non-CUDA accelerator. Both `fused_recurrent_gated_delta_rule_packed_decode` and its `fused_recurrent_gated_delta_rule` reference are pure Triton with no CUDA-only intrinsics, so the gate was stricter than the code requires. **`test_merge_attn_states.py`** — hardcoded `device=\"cuda\"` in eleven places and gated `test_merge_attn_states` on `current_platform.is_cuda()`. `test_merge_attn_states_both_empty` had no guard at all, so on XPU it did not skip but failed outright with `AssertionError: Torch not compiled with CUDA enabled`. vllm-project/vllm-xpu-kernels#467 (\"align merge_attn_states kernel interface\") registers `merge_attn_states` for `torch::kXPU` under the same `_C` schema and argument order that `vllm._custom_ops.merge_attn_states` already calls, and `vllm/platforms/xpu.py` imports `vllm_xpu_kernels._C` at module load. The op therefore resolves on XPU with no test-side dispatch, so the only platform-specific thing left in that test was the device string. No separate code path, no per-platform reference and no FP8 skip are needed: the XPU kernel supports fp32/fp16/bf16 inputs and the FP8-output path (static per-tensor `output_scale`), matching the CUDA coverage. No production code changed, and CUDA behaviour is unchanged. ## Test Plan ``` pytest tests/kernels/test_fused_recurrent_packed_decode.py -v pytest tests/kernels/attention/test_merge_attn_states.py -v ``` ## Test Result Intel Arc Pro B70 (torch 2.13.0+xpu, Triton 3.7.2): | Test | Before | After | | --- | --- | --- | | `test_fused_recurrent_packed_decode` | 6 skipped | **6 passed** | | `test_merge_attn_states` | 6 failed, 2592 skipped | **2598 passed** | The 6 failures before are `test_merge_attn_states_both_empty`, which had no skip guard and hit `Torch not compiled with CUDA enabled`. No regression on NVIDIA L40S: `test_merge_attn_states` is 2598 passed both before and after; `test_fused_recurrent_packed_decode` was also checked on L40S. The `use_fp8=True` x `head_size=613` subset (216 cases) passes on XPU, confirming both the FP8 output path and the `head_size % pack_size` constraint.", "url": "https://github.com/vllm-project/vllm/pull/51968", "createdAt": "2026-08-12T09:09:05Z", "updatedAt": "2026-08-13T14:29:43Z", "timestamp": "2026-08-13T14:29:43Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "intel-gpu" ], "author": "pmanczak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1abe77a737e6ae60a2eb", "signalId": "github:vllm-project/vllm:pull_request:50594", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50594", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix MiniMax M3 prompt reasoning initialization", "text": "## Purpose Fixes #46042. Fixes #50549. MiniMax-M3 adaptive reasoning can leak a leading `` into streaming `delta.content`. The existing initialization scans the entire rendered prompt with `is_reasoning_end()`. Because the prompt can contain paired `` markers as instructions or examples, the parser may incorrectly conclude that reasoning has already ended before generation starts. This change: - adds a prompt-boundary-specific reasoning-state hook while preserving the existing behavior by default - initializes MiniMax-M3 from `thinking_mode`: `enabled` starts in reasoning, `disabled` starts after reasoning, and `adaptive` lets generated markers determine the state - uses the hook in Chat Completions, the Responses API, streaming parsing, and structured-output initialization - preserves the deferred `adaptive=None` state through the structured-output gate instead of falling back to a full-prompt scan - scopes generated-marker checks to output tokens so instructional prompt markers cannot activate the grammar early - constrains unresolved adaptive sampling to grammar-valid tokens plus every decoded reasoning-marker tokenization - replays adaptive direct-content tokens through the grammar, including ambiguous prefixes and speculative draft state - respects the final reasoning boundary when continuing an existing assistant message, including explicit open continuations under disabled mode, and scopes continuation marker scans to that final message only - preserves open and closed `thinking` content-part state during continuation - propagates incomplete Responses reasoning items to frontend and engine parsers with the same request-local state - clears continuation-only parser state and refreshes the frontend parser after Responses built-in tool turns rerender the prompt - forwards request-local reasoning mode and prompt state through offline `LLM.chat()` and `enqueue_chat()` into the synchronous engine - keeps batch reasoning parsers request-local so continuation state cannot leak between conversations - initializes each engine-side structured-output parser from its prompt exactly once - adds focused regressions for instructional markers, adaptive marker boundaries, structured output, and all MiniMax-M3 thinking modes ## Related work and scope - #45718 fixed split-token MiniMax-M3 marker parsing, but not prompt-boundary state initialization. - #50152 identified the same full-prompt scan failure and demonstrated the production symptom, but closed without merging. - #48550 also addresses prompt-mode initialization while adding broader speculative-decoding and tool-parser changes. - This PR limits the fix to prompt-boundary reasoning initialization across online and offline chat paths and Responses, plus the structured-output state transitions needed to preserve it. ## Test Plan - Run focused parser and structured-output regressions: `uv run pytest -q tests/reasoning/test_minimax_m3_reasoning_parser.py tests/v1/structured_output/test_reasoning_structured_output.py` - Run offline chat prompt-state regression: `uv run pytest -q tests/entrypoints/llm/test_offline_reasoning_state.py` - Run batch prompt-state regressions: `uv run pytest -q tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py -k batch_forwards_prompt_reasoning_state` - Run Responses continuation regressions: `uv run pytest -q tests/entrypoints/openai/responses/test_serving_responses.py` - Run static validation on the changed Python files. - Serve `MiniMaxAI/MiniMax-M3` with the patch and send adaptive and enabled streaming chat-completion requests, including a structured response format. - Confirm that reasoning markers never appear in `delta.content`, reasoning remains in the reasoning field, and the output grammar starts only after generated reasoning ends or advances the first direct adaptive content token. ## Test Result Passed: - MiniMax-M3 reasoning parser: 44 tests passed - reasoning-aware structured output: 21 tests passed - offline chat prompt-state forwarding: 1 test passed - batch prompt-state forwarding: 3 tests passed - Responses serving: 23 tests passed, 1 expected xfail - `ruff check`: all checks passed - `ruff format`: all changed files formatted - Python `compileall` - `git diff --check` The 92 focused tests were run in a CPU-only temporary environment with `VLLM_TARGET_DEVICE=empty`; the repository-wide GPU CI has not run yet. Live MiniMax-M3 evaluation is pending because this workspace has no GPU/model deployment. It should be completed before merge. ## AI assistance OpenAI Codex assisted with implementation and regression-test drafting. The commits include `Assisted-by` trailers. The submitter requested the changes after reviewing the proposed scope.", "url": "https://github.com/vllm-project/vllm/pull/50594", "createdAt": "2026-07-31T16:34:24Z", "updatedAt": "2026-08-13T14:29:12Z", "timestamp": "2026-08-13T14:29:12Z", "metrics": { "reactions": 2, "comments": 22 }, "labels": [ "bug", "structured-output", "frontend", "tool-calling" ], "author": "jinbagi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:277577ef4c446556079c", "signalId": "github:vllm-project/vllm:issue:51744", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:issue:51744", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vllm/vllm-openai:latest fails to start Gemma4 with Transformers 5.15.0", "text": "### Your current environment ```text Docker image: vllm/vllm-openai:latest vLLM: 0.27.0 Transformers: 5.15.0 Model: yasu-oh/gemma-4-31B-it-qat-NVFP4 Tensor parallel size: 2 KV cache dtype: fp8 ``` The Transformers version in the image can be confirmed with: ```bash docker run --rm \\ --entrypoint python3 \\ vllm/vllm-openai:latest \\ -c 'import vllm, transformers; print(\"vLLM:\", vllm.__version__); print(\"Transformers:\", transformers.__version__)' ``` Output: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` ### 🐛 Describe the bug The current official release image `vllm/vllm-openai:latest` contains: ```text vLLM: 0.27.0 Transformers: 5.15.0 ``` When serving: ```text yasu-oh/gemma-4-31B-it-qat-NVFP4 ``` vLLM fails during startup with: ```text transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via config.per_layer_config[i].head_dim. ``` The exception occurs in: ```text vllm/transformers_utils/model_arch_config_convertor.py ``` while accessing the Gemma4 `head_dim`. ### Workaround Downgrading only Transformers to 5.14.1 while keeping the same `vllm/vllm-openai:latest` base image and serving configuration resolves the issue. ```dockerfile FROM vllm/vllm-openai:latest RUN /usr/bin/python3 -m pip install --no-cache-dir 'transformers==5.14.1' ``` With Transformers 5.14.1: * the same model loads successfully, * the vLLM engine initializes successfully, * the API server starts successfully, * `/v1/chat/completions` returns HTTP 200, * 32 concurrent requests run successfully. ## Related PRs * #49797 ? Fix Gemma 4 for upcoming Transformers version * #49959 ? Handle Transformers 5.15 Gemma4 schema ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51744", "createdAt": "2026-08-11T00:42:58Z", "updatedAt": "2026-08-13T14:26:47Z", "timestamp": "2026-08-13T14:26:47Z", "metrics": { "reactions": 5, "comments": 14 }, "labels": [ "bug" ], "author": "yasu-oh", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a4caa9f56341a5b9d2d9", "signalId": "github:vllm-project/vllm:pull_request:52177", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52177", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[DBO] Add DBO support to the DeepEP V2 backend", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52177", "createdAt": "2026-08-13T14:25:20Z", "updatedAt": "2026-08-13T14:26:33Z", "timestamp": "2026-08-13T14:26:33Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [], "author": "SageMoore", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f6812284edd988aaf402", "signalId": "github:vllm-project/vllm:pull_request:52176", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52176", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][DBO] Run the profile run without DBO when DBO is enabled.", "text": "## Purpose Running the profile run with DBO effectively sets the max batch size to `max_num_tokens // 2`. If vllm bails out of DBO for any reason, it can end up in a situation where it's running with a batch size that is larger than `max_num_tokens // 2`. Deep EP v2 crashes in this case because an internal buffer wasn't sized correctly, but it's plausible that there is other undesirable behavior in the system when we encounter this scenario. The fix is to run the profile run with DBO explicitly disabled. ## Test Result ``` vllm serve deepseek-ai/DeepSeek-V2-Lite \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --all2all-backend deepep_highthroughput \\ --enable-dbo \\ --dbo-decode-token-threshold 32 \\ --dbo-prefill-token-threshold 32 ``` lm_eval result ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3798|± |0.0134| | | |strict-match | 5|exact_match|↑ |0.3783|± |0.0134| ```", "url": "https://github.com/vllm-project/vllm/pull/52176", "createdAt": "2026-08-13T14:24:06Z", "updatedAt": "2026-08-13T14:24:44Z", "timestamp": "2026-08-13T14:24:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug" ], "author": "SageMoore", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:4773456ecf32318a8dd1", "signalId": "github:vllm-project/vllm:pull_request:52175", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52175", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Cohere ChatV2 citation and tool handling issues", "text": "## Purpose Fix issues ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52175", "createdAt": "2026-08-13T14:21:42Z", "updatedAt": "2026-08-13T14:22:22Z", "timestamp": "2026-08-13T14:22:22Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "frontend", "needs-rebase", "tool-calling" ], "author": "andrewbcohere", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:1694296132bf5836482e", "signalId": "github:vllm-project/vllm:issue:48197", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:48197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[RFC]: StructuredOutputManager x Speculative Decoding Refactor", "text": "### Motivation. Simplify the contract between `Scheduler`, grammar backend, and `StructuredOutputManager`, especially when speculative decoding is used. `StructuredOutputManager` exists because constrained decoding often doesn't kick in until a model has finished reasoning. With speculative decoding, grammar-invalid draft tokens need to be rejected, draft tokens need to temporarily advance grammar state to generate bitmasks, rejected draft tokens should not advance grammar state. Reasoning end boundary can also fall in the middle of the draft token window, so only draft tokens after said boundary should advance grammar state. `Scheduler` and the `StructuredOutputManager` need to do a careful dance to make this is all correct. This leads to some bugs such as [#34650](https://github.com/vllm-project/vllm/issues/34650) -- only recently fixed thanks to [#44297](https://github.com/vllm-project/vllm/pull/44297). ``` Per request inside a scheduler step 1. StructuredOutputManager.should_advance(request) # Should we constrain some draft tokens? Grammar.validate_tokens(draft tokens) # Reject grammar invalid draft tokens 2. StructuredOutputManager.grammar_bitmask(...) # Generate bitmasks for the draft tokens 3. Verifier runs 4. StructuredOutputManager.should_advance(request) # Should we constrain some draft token? StructuredOutputManager.trim_reasoning_for_advance(verified tokens) # Trim the draft tokens that are not constrained Grammar.accept_tokens(verified tokens) # Advance grammar state ``` In fact, speculative decoding x constrained decoding remains fragile / subtly incorrect: 1. ~~`StructuredOutputManager.should_advance(request)` treats STRUCTURAL_TAG vs other constraint types differently. It defers non STRUCTURAL_TAG constraints. [#44297](https://github.com/vllm-project/vllm/pull/44297) only fix the strict tool calling case (which uses STRUCTURAL_TAG); [#34650](https://github.com/vllm-project/vllm/issues/34650) is still unfixed for JSON, REGEX, and other constraint types.~~ Fixed by https://github.com/vllm-project/vllm/pull/44993 2. ~~`StructuredOutputManager.should_advance(request)` also doesn't answer \"should we constrain some draft tokens?\" reliably. It incorrectly infers the draft token window from `num_output_placeholders`. ~~ Fixed by https://github.com/vllm-project/vllm/pull/44993 3. `Grammar.validate_tokens(draft_tokens)` only works on a best effort basis -- it doesn't validate any draft token unless all draft tokens should be constrained. We rely on `StructuredOutputManager.grammar_bitmask(...)` and rejection sampling to discard grammer-invalid tokens. ### Proposed Change. Let `StructuredOutputManager` handle the token validation and grammar advancement. Remove: - `StructuredOutputManager.should_advance(request)` - `StructuredOutputManager.trim_reasoning_for_advance(verified tokens)` Add: - `StructuredOutputManager.validate_tokens(draft tokens)` - `StructuredOutputManager.accept_tokens(verified tokens)` ``` Per request inside a scheduler step 1. StructuredOutputManager.validate_tokens(draft tokens) 2. StructuredOutputManager.grammar_bitmask(...) 3. Verifier runs 4. StructuredOutputManager.accept_tokens(verified tokens) ``` Internally, `validate_tokens()`, `grammar_bitmask()`, and `accept_tokens()` all use the same `_get_constraint_start()` helper to determine whether and where draft tokens should be constrained and grammar state advanced. Concretly, `_get_constraint_start()` computes the first token that should be grammar-constrained. - `validate_tokens()` preserves any unconstrained prefix and validates only the constrained suffix. - `grammar_bitmask()` emits unconstrained bitmaks before the boundary and constrained bitmasks after it. - `accept_tokens()` advances the grammar state only on the constrained suffix only This gives a simpler mental model for the system. The scheduler is responsible for request lifecycle and token plumbing. `StructuredOutputManager` is responsible for draft token validation, bitmask generation, and grammar backends. ~100 insertions, ~200 deletions of changes excluding test updates. Would fix issues like: #34650 and https://github.com/vllm-project/vllm/issues/43388 ### CC List. @njhill @mgoin @chaunceyjiang @benchislett @arpera ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/48197", "createdAt": "2026-07-10T02:13:01Z", "updatedAt": "2026-08-13T14:22:20Z", "timestamp": "2026-08-13T14:22:20Z", "metrics": { "reactions": 1, "comments": 5 }, "labels": [ "RFC" ], "author": "yzong-rh", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:9af549a55328fc3736ee", "signalId": "github:vllm-project/vllm:pull_request:48200", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:48200", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Refactor]: StructuredOutputManager x Speculative Decoding Refactor", "text": "## Purpose See https://github.com/vllm-project/vllm/issues/48197. ~100 insertions, ~200 deletions of non-test changes. Then ~400 insertions, ~600 deletions for test consolidation. Moves `tests/v1/spec_decode/test_mtp_structured_output.py` and `tests/v1/structured_output/test_reasoning_structured_output.py` into `tests/v1/structured_output/test_structured_output_manager.py`. Tests are carefully picked to be more systematic while maintaining existing coverage. We can defer test consolidation to another PR to ensure that refactor pass current tests. ## Test Plan ``` venv/bin/python -m pytest \\ tests/v1/structured_output/test_structured_output_manager.py \\ tests/v1/core/test_scheduler.py \\ tests/v1/core/test_async_scheduler.py ``` Same repro as in https://github.com/vllm-project/vllm/pull/44297: https://gist.github.com/yzong-rh/b01fb276feb8f457aa68275cd44cae10 - Expected to pass both on main and on this branch Similar repro but with JSON constaints instead of STRUCTURAL_TAG constraints (see https://github.com/vllm-project/vllm/pull/44993): https://gist.github.com/yzong-rh/ed62c5f8a9fc85fb9633bb630dcc0244 - ~~Failing on main~~ (Fixed by https://github.com/vllm-project/vllm/pull/44993) - Pass on this PR ## Test Result ``` 178 passed, 1 xfailed, 31 warnings in 109.18s (0:01:49) ``` Structural Tag constraints: ``` Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Successes: 50 (100.00%) Failures: 0 (0.00%) Successful tool calls: 50 ``` JSON Constraints: ``` Main: Using model: Qwen/Qwen3.6-35B-A3B Progress: S...SS..S.S.S.S.S......S..S.......S..S....SSS.S... Summary Total requests: 50 Valid schema-constrained responses: 34 (68.00%) Failures: 16 (32.00%) invalid_json: 16 This: Using model: Qwen/Qwen3.6-35B-A3B Progress: .................................................. Summary Total requests: 50 Valid schema-constrained responses: 50 (100.00%) Failures: 0 (0.00%) ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48200", "createdAt": "2026-07-10T02:24:24Z", "updatedAt": "2026-08-13T14:21:19Z", "timestamp": "2026-08-13T14:21:19Z", "metrics": { "reactions": 3, "comments": 5 }, "labels": [ "structured-output", "speculative-decoding", "v1" ], "author": "yzong-rh", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7f79a40c8bc85311f336", "signalId": "github:vllm-project/vllm:pull_request:52112", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52112", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm] Fix a few int4/int8 quantization errors", "text": "## Purpose Fix a few quantization bugs on ROCm introduced by #44120 to enable int4/int8 quantized models like cyankiwi/MiniMax-M3-AWQ-INT4, QuantTrio/Qwen3-235B-A22B-GPTQ-Int8. 1. Add asym quantization support for TRITON moe backend; 2. Add SWIGLUOAI_UNINTERLEAVE activation for TRITON moe backend; 3. Remove wrong assertion (GROUP_SIZE==-1) from MoeWNA16Method; ## Test Plan 1. VLLM_USE_BREAKABLE_CUDAGRAPH=0 vllm serve cyankiwi/MiniMax-M3-AWQ-INT4/ --block-size 128 -tp 4 --attention-backend TRITON_ATTN --tool-call-parser minimax_m3 --enable-auto-tool-choice --reasoning-parser minimax_m3 --moe-backend triton --max-model-len 32768 2. vllm serve QuantTrio/Qwen3-235B-A22B-GPTQ-Int8/ -tp 8 --moe-backend triton Both generate correct answers.", "url": "https://github.com/vllm-project/vllm/pull/52112", "createdAt": "2026-08-13T06:44:17Z", "updatedAt": "2026-08-13T14:20:57Z", "timestamp": "2026-08-13T14:20:57Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "rocm", "quantization" ], "author": "qli88", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:701b3006382a5e351277", "signalId": "github:vllm-project/vllm:pull_request:52101", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52101", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Distributed][MoonEP] BF16 PoC integration of MoonEP balanced EP backend", "text": "## Purpose First item of the MoonEP integration roadmap RFC #52095: a correctness-first BF16 proof of concept of MoonEP (https://github.com/MoonshotAI/MoonEP) as a vLLM all2all backend. Included: - `\"moonep\"` `All2AllBackend` registration and `has_moonep()` probe - `MoonEPAll2AllManager` owning the cached `moonep.Buffer` - `MoonEPPrepareAndFinalize` conforming to the modular prepare/finalize interface: pad to static capacity → `dispatch` → `prefetch_weight` for the planned redundant experts (the `MoonEPCommPlan` is stashed between prepare and finalize, the same pattern DeepEP-HT uses for its handle) → `combine` in finalize - Replicated `[E+B, …]` BF16 weight-layout helper and a reference segment-loop expert runner over `cu_seqlens` - `tools/moonep/validate_moonep_bf16_poc.py`: distributed validation of the full dispatch/prefetch/compute/combine path against a dense per-token reference MoE Known PoC limitations, tracked as follow-up items in #52095: BF16/unquantized only; eager only; reference segment loop rather than a grouped GEMM; expert weights replicated per rank rather than MoonEP's sharded symmetric-memory ownership; no FusedMoE layer/engine wiring yet (engine use of the backend is not supported by this PR). ## Test Plan On an NVLink symmetric-memory capable node with the `moonep` package installed: ```bash torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py --skew 8.0 torchrun --nproc_per_node=4 tools/moonep/validate_moonep_bf16_poc.py \\ --num-tokens 512 --capacity 512 --hidden-size 2048 \\ --intermediate-size 768 --topk 8 --num-experts 64 --skew 4.0 ``` ## Test Result All three configurations pass on 4× GB300 (EP=4, NVLink), every rank `global_ok=True`: | Config | max rel_err | |---|---| | Default (S=128, H=512, top-4 of 32 experts) | 7.7e-3 | | Heavy router skew (`--skew 8`, exercises the redundant-expert planner + prefetch slots) | 5.7e-3 | | Larger shapes (S=512, H=2048, top-8 of 64 experts, skew 4) | 1.2e-2 | All within BF16 tolerance (2e-2) against the dense per-token reference. No model evaluation is included: this backend is opt-in, is not wired into the engine by this PR, and does not affect any existing serving path or model output. ## Notes - Not a duplicate: no open PR or issue in this repo implements a MoonEP backend (searched \"MoonEP\" across PRs/issues; no implementation PRs exist). - I have reviewed and tested every changed line.", "url": "https://github.com/vllm-project/vllm/pull/52101", "createdAt": "2026-08-13T04:57:26Z", "updatedAt": "2026-08-13T14:20:27Z", "timestamp": "2026-08-13T14:20:27Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "nvidia" ], "author": "kaijunli-infr", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3df0649cf3719110ddfe", "signalId": "github:vllm-project/vllm:issue:47522", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:47522", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: MLA chunked-context prefill crashes on sm80 with Marlin FP8: kv_c_normed cast to packed-int32 weight dtype (`unsupported \\`a\\` scalar_type`)", "text": "### Your current environment - vLLM 0.23.0 (official image), PyTorch 2.x, Triton, NCCL 2.28.9 - 12× NVIDIA A100-SXM4-80GB (compute capability **8.0**, no native FP8) across 3 nodes - Model: GLM-5.2-FP8 (fp8 weights → `MarlinFP8ScaledMMLinearKernel` / `MARLIN Fp8 MoE`), PP=3 × TP=4, `--enable-expert-parallel` - Attention: `TRITON_MLA` decode backend, `FLASH_ATTN` MLA prefill backend, bf16 KV cache - (We run with sparse attention disabled via a local patch because of #35021, but this bug is in the generic MLA chunked-context path and should reproduce with any fp8 MLA model on sm80, e.g. DeepSeek-V3/R1-FP8 on A100.) ### 🐛 Describe the bug On sm80 with fp8 weights (Marlin), any request that takes the **chunked-context prefill path** — i.e. a prefix-cache hit or a prompt longer than `max_num_batched_tokens` — crashes the worker with: ``` RuntimeError: unsupported `a` scalar_type ``` in `torch.ops._C.marlin_gemm`, called from `kv_b_proj` inside `MLACommonImpl._compute_prefill_context`. Requests that don't hit that path (short prompts, no shared prefix, plain decode) work fine, so the server passes startup, warmup, and simple smoke tests, then dies on the first prefix-cache hit in real traffic. **Root cause** — `vllm/model_executor/layers/attention/mla_attention.py`, in `_compute_prefill_context`: ```python _kv_b_proj_w_dtype = ( self.kv_b_proj.weight.dtype if hasattr(self.kv_b_proj, \"weight\") else self.kv_b_proj.params_dtype ) if ( use_fp8_prefill or _kv_b_proj_w_dtype != current_platform.fp8_dtype() ) and _kv_b_proj_w_dtype != torch.uint8: kv_c_normed = kv_c_normed.to(self.kv_b_proj.weight.dtype) ``` The intent is \"cast the gathered KV back to the layer's expected input dtype, except for quantized layouts\". But on sm80, `MarlinFP8ScaledMMLinearKernel.process_weights_after_loading` repacks the fp8 weight via `pack_fp8_to_int32` + `gptq_marlin_repack` and `replace_parameter(layer, \"weight\", marlin_qweight)` — so `kv_b_proj.weight.dtype` is **`torch.int32`** (packed), not `float8_e4m3fn`. The condition (`!= fp8_dtype()`, `!= uint8`) doesn't exclude it, so the bf16 activations are cast to `torch.int32` and `marlin_gemm` rejects them. On Hopper the weight stays fp8 so the cast is skipped and nothing breaks, which is presumably why this wasn't caught. ### Reproduction 1. Serve any fp8-quantized MLA model on an sm80 GPU (Marlin path), with prefix caching enabled (default). 2. Send one short request. It succeeds. 3. Send a second request sharing a token prefix with the first (or a single prompt longer than `max_num_batched_tokens`). 4. Worker crashes with the traceback below; engine core dies. ### Traceback (trimmed) ``` File \".../vllm/model_executor/layers/attention/mla_attention.py\", line 2300, in forward_mha context_output, context_lse = self._compute_prefill_context( File \".../vllm/model_executor/layers/attention/mla_attention.py\", line 2097, in _compute_prefill_context kv_nope = self.kv_b_proj(kv_c_normed)[0].view( File \".../vllm/model_executor/layers/linear.py\", line 582, in forward output_parallel = self.quant_method.apply(self, input_, bias) File \".../vllm/model_executor/layers/quantization/fp8.py\", line 474, in apply return self.fp8_linear.apply_weights(layer, x, bias) File \".../vllm/model_executor/kernels/linear/scaled_mm/marlin.py\", line 114, in apply_weights return apply_fp8_marlin_linear( File \".../vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py\", line 69, in apply_fp8_marlin_linear output = ops.marlin_gemm( RuntimeError: unsupported `a` scalar_type ``` ### Suggested fix Only perform the cast when the target dtype is a floating-point type — for packed integer weight layouts (Marlin int32, NVFP4 uint8, …) the quant method expects model-dtype input and quantizes internally, exactly like the normal (non-context) prefill path already does: ```python if ( (use_fp8_prefill or _kv_b_proj_w_dtype != current_platform.fp8_dtype()) and _kv_b_proj_w_dtype != torch.uint8 and _kv_b_proj_w_dtype.is_floating_point ): kv_c_normed = kv_c_normed.to(self.kv_b_proj.weight.dtype) ``` (The existing `!= torch.uint8` special case then becomes redundant.) We've been running this fix in production on 12×A100 and it resolves the crash: prefix-cache hits and 6.5K-token chunked prefills work, outputs are correct. Happy to send a PR if the approach looks right. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and checked the documentation and examples.", "url": "https://github.com/vllm-project/vllm/issues/47522", "createdAt": "2026-07-03T06:10:07Z", "updatedAt": "2026-08-13T14:19:27Z", "timestamp": "2026-08-13T14:19:27Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [], "author": "biondogs", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:ce336b260e8bd7269b3e", "signalId": "github:vllm-project/vllm:pull_request:52030", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52030", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix packed GDN decode launch for large batch-head grids", "text": "## Purpose Avoid a CUDA launch failure in packed GDN decode when `batch_size * num_value_heads` exceeds the maximum CUDA grid Y/Z dimension of 65,535. The existing launch is preserved for normal sizes. Only overflowing cases use a split `(value_tiles, value_heads, batch)` grid. ## Test Plan ## Test Result - Verified the failing Qwen shape (`B=1024`, `HV=64`, `K=V=128`) launches successfully. - Running `vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned94 -tp=2` doesn't crash anymore ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52030", "createdAt": "2026-08-12T17:44:32Z", "updatedAt": "2026-08-13T14:19:09Z", "timestamp": "2026-08-13T14:19:09Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready" ], "author": "mgoin", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:6c70008f3e6cc578b427", "signalId": "github:vllm-project/vllm:pull_request:52172", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52172", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Disable sequence parallelism for Dots3 NOTE", "text": "## Why The DeepSeek V3.2 sequence-parallel refactor changed the inherited forward paths to use `use_sequence_parallel`. Dots3 NOTE uses custom model and decoder initializers and has not adopted the new sequence-parallel execution path. This causes serving to fail during KV cache profiling with: `AttributeError: 'Dots3NoteModel' object has no attribute 'use_sequence_parallel'` ## What changed Explicitly disable sequence parallelism for Dots3 NOTE and keep its existing non-SP execution path. ## Validation - BF16 DP8 + EP + MTP3: startup and inference passed. - BF16 TP8 + EP + MTP3: startup and inference passed. - Text and image requests completed successfully. - No duplicate issue or PR was found.", "url": "https://github.com/vllm-project/vllm/pull/52172", "createdAt": "2026-08-13T13:57:37Z", "updatedAt": "2026-08-13T14:13:01Z", "timestamp": "2026-08-13T14:13:01Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "verified" ], "author": "KurodaKanbei", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:7dec1611155afeaed7e7", "signalId": "github:vllm-project/vllm:pull_request:52174", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52174", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add forward_xpu to XDRotaryEmbedding for HunyuanOCR on XPU", "text": "## Purpose `XDRotaryEmbedding` (rope_type `xdrope`, used by `tencent/HunyuanOCR`, arch `HunYuanVLForConditionalGeneration`) overrides `forward_native` and `forward_cuda` but not `forward_xpu`. `CustomOp.dispatch_forward()` binds the method once at init based on `compilation_config.custom_ops`: - Default (Inductor on): `custom_ops=[]` -> `'none'` -> custom op disabled -> `forward_native`. Handles the 2-D `[4, num_tokens]` xdrope positions correctly -> PASS. - `--enforce-eager`: Inductor off -> `custom_ops=['all']` -> custom op enabled -> `forward_xpu`. `XDRotaryEmbedding` has none, so dispatch falls back to base `RotaryEmbedding.forward_xpu`, which calls the generic C++ kernel `torch.ops._C.rotary_embedding`. That kernel expects 1-D positions matching q/k `num_tokens`; given the 2-D `[4, num_tokens]` (P/W/H/T sections) positions it raises: ``` RuntimeError: query, key and positions must have the same batch_size and seq_len ``` crashing `EngineCore` during `profile_run`, so the model cannot serve at all under `--enforce-eager` on XPU. Fix: add `forward_xpu` to `XDRotaryEmbedding` delegating to `forward_native` (no fused XPU xdrope kernel exists), mirroring how mrope handles XPU. ## Test Plan ```bash vllm serve tencent/HunyuanOCR --dtype bfloat16 --tensor-parallel-size 1 \\ --max-model-len 8192 --gpu-memory-utilization 0.85 \\ -cc '{\"inductor_compile_config\":{\"benchmark_combo_kernel\":false}}' \\ --port 8005 --trust-remote-code --enforce-eager \\ --limit-mm-per-prompt '{\"image\": 1}' ``` ## Test Result Before fix - `EngineCore` crashes during `profile_run`: ``` File \".../vllm/model_executor/models/hunyuan_v1.py\", line 237, in forward q, k = self.rotary_emb(positions, q, k) File \".../vllm/model_executor/custom_op.py\", line 136, in forward return self._forward_method(*args, **kwargs) File \".../vllm/model_executor/layers/rotary_embedding/base.py\", line 288, in forward_xpu ops.rotary_embedding(...) File \".../vllm/_custom_ops.py\", line 212, in rotary_embedding torch.ops._C.rotary_embedding(...) RuntimeError: query, key and positions must have the same batch_size and seq_len ``` After fix: - `Application startup complete` - `/health` -> 200; `/v1/models` serves `tencent/HunyuanOCR` Default (torch.compile) path already routed to `forward_native` and is unaffected.", "url": "https://github.com/vllm-project/vllm/pull/52174", "createdAt": "2026-08-13T14:12:13Z", "updatedAt": "2026-08-13T14:12:57Z", "timestamp": "2026-08-13T14:12:57Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "intel-gpu" ], "author": "jbyczkow", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:d49611cf6d70121126be", "signalId": "github:vllm-project/vllm:pull_request:52173", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52173", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Apply logit softcapping in Transformers modelling backend", "text": "This is mainly used by Gemma models and only affects workloads where the exact logit values are important. Generation is unaffected because it does not reorder anything.", "url": "https://github.com/vllm-project/vllm/pull/52173", "createdAt": "2026-08-13T14:01:25Z", "updatedAt": "2026-08-13T14:12:52Z", "timestamp": "2026-08-13T14:12:52Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "hmellor", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:2fbefc75c952221a8780", "signalId": "github:vllm-project/vllm:pull_request:50288", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50288", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[SM120] Add NVFP4 KV cache support for consumer Blackwell (RTX 5090)", "text": "## Summary - Fix V scale swizzle mismatch: the NVFP4 KV store kernel unconditionally swizzled V block scales for the SM100 trtllm-gen MHA kernel, but FlashInfer FA2 (used on SM120) reads scales linearly. Add runtime SM detection so SM≥120 writes V scales without swizzle, resolving garbled output on RTX 5090. - Enable FlashInfer NVFP4 KV cache on SM120: `supports_kv_cache_dtype` accepts SM120, HND layout, BF16 query/output (SM120 FA2 native path does not use FP8 quantization or FP8 output buffers), and \"auto\" backend selection for prefill/decode wrappers. - Route Gemma4 heterogeneous-head models (head_dim_qk=512, head_dim_vo=256) to FLASHINFER on CC 12.x with NVFP4 KV via `VLLM_NVFP4_KV_VOSPLIT` env var (default on). - Add `FlashInferBackend.supports_mm_prefix()` gated on `VLLM_FLASHINFER_MM_PREFIX` (default on) so multimodal Gemma3/4 models are not rejected by backend validation. - Bump flashinfer-python/cubin pin to 0.6.16. ## Context SM120 (RTX 5090) does not have trtllm-gen cubins. The NVFP4 KV cache path previously only worked on SM100 (B200) via trtllm-gen. This PR enables the FlashInfer FA2 tensor-core native path for SM120, which dequantizes FP4 KV to BF16 internally during attention computation. Tested on RTX 5090 with: - Gemma-4-26B-A4B (heterogeneous heads: sliding head_dim=256, full head_dim=512) - Qwen3.6-27B, Qwen3.6-35B-A3B (uniform heads) All produce correct output with `--kv-cache-dtype nvfp4`. ## Test plan - [ ] `vllm serve --kv-cache-dtype nvfp4` on RTX 5090 produces coherent output - [ ] `vllm serve --kv-cache-dtype nvfp4` on RTX 5090 produces coherent output - [ ] SM100 trtllm-gen path unchanged (V scales still swizzled for CC < 120) - [ ] FP8 KV cache unaffected on both SM100 and SM120", "url": "https://github.com/vllm-project/vllm/pull/50288", "createdAt": "2026-07-29T15:46:04Z", "updatedAt": "2026-08-13T14:09:32Z", "timestamp": "2026-08-13T14:09:32Z", "metrics": { "reactions": 3, "comments": 9 }, "labels": [ "needs-rebase", "ci/build", "v1", "nvidia", "quantization" ], "author": "ch2lab", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:4e7e4bcee6c65ec47755", "signalId": "github:vllm-project/vllm:pull_request:43091", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:43091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Support spec decode with draft model", "text": "## Purpose Support spec decode with draft models in model runner v2 ## Test Plan ``` python examples/features/speculative_decoding/spec_decode_offline.py \\ --method draft_model \\ --model-dir /shared/models/modelscope/models/Qwen/Qwen3-32B \\ --draft-model /shared/models/modelscope/models/Qwen/Qwen3-1.7B \\ --num-spec-tokens 3 \\ --dataset-name hf \\ --dataset-path philschmid/mt-bench \\ --num-prompts 400 \\ --temp 0 \\ --output-len 256 \\ --max-model-len 10000 ``` ## Test Result **Model Runner V1:** ``` -------------------------------------------------- total_num_output_tokens: 102355 num_drafts: 36593 num_draft_tokens: 109779 num_accepted_tokens: 65181 mean acceptance length: 2.78 -------------------------------------------------- acceptance at token 0: 0.75 acceptance at token 1: 0.57 acceptance at token 2: 0.45 ``` **Model Runner V2:** ``` -------------------------------------------------- total_num_output_tokens: 102312 num_drafts: 36394 num_draft_tokens: 109182 num_accepted_tokens: 65379 mean acceptance length: 2.80 -------------------------------------------------- acceptance at token 0: 0.76 acceptance at token 1: 0.58 acceptance at token 2: 0.46 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/43091", "createdAt": "2026-05-19T11:35:21Z", "updatedAt": "2026-08-13T14:08:50Z", "timestamp": "2026-08-13T14:08:50Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "ready", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "wxsIcey", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2a74bb8c42c1314fa501", "signalId": "github:vllm-project/vllm:pull_request:42662", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:42662", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[LoRA][Gemma4] Support vision tower LoRA", "text": "This PR adds the remaining LoRA plumbing needed for Gemma4 multimodal LoRA support. After #43798, Gemma4-MM vision linear layers are already converted through the Transformers backend path, so this PR no longer reimplements the Gemma4 vision tower. Instead, it focuses on the runtime LoRA mapping and token-counting pieces needed by Gemma4 image/video/audio inputs. Main changes: - Add a multimodal LoRA token-count interface so models can report separate tower and connector token counts. - Update Gemma4-MM to report modality-specific LoRA token counts for image, video, and audio inputs. - Size multimodal LoRA wrappers using the largest tower/connector token budget across modalities. FIX https://github.com/vllm-project/vllm/issues/40693 ## Test Plan Additional end-to-end tests for real Gemma4 vision LoRA adapters should also be added in a follow-up. ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42662", "createdAt": "2026-05-14T16:37:41Z", "updatedAt": "2026-08-13T14:07:44Z", "timestamp": "2026-08-13T14:07:44Z", "metrics": { "reactions": 4, "comments": 17 }, "labels": [ "documentation", "ready", "ci/build", "v1", "mrv2" ], "author": "linitra24", "state": "closed", "assignees": [ "jeejeelee" ], "change": "updated" } }, { "id": "event:adab91dc2d8d2e704ab4", "signalId": "github:vllm-project/vllm:pull_request:52171", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Declare SupportsEagle3 on KimiLinearForCausalLM", "text": "### Purpose `KimiK3ForConditionalGeneration` (multimodal) declares `SupportsEagle3`; the text-only `KimiLinearForCausalLM` does not — even though both serve the same inner `KimiLinearModel`, which already inherits `EagleModelMixin` and implements the aux-hidden-state tap machinery. Serving a text-only Kimi-K3 checkpoint with EAGLE3-family speculative decoding (e.g. `dspark`) therefore dies at startup: ``` RuntimeError: Model does not support EAGLE3 interface ``` Adding the interface to the class bases is the whole fix: the protocol's default `set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` delegate to `self.model`, which satisfies their `EagleModelMixin` requirement. ### Test plan - `tests/models/kimi_k3/test_eagle3.py` gains `test_kimi_linear_advertises_eagle3_support`, mirroring the existing multimodal assertion; it fails on `main` and passes with this change. - Runtime-validated: a text-only Kimi-K3 checkpoint served with a `dspark` draft on 8× RTX 3090 using an equivalent patch (that is how we hit the error). --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52171", "createdAt": "2026-08-13T13:55:11Z", "updatedAt": "2026-08-13T14:07:01Z", "timestamp": "2026-08-13T14:07:01Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "verified", "kimi", "k3" ], "author": "nickus", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:41080ad309e25deafa0e", "signalId": "github:vllm-project/vllm:pull_request:33763", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:33763", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add vllm_enable_compile_cache config flag with backward compatibility", "text": "- Add vllm_enable_compile_cache boolean flag to CompilationConfig (defaults to True) - Keep VLLM_DISABLE_COMPILE_CACHE environment variable for backward compatibility - Environment variable overrides config flag when set via model_validator - Update is_compile_cache_enabled() to accept config flag parameter - Update all test files to use config flag instead of environment variable - Update documentation to mention both config flag (preferred) and env var (backward compatibility) This change provides a cleaner API via config while maintaining backward compatibility with existing code using the environment variable. For https://github.com/vllm-project/vllm/issues/29917 ---
- [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. - [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).
", "url": "https://github.com/vllm-project/vllm/pull/33763", "createdAt": "2026-02-04T04:50:40Z", "updatedAt": "2026-08-13T14:03:27Z", "timestamp": "2026-08-13T14:03:27Z", "metrics": { "reactions": 2, "comments": 19 }, "labels": [ "documentation", "frontend", "ready", "llama", "qwen" ], "author": "elizabetht", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:07027eeafd28deec9fde", "signalId": "github:vllm-project/vllm:pull_request:48263", "event": "discovered", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48263", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KV Connector][4/N][NIXL] Recover all dedup'd HMA pool members (FA + multi-group SSM) under PP", "text": "## Purpose Fixes silent KV-cache corruption for **hybrid (Mamba/SSM) models under pipeline parallelism** in NIXL PD-disaggregated serving. Under PP>1 + the hybrid-memory allocator (HMA), the general-case allocator pools **one layer from every `kv_cache_group`** into a single shared tensor. For a multi-Mamba-group hybrid (e.g. NVIDIA Nemotron-3-Ultra-550B: 4 Mamba groups + 1 FullAttention), each pooled tensor holds *N* Mamba layers + 1 FA layer, all full-aliased at offset 0. `register_kv_caches` dedups by base address and keeps only the **group-0 Mamba representative**; the other (N−1) dedup'd Mamba members and the dedup'd FA member fall out of `registered_layer_names`. As a result `mamba_region_group_ids` collapses to `[0]*N`: only the group-0 Mamba layers transfer, and the remaining `(num_mamba_groups−1)/num_mamba_groups` of the Mamba layers run on **stale/zero SSM+conv state** on the decode side. The output is fluent but wrong-content — a silent correctness bug, not a crash. The stacked base #43368 introduced `region_members` (the per-region member list) and per-layer HMA routing, but its dedup branch only records members and `continue`s — it re-advertises **no** dedup'd member (verified against `zixi-qi:pr2/pp-disagg-nixl-hma`), so every non-representative member of a pooled tensor is dropped from the transfer set. This PR makes the producer **re-advertise every dedup'd member** (each FA member with the FA page stride; each extra-group Mamba member with the Mamba page stride) as its own NIXL region, so `registered_layer_names` becomes complete and `mamba_region_group_ids` de-collapses to `[0,1,2,…]`. The consumer applies an SSM-only region filter — gated on the new capability flags — before the Mamba descriptor builder so the SSM desc-id math stays aligned once FA regions are present. Fixes #46407. Root cause + live evidence tracked on #43765 / #43508. > **Stacked on #43368** (per-layer-name HMA routing), which is stacked on #43366 (NIXL PP support). This is [4/N] in that series. It is **Draft** until #43368 merges; the diff below is scoped to the incremental delta on top of #43368's branch and will rebase onto `main` unchanged once #43368 lands. Not related to #45880 (push-mode PP prefill, orthogonal; merged 2026-07-08) — this is the pull-mode path. ## Changes Net delta on top of #43368 — **2 files, ~2 new methods + 2 fields + one producer branch + one consumer gate** (self-contained; no change to the PP-routing, TP-mapping, or handshake wire format that #43368 owns): ``` vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py | +2 fields, version bump vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py | +2 methods, +1 producer branch, +1 consumer gate ``` **`metadata.py`** - Add `fa_members_registered: bool = False` and `ssm_members_registered: bool = False` to `NixlAgentMetadata`. Both are read defensively (`getattr(meta, ..., False)`) so a v6 producer that omits them stays wire-compatible, and a newer consumer paired with an older producer degrades to the legacy path safely. - Bump `NIXL_CONNECTOR_VERSION` and document the two new versions in the existing version-history block. - (`region_members` and `registered_layer_names` already exist from #43368 — unchanged here.) **`worker.py`** - `register_kv_caches`: in the existing dedup branch (`existing_region_idx is not None`), when a dedup'd member is a FullAttention-on-Mamba alias **or** a distinct-Mamba-group-on-Mamba alias, **re-advertise it as its own region** at the same base addr (full-alias, offset 0) with the member's own page stride (FA physical page for FA; `physical_page_size // physical_blocks_per_logical_kv_block` for SSM). Set `fa_members_registered` / `ssm_members_registered` accordingly. Two by-construction `assert`s enforce the HMA full-alias invariant (equal base, equal `page_size_bytes`) so any future layout change fails loud at registration instead of silently mis-transferring. - New `_region_fa_recovery_for_members(...)`: per producer region, recover a dedup'd FA member hidden behind a non-FA representative, resolving its kv-group by layer name (PP-stable). - New `_ssm_layer_names(...)`: filter advertised layer names to SSM-group layers only. - `_mamba_region_group_ids_for_layer_names(...)`: route through `_ssm_layer_names` so the Mamba desc bookkeeping covers only SSM layers (attention layers no longer inflate the count / shift SSM offsets). - Consumer (`add_remote_agent`, local + remote descriptor layouts): apply the SSM-only region filter to the meta the Mamba builder sees, gated on `(fa_members_registered or ssm_members_registered)`. **Inert on unaffected paths (byte-unchanged):** PP=1 (each layer is already its own representative), all-attention, single-Mamba-group, and MLA producers never set either flag, so the re-advertise branch never fires and the consumer filter is a no-op — matching the #43366/#43368 gating convention. Verified by the +486 MB PP1 control being unchanged. ## Test Plan **Unit** (deterministic, no GPU — the CI gate): ``` pytest -q tests/v1/kv_connector/unit/test_nixl_connector_pp_hma_multimember.py ``` New test constructs a producer whose HMA pool shares one tensor across *N*>1 Mamba groups + 1 FA group (multi-group, mirroring Nemotron-3-Ultra), runs `register_kv_caches`, and asserts: - every dedup'd member (all *N* Mamba reps + the FA member) appears in `registered_layer_names`; - `fa_members_registered is True` and `ssm_members_registered is True`; - `_mamba_region_group_ids_for_layer_names(...)` de-collapses from the pre-fix `[0]*4N` to `[0,1,…,N-1]` (×4 for conv0/1/2+ssm); - a single-Mamba-group / PP1 / all-attention config leaves both flags `False` and the region set byte-identical to pre-fix (guards the inert paths). **E2E accuracy** (registered into CI): add a multi-Mamba-group hybrid PP config to `tests/v1/kv_connector/nixl_integration/config_sweep_accuracy_test.sh` (`HYBRID_SSM` group) and wire the label in `.buildkite/test_areas/disaggregated.yaml`, running GSM8K 5-shot parity between an aggregated baseline and the PP-disaggregated deployment (0 transfer errors, accuracy equal within noise). Reproducible serve commands (producer PP=2 / consumer PP=2, `kv_both`, LIBFABRIC backend over EFA) and the aggregated control are in the PR's `tests/.../nixl_integration/` harness. ## Test Result **Unit:** _to be pasted from the PR branch after `pytest` on the rebased tree (expected: N passed)._ **E2E — live on 2× p5en (8× H200) over EFA, real Nemotron-3-Ultra-550B (hybrid Mamba2 + attention), TP8×PP2 both roles, NIXL/LIBFABRIC, 2026-06-26 → 2026-07-07:** | Metric | Pre-fix (PP2) | This PR (PP2) | Aggregated PP1 control | |---|---|---|---| | `mamba_region_group_ids` | `[0]*24` (collapsed) | `[0×24, 1×24, 2×24, 3×24]` | n/a | | `region_group_ids` | all-SSM | `[0..4 ×12]` (5 groups) | n/a | | `num_regions` | 24 | 60 | n/a | | KV over EFA (decode) | degenerate | **~487 MB/req**, 4221–4917 MB/s, 804 descriptors, 48 transfers, 0 failures | +486 MB/req | | Decode output | fluent but **wrong-content** (stale SSM) | **byte-parity vs control** | ground truth | - Correctness parity vs the aggregated PP1 control (same image, temp 0, seed 42, identical prompts): **6/6 answers correct**; **5/6 strict byte-identical** (the one diff is a reasoning-preamble prose difference with the answer \"Paris\" identical — cross-parallelism fp reduction-order nondeterminism, TP8×PP2 vs TP8×PP1). Both verbatim needle recalls (`MAROON-8817-QUARTZ`, `73912`) are byte-perfect — a stale-SSM/wrong-KV bug corrupts random-string recall first, and it did not. - Spot checks post-fix: \"capital of France\" → Paris; \"17 plus 25\" → 42; monotonic sequence continuation correct.
Essential Elements of an Effective PR Description Checklist - [x] Purpose — fixes the multi-member HMA region-group collapse left by #43368; links #46407 / #43368 / #43366 - [x] Test plan — new deterministic unit test + GSM8K PD accuracy config wired into `disaggregated.yaml` - [x] Test result — de-collapse table + ~487 MB/req KV-over-EFA + GSM8K/byte parity (unit counts pasted on the PR branch pre-merge) - [ ] (Optional) Documentation update — n/a (internal connector path; no user-facing model/API/example change)
## Related PRs / Issues - Fixes #46407 (bug report + root cause, ours). - Stacked on #43368 (per-layer-name HMA routing) → #43366 (NIXL PP support). Credits @zixi-qi for the PP + `region_members` foundation this extends. - Context: #43765 (hybrid SSM/GDN PD feature), #43508 (Mamba `_mamba_block_aligned_split` assertion). - Not #45880 (push-mode PP prefill; orthogonal, merged) — this is the pull-mode path. Signed-off-by: Anton Alexander <dmvevents@gmail.com> This PR was AI-assisted; the submitter has reviewed every line and reproduced the results above on live hardware.", "url": "https://github.com/vllm-project/vllm/pull/48263", "createdAt": "2026-07-10T16:52:01Z", "updatedAt": "2026-08-13T14:00:54Z", "timestamp": "2026-08-13T14:00:54Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "needs-rebase", "v1", "kv-connector" ], "author": "dmvevents", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:9b8298a7a81a2363554c", "signalId": "github:vllm-project/vllm:pull_request:52005", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52005", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix .../mrope.py::apply_interleaved_rope() when torch.compile is used in torch==2.13", "text": "## Purpose This PR fixes incorrect outputs of `torch.compile` `.../mrope.py::apply_interleaved_rope()` when it's used with `torch==2.13` (which newest vLLM uses), In `torch==2.11` it worked correctly. We fix it by computing the same output in a way that `torch.compile` doesn't break . This method does only indexing and assigning so compiling it should not introduce errors. Added test case, comparing eager to `torch.compile` without updated function test case returns. ``` E Mismatched elements: 98229 / 524288 (18.7%) E Greatest absolute difference: 6.25 at index (3749, 9) E Greatest relative difference: 32512.0 at index (1541, 5) ``` [Issue](https://github.com/pytorch/pytorch/issues/193183) in pytorch github ## Test Plan ``` python -m pytest tests/kernels/core/test_mrope.py::test_apply_interleaved_rope tests/kernels/core/test_mrope.py::test_apply_interleaved_rope_torch_compile ``` Additionally it can be tested by running request to `nvidia/Cosmos3-Edge` (need to apply fixes from [PR51989](https://github.com/vllm-project/vllm/pull/51989)), server started with `VLLM_DISABLE_COMPILE_CACHE=1 vllm serve nvidia/Cosmos3-Edge --reasoning-parser nemotron_v3`: ``` curl -sS http://localhost:8000/v1/chat/completions \\ -H 'Content-Type: application/json' \\ --data-binary @- <<'JSON' | { \"model\": \"nvidia/Cosmos3-Edge\", \"messages\": [ { \"role\": \"user\", \"content\": [ { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\" } }, { \"type\": \"image_url\", \"image_url\": { \"url\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg\" } }, { \"type\": \"text\", \"text\": \"Identify the images independently. In the final answer, write exactly two short lines. Start the first line with 'Image 1:' and the second line with 'Image 2:'. Do not return JSON.\" } ] } ], \"max_tokens\": 1024, \"temperature\": 0, \"stream\": false, \"chat_template_kwargs\": { \"enable_thinking\": true } } JSON jq -r '.choices[0].message.content' ``` With the fix it outputs: ``` Image 1: A bee on a pink flower in a garden setting. Image 2: A rabbit in a blue coat standing on a dirt path with flowers and a house in the background. ```` Without: ``` Image 1: A rabbit in a blue coat standing on a path with flowers. Image 2: A cartoon rabbit dressed as a boy on a dirt path with a house and hills. ``` ## Test Result ``` 2 passed, 14 warnings in 0.77s ``` ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52005", "createdAt": "2026-08-12T15:03:07Z", "updatedAt": "2026-08-13T14:00:49Z", "timestamp": "2026-08-13T14:00:49Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "torch.compile", "verified" ], "author": "bastefaniak", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9971435f8e5b62afd944", "signalId": "github:vllm-project/vllm:pull_request:52168", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Restore multimodal support on the plain \"vllm\" throughput backend", "text": "## Purpose `vllm bench throughput --backend vllm` crashes on any multimodal dataset (e.g. ASR/Whisper) with: ValueError: Multi-modal content is not supported on backend 'vllm'; use one of []. Regression from #50981. `get_requests()` computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handle: ```python mm_backends = (\"vllm-chat\",) if args.backend == \"vllm-chat\" else () ``` This is circular — it only lets the already-selected backend through the gate, and only if it's literally \"vllm-chat\". So --backend vllm always computes mm_backends=() and self-rejects, even though _run_vllm_requests (the function that executes --backend vllm requests) already forwards multi_modal_data straight into LLM.generate — it was always multimodal-capable, same as vllm-chat. Only hf (asserts prompts must be str) and mii (text-only) genuinely can't carry multimodal content, so the allowlist should be static rather than derived from the chosen backend: ```python mm_backends = (\"vllm\", \"vllm-chat\") ``` ## Test Plan - Added test_get_requests_allows_multimodal_on_plain_vllm_backend to tests/benchmarks/test_throughput_cli.py, mirroring the existing test_get_requests_resolves_mmvu pattern: stubs ASRDataset (network-free), parses --backend vllm --dataset-name hf --dataset-path openslr/librispeech_asr, asserts get_requests() resolves without raising. - Ran the exact vllm bench throughput command from the bug report end-to-end on real hardware (Intel XPU B70) before and after the fix. - ruff check / ruff format on both changed files. ## Test Result - New test: fails with the reported ValueError against pre-fix code, passes against post-fix code. - python -m pytest tests/benchmarks/test_throughput_cli.py -k \"allows_multimodal_on_plain_vllm_backend or resolves_mmvu\" → 2 passed. - End-to-end on B70: pre-fix reproduces the ValueError verbatim; post-fix completes cleanly (Throughput: 20.96 requests/s, 5386.74 total tokens/s). - ruff check: all checks passed. ruff format --diff: no changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52168", "createdAt": "2026-08-13T13:27:39Z", "updatedAt": "2026-08-13T13:59:02Z", "timestamp": "2026-08-13T13:59:02Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "bug", "performance" ], "author": "mganczarenko", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:77dfa29ac7929429dcf8", "signalId": "github:vllm-project/vllm:pull_request:51952", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51952", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "NIXL: Use int32 array for indices to avoid intermediate conversion", "text": "## Purpose As documented NIXL accepts KV indices as `int32` array, while vLLM creates and passes `int64` array. This results in extra allocation and conversion on NIXL side, which is redundant. Removing this conversion + adding zero-copy on NIXL side improves performance https://github.com/ai-dynamo/nixl/pull/1718 ## Test Result \"pareto_compare_nixl_zci_vs_concurrency_100538\"", "url": "https://github.com/vllm-project/vllm/pull/51952", "createdAt": "2026-08-12T08:15:50Z", "updatedAt": "2026-08-13T13:58:20Z", "timestamp": "2026-08-13T13:58:20Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "kv-connector" ], "author": "iyastreb", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:03212cc0292c60b003bc", "signalId": "github:vllm-project/vllm:pull_request:49357", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49357", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] bound FlashMLA sparse decode intermediate tensors size", "text": "## Purpose Fix #44545 - not in an ideal way though. When FlashMLA sparse kernel is selected with FP8 KV cache dtype, FlashMLA sparse decode kernel(https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_decode.h#L184) is called in mixed batch mode. This kernel internally allocates intermediate tensors (`o_accum`) on call, which quickly dominates the usual unallocated workspace size (~2GB per the comment in `flashmla_sparse.py`) for long prefill: For 32k input on H200, the bytesize of `o_accum` is ``` # b = 1 # num_sm_parts = 1 # s_q = 32768 # h_q = 64 # d_v = 512 (b + num_sm_parts) * s_q * h_q * d_v * 4 bytes = 8 GiB ``` For BF16 KV cache, a different kernel (FlashMLA sparse prefill kernel - https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_fwd.h#L101) is called, which does not allocate a massive VRAM on kernel call and thus does not suffer an OOM contrary to the FP8 code branch. (Note the additional `d_v` factor in the formular for `o_accum`, no other scratch tensors in FlashMLA have this factor thus negligible in terms of VRAM occupancy) Hence we chunk the mixed-batch input into manageable bytes (`VLLM_FLASHMLA_SPARSE_MAX_SCRATCH_MB`) and loop over the chunks to aggregate the output attention tensor, which is in the same spirit as the indexer budget management logic in #36178. Due to the mechanism (looping the kernel over sub-chunks instead of single big shot), it may incur a throughput penalty as a tradeoff, but I have no better idea yet. As the context window for MLA model families keep growing, FP8 KV cache is kind of inevitable, so I think this PR can work as a bandaid. ## Test Plan No OOM for the reproducer in #44545 ## Test Result pass ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49357", "createdAt": "2026-07-21T18:06:42Z", "updatedAt": "2026-08-13T13:57:41Z", "timestamp": "2026-08-13T13:57:41Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "needs-rebase", "v1" ], "author": "cjackal", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a8dd876a6068932c6736", "signalId": "github:vllm-project/vllm:issue:51964", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:51964", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]:amd mi308x gpu, vllm 0.27.0~0.27.1, rocm 7.2.3, Kimi-K2.7-Coder start fails:AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942", "text": "### Your current environment
The output of python collect_env.py ```text (vllm) [fanlf@2025-h3c-83 tmp]$ python ./collect_env.py Collecting environment information... ============================== System Info ============================== OS : Rocky Linux 10.2 (Red Quartz) (x86_64) GCC version : (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4) Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.39 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+gitd0c8b1f Is debug build : False CUDA used to build PyTorch : N/A ROCM used to build PyTorch : 7.2.53211 XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Jul 10 2026, 00:00:00) [GCC 14.3.1 20251022 (Red Hat 14.3.1-4)] (64-bit runtime) Python platform : Linux-6.12.0-124.29.1.el10_1.x86_64-x86_64-with-glibc2.39 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : Could not collect CUDA_MODULE_LOADING set to : GPU models and configuration : AMD Instinct MI308X (gfx942:sramecc+:xnack-) Nvidia driver version : Could not collect cuDNN version : Could not collect HIP runtime version : 7.2.53211 MIOpen runtime version : 3.5.1 Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU(s) scaling MHz: 66% CPU max MHz: 3709.3569 CPU min MHz: 1500.0000 BogoMIPS: 4793.03 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap Virtualization: AMD-V L1d cache: 6 MiB (192 instances) L1i cache: 6 MiB (192 instances) L2 cache: 192 MiB (192 instances) L3 cache: 768 MiB (24 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-95,192-287 NUMA node1 CPU(s): 96-191,288-383 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace ============================== Versions of relevant libraries ============================== [pip3] conch-triton-kernels==1.2.1 [pip3] numpy==2.3.5 [pip3] onnx==1.22.0 [pip3] onnx-ir==0.2.1 [pip3] onnxscript==0.7.1 [pip3] onnxslim==0.1.95 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0+gitd0c8b1f [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.9.0+eaa9e4e [pip3] torchvision==0.24.1+d801a34 [pip3] transformers==5.15.0 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : 7.2.53211-c2d9476115 vLLM Version : 0.27.0 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 15 15 15 15 15 15 15 GPU1 15 0 15 15 15 15 15 15 GPU2 15 15 0 15 15 15 15 15 GPU3 15 15 15 0 15 15 15 15 GPU4 15 15 15 15 0 15 15 15 GPU5 15 15 15 15 15 0 15 15 GPU6 15 15 15 15 15 15 0 15 GPU7 15 15 15 15 15 15 15 0 ================================= Hops between two GPUs ================================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 1 1 1 1 1 1 1 GPU1 1 0 1 1 1 1 1 1 GPU2 1 1 0 1 1 1 1 1 GPU3 1 1 1 0 1 1 1 1 GPU4 1 1 1 1 0 1 1 1 GPU5 1 1 1 1 1 0 1 1 GPU6 1 1 1 1 1 1 0 1 GPU7 1 1 1 1 1 1 1 0 =============================== Link Type between two GPUs =============================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 XGMI XGMI XGMI XGMI XGMI XGMI XGMI GPU1 XGMI 0 XGMI XGMI XGMI XGMI XGMI XGMI GPU2 XGMI XGMI 0 XGMI XGMI XGMI XGMI XGMI GPU3 XGMI XGMI XGMI 0 XGMI XGMI XGMI XGMI GPU4 XGMI XGMI XGMI XGMI 0 XGMI XGMI XGMI GPU5 XGMI XGMI XGMI XGMI XGMI 0 XGMI XGMI GPU6 XGMI XGMI XGMI XGMI XGMI XGMI 0 XGMI GPU7 XGMI XGMI XGMI XGMI XGMI XGMI XGMI 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: 0 GPU[1] : (Topology) Numa Node: 0 GPU[1] : (Topology) Numa Affinity: 0 GPU[2] : (Topology) Numa Node: 0 GPU[2] : (Topology) Numa Affinity: 0 GPU[3] : (Topology) Numa Node: 0 GPU[3] : (Topology) Numa Affinity: 0 GPU[4] : (Topology) Numa Node: 1 GPU[4] : (Topology) Numa Affinity: 1 GPU[5] : (Topology) Numa Node: 1 GPU[5] : (Topology) Numa Affinity: 1 GPU[6] : (Topology) Numa Node: 1 GPU[6] : (Topology) Numa Affinity: 1 GPU[7] : (Topology) Numa Node: 1 GPU[7] : (Topology) Numa Affinity: 1 ================================== End of ROCm SMI Log =================================== ============================== Environment Variables ============================== VLLM_ROCM_USE_AITER=1 PYTORCH_ROCM_ARCH=gfx942 TORCH_BLAS_PREFER_HIPBLASLT=1 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_fanlf ```
### 🐛 Describe the bug 1. install： uv pip install vllm==\"0.27.0+rocm723\" --extra-index-url https://wheels.vllm.ai/rocm --no-cache 2. startup and error log： ``` log (vllm) [fanlf@2025-h3c-83 vllm]$ export AITER_ROCM_ARCH=\"gfx942\" export PYTORCH_ROCM_ARCH=\"gfx942\" export MORI_GPU_ARCHS=\"gfx942\" export VLLM_ROCM_USE_AITER=1 export TORCH_BLAS_PREFER_HIPBLASLT=1 export HIP_FORCE_DEV_KERNARG=1 (vllm) [fanlf@2025-h3c-83 vllm]$ vllm serve \\ /pool/models/moonshotai/Kimi-K2.7 \\ --served-model-name kimi \\ --trust-remote-code \\ --tensor-parallel-size 8 \\ --mm-encoder-tp-mode data \\ --max-model-len 262144 \\ --gpu-memory-utilization 0.6 \\ --port 8003 \\ --host 0.0.0.0 \\ --tool-call-parser kimi_k2 \\ --enable-auto-tool-choice \\ --reasoning-parser kimi_k2 \\ --kv-cache-dtype fp8 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █ █ █▄ ▄█ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.0 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] █▄█▀ █ █ █ █ model /pool/models/moonshotai/Kimi-K2.7 (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:345] (APIServer pid=204677) INFO 08-12 14:53:04 [api_utils.py:273] non-default args: {'model_tag': '/pool/models/moonshotai/Kimi-K2.7', 'enable_auto_tool_choice': True, 'tool_call_parser': 'kimi_k2', 'host': '0.0.0.0', 'port': 8003, 'model': '/pool/models/moonshotai/Kimi-K2.7', 'trust_remote_code': True, 'max_model_len': 262144, 'served_model_name': ['kimi'], 'reasoning_parser': 'kimi_k2', 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.6, 'kv_cache_dtype': 'fp8', 'mm_encoder_tp_mode': 'data'} ...... (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] WorkerProc hit an exception. (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] Traceback (most recent call last): (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py\", line 1010, in worker_busy_loop (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] output = func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_worker.py\", line 517, in determine_available_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory() (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6732, in profile_cudagraph_memory (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._warmup_and_capture( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6935, in _warmup_and_capture (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] self._dummy_run( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/utils/_contextlib.py\", line 124, in decorate_context (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py\", line 6156, in _dummy_run (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] outputs = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/cuda_graph.py\", line 254, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.runnable(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/kimi_k25.py\", line 454, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.language_model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1779, in _wrapped_call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._call_impl(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/nn/modules/module.py\", line 1790, in _call_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return forward_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1899, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] hidden_states = self.model( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/decorators.py\", line 520, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.aot_compiled_fn(self, *args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_dynamo/aot_compile.py\", line 224, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.fn(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/models/deepseek_v2.py\", line 1433, in forward (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] def forward( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/compilation/caching.py\", line 217, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self.optimized_call(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 502, in execution_fn (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"\", line 9, in __vllm_inlined_submods__2 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/torch/_ops.py\", line 1269, in __call__ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return self._op(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/kv_transfer_utils.py\", line 40, in wrapper (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] return func(*args, **kwargs) (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 1254, in unified_mla_attention_with_output (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] layer.forward_impl( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/model_executor/layers/attention/mla_attention.py\", line 919, in forward_impl (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] attn_out, lse = self.impl.forward_mqa(mqa_q, kv_cache, attn_metadata, self) # type: ignore[attr-defined] (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/vllm/v1/attention/backends/mla/rocm_aiter_mla.py\", line 1093, in forward_mqa (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] mla_gluon( (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] File \"/home/fanlf/vllm/lib64/python3.12/site-packages/aiter/ops/triton/gluon/mla_gluon.py\", line 865, in mla_gluon (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] arch_info.get_arch() == \"gfx950\" (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942 (Worker_TP7 pid=205643) ERROR 08-12 14:58:20 [multiproc_executor.py:1018] [vllm-rocm.md.txt](https://github.com/user-attachments/files/30973739/vllm-rocm.md.txt) ``` ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/51964", "createdAt": "2026-08-12T08:49:47Z", "updatedAt": "2026-08-13T13:57:20Z", "timestamp": "2026-08-13T13:57:20Z", "metrics": { "reactions": 0, "comments": 5 }, "labels": [ "bug", "rocm", "kimi" ], "author": "badrecover", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a5cbdc47c5c6fd469321", "signalId": "github:vllm-project/vllm:pull_request:51794", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51794", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4", "text": "## Purpose > Part of https://github.com/vllm-project/vllm/issues/41820. > Continue working on https://github.com/vllm-project/vllm/pull/43718. > Based on https://github.com/vllm-project/vllm/pull/50866. > Added https://github.com/vllm-project/vllm/pull/52093 and https://github.com/vllm-project/vllm/pull/52094. To be completed... ## Test Plan - Benchmark with SA InferenceX `8k1k` and `agentic` workload. - Acc test with `gsm8k` dataset. ## Test Result ### Benchmark (workload=8k1k dp8+ep) | Concurrency | Metric | Disable CSA multi-stream | Enable CSA multi-stream | Comparison | | :---------: | :------------------------------ | :----------------------- | :---------------------- | :--------- | | 1 | Output token throughput (tok/s) | 29.67 | 23.71 | -20.09% ↓ | | 1 | Mean TTFT (ms) | 1457.20 | 1451.69 | -0.38% ↓ | | 1 | Mean TPOT (ms) | 32.18 | 40.67 | +26.38% ↑ | | 8 | Output token throughput (tok/s) | 168.20 | 146.34 | -13.00% ↓ | | 8 | Mean TTFT (ms) | 1768.56 | 1708.51 | -3.40% ↓ | | 8 | Mean TPOT (ms) | 44.85 | 51.90 | +15.72% ↑ | | 64 | Output token throughput (tok/s) | 477.18 | 451.11 | -5.46% ↓ | | 64 | Mean TTFT (ms) | 2966.52 | 3052.56 | +2.90% ↑ | | 64 | Mean TPOT (ms) | 129.00 | 136.34 | +5.69% ↑ | ### Benchmark (workload=agentx) To be completed... ### Acc Test To be completed... ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51794", "createdAt": "2026-08-11T09:48:40Z", "updatedAt": "2026-08-13T13:52:16Z", "timestamp": "2026-08-13T13:52:16Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "rocm", "deepseek", "nvidia" ], "author": "shen-shanshan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:941775ef9a9cfdbc5f7b", "signalId": "github:vllm-project/vllm:pull_request:52033", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52033", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][ROCm] Dual-stream decode with hipgraphs", "text": "## Changes since reverted in https://github.com/vllm-project/vllm/pull/52024 Re-opened https://github.com/vllm-project/vllm/pull/48223 with fix for Qwen3.5-style models (CI failed test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] after merging this PR previously). **Changes made since revert:** https://github.com/vllm-project/vllm/pull/52033/changes/50e80bb721d43c0d032d0a71d44546e3cdedcc6c. Issue stemmed from fact that Qwen3.5 uses the same tensor for hidden state and shared expert inputs, and later on mutates one of them in the routed experts. That causes a race condition during multi-stream. **Validation:** Re-ru nthe test with this config: https://github.com/vllm-project/vllm/blob/fe889ac9255491e82e58012d7787e920c6c7fbc2/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml#L6-L9 ```bash vllm serve Qwen/Qwen3.5-35B-A3B \\ --max-model-len 4096 \\ --data-parallel-size 2 \\ --enable-expert-parallel \\ --trust-remote-code \\ --disable-uvicorn-access-log ``` ``` docker run --rm -it \\ --entrypoint bash \\ --network host \\ --ipc host \\ vllm/vllm-openai-rocm:latest \\ -c \"pip install --quiet 'lm_eval[api]' && \\ lm_eval \\ --model local-completions \\ --model_args model=Qwen/Qwen3.5-35B-A3B,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5 \\ --output_path /tmp/lm_eval_gsm8k\" ``` **Results:** Running GSM8k on Qwen3.5 on MI350 with DP2EP gives (test pass requires acc >= 0.84 - 0.03 = 0.81) |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8522|± |0.0098| | | |strict-match | 5|exact_match|↑ |0.8362|± |0.0102| _From original PR:_ ## Purpose Fixes #48111. Enables (1) dual-stream decode for CUDA-like platforms with proper overlap (2) make them hip/cudagraph compatible. Only enabled on ROCm when using DP, as we observed [performance regression under TP](https://github.com/vllm-project/vllm/pull/48223#issuecomment-4993855510). Mutually exclusive with `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS`. Disable with `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` as usual. **Gain:** About -3-4% TPOT on 1k/1k and 8k/1k when using DPA vs default. **Note:** When Fused Shared experts (FSE) can be used, it provides even better TPOT than dual-stream. However, FSE cannot be used everywhere (like with MoRI a2a kernels). ### Technical details - Simply allowing `MULTI_STREAM_OVERLAPPED` for CUDA-like platforms is not sufficient for performant dual-stream decode (on ROCm, at least), because the shared expert was launched on the aux stream only _after_ the routed experts had completed. That leads to a sequential launch with no overlap: \"image\" - Instead, we need to launch the shared expert in the aux stream _before_ the dispatch. #### After this PR we get the following traces - Pure decodes: properly overlapped \"image\" - Mixed batches: no aux stream \"image\" - Pure prefills: no aux stream \"image\" ## Test Plan Tested on 8xMI300. 1. Run DSv3 with DPA+TP: ```bash export VLLM_ROCM_USE_AITER=1 vllm bench deepseek-ai/DeepSeek-V3-0324 \\ --data-parallel-size 8 \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.8 \\ --max-model-len 16K ``` 2. Bench serve sweep, 1k/1k and 8k/1k ISL/OSL - 10x as many prompts as the concurrency level; 2x number of warmups. - Comparing to nightly `vllm/vllm-openai-rocm:nightly-a16dbd5b8572d4128be9f10b9dcff4999b594b25` with fused shared experts ```bash vllm bench serve \\ --backend vllm \\ --model deepseek-ai/DeepSeek-V3-0324 \\ --dataset-name random \\ --random-input-len 1024 \\ --random-output-len 1024 \\ --host localhost \\ --port 8000 \\ --ignore-eos \\ --ready-check-timeout-sec 6000 \\ --seed 5678 \\ --max-concurrency 128 \\ --num-prompts 1280 \\ --num-warmups 256 ``` 4. GSM8k accuracy non-regression ```bash lm_eval \\ --model local-completions \\ --model_args model=deepseek-ai/DeepSeek-V3-0324,base_url=http://localhost:8000/v1/completions,tokenized_requests=False,trust_remote_code=True,num_concurrent=256 \\ --tasks gsm8k \\ --num_fewshot 5\" ``` ## Test Result - Couple % improved TPOT ### 1k/1k (P50's only) | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 232.57 | 23.67 | **+4.24%** | | 1 | nightly | 0.04 | 273.16 | 24.72 | - | | 2 | this branch | 0.08 | 258.38 | 24.16 | **+4.58%** | | 2 | nightly | 0.08 | 298.14 | 25.32 | - | | 4 | this branch | 0.15 | 397.90 | 25.52 | **+7.03%** | | 4 | nightly | 0.14 | 366.23 | 27.45 | - | | 8 | this branch | 0.25 | 602.93 | 30.84 | **+3.41%** | | 8 | nightly | 0.24 | 664.92 | 31.93 | - | | 16 | this branch | 0.44 | 1268.65 | 35.02 | -6.77% | | 16 | nightly | 0.46 | 851.06 | 32.80 | - | | 32 | this branch | 0.71 | 1830.04 | 42.03 | **+2.75%** | | 32 | nightly | 0.70 | 1698.74 | 43.22 | - | | 64 | this branch | 1.34 | 3228 | 43.51 | **+4.71%** | | 64 | nightly | 1.29 | 2528 | 45.66 | - | | 128 | this branch | 2.15 | 4487.07 | 53.85 | **+1.86%** | | 128 | nightly | 2.10 | 4543.42 | 54.87 | - | | 256 | this branch | 3.46 | 6554.86 | 65.74 | **+3.12%** | | 256 | nightly | 3.37 | 6386.54 | 67.86 | - | Note: TTFT numbers seems highty variable. Ran 64 concurrency yet another time, and then I got 1902 ms TTFT @ 1.37 QPS, substantially better than the first time. This PR should not affect prefills anyhow so not focusing too much on TTFT here anyways. ### 8k/1k | Concurrency | Variant | QPS | TTFT (ms) | TPOT (ms) | % TPOT improved | |---|---|---|---|---|---| | 1 | this branch | 0.04 | 1868.16 | 25.22 | **+3.59%** | | 1 | nightly | 0.04 | 1860 | 26.16 | - | | 2 | this branch | 0.07 | 2034.06 | 25.42 | **+5.47%** | | 2 | nightly | 0.07 | 2014.35 | 26.89 | - | | 4 | this branch | 0.11 | 3800.62 | 31.35 | **+2.82%** | | 4 | nightly | 0.11 | 3687 | 32.26 | - | | 8 | this branch | 0.19 | 4105.90 | 37.01 | **+5.54%** | | 8 | nightly | 0.18 | 4122.95 | 39.18 | - | | 16 | this branch | 0.31 | 6355 | 45.37 | -3.07% | | 16 | nightly | 0.31 | 6321 | 44.02 | - | | 32 | this branch | 0.45 | 9616.31 | 60.48 | **+11.44%** | | 32 | nightly | 0.41 | 9173.59 | 68.29 | - | | 128 | this branch | 0.85 | 15579.93 | 131.24 | -3.71% | | 128 | nightly | 0.89 | 13966.30 | 126.55 | - | ### GSM8k This branch: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9416|± |0.0065| | | |strict-match | 5|exact_match|↑ |0.9409|± |0.006| Nightly: |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9484|± |0.0061| | | |strict-match | 5|exact_match|↑ |0.9492|± |0.0060| ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52033", "createdAt": "2026-08-12T18:30:25Z", "updatedAt": "2026-08-13T15:19:49Z", "timestamp": "2026-08-13T15:19:49Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "rocm" ], "author": "simondanielsson", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:e1a943eace408cbf1577", "signalId": "github:vllm-project/vllm:issue:52170", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52170", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "OffloadingConnector AssertionError in _build_store_jobs under MultiConnector with high concurrency multi-turn", "text": "## Summary The OffloadingConnector's scheduler crashes with an `AssertionError` in `_build_store_jobs()` when running under a MultiConnector (NixlConnector + OffloadingConnector) with high-concurrency multi-turn agentic workloads. ## Environment - **vLLM**: v0.26.0 (`vllm/vllm-openai:v0.26.0`) - **Model**: RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block (Mamba-hybrid MoE, TP8) - **Hardware**: 4× H200 nodes (140GB each), 8 GPUs per node, InfiniBand RDMA - **Architecture**: Prefill/Decode disaggregation (3 prefill + 1 decode, TP8) - **KV transfer config**: MultiConnector with NixlConnector (kv_consumer) + OffloadingConnector (200GB CPU, lazy_offload=true) - **Workload**: 30 concurrent multi-turn conversations, 540 turns per conversation, 160K first-prompt tokens, 15s inter-turn delay, FP8 KV cache, prefix caching enabled ## Error ``` (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] EngineCore encountered a fatal error. (EngineCore pid=10329) ERROR 08-13 06:13:44 [core.py:1332] Traceback (most recent call last): File \".../vllm/v1/engine/core.py\", line 1323, in run_engine_core engine_core.run_busy_loop() File \".../vllm/v1/engine/core.py\", line 1364, in run_busy_loop self._process_engine_step() File \".../vllm/v1/engine/core.py\", line 1403, in _process_engine_step outputs, model_executed = self.step_fn() File \".../vllm/v1/engine/core.py\", line 645, in step_with_batch_queue scheduler_output = self.scheduler.schedule(self._should_throttle_prefills()) File \".../vllm/v1/core/sched/scheduler.py\", line 1174, in schedule meta = self._build_kv_connector_meta(self.connector, scheduler_output) File \".../vllm/v1/core/sched/scheduler.py\", line 1196, in _build_kv_connector_meta return connector.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 422, in build_connector_meta metadata=tuple( File \".../vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py\", line 423, in c.build_connector_meta(scheduler_output) for c in self._connectors File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py\", line 157, in build_connector_meta return self.connector_scheduler.build_connector_meta(scheduler_output) File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 1157, in build_connector_meta store_jobs=self._build_store_jobs(scheduler_output), File \".../vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py\", line 975, in _build_store_jobs assert len(offload_keys) == len(offload_block_ids) AssertionError ``` ## Reproduction The crash occurs on the **decode** pod after several minutes of sustained multi-turn traffic. It does not happen immediately on startup — the model loads and serves requests successfully before the offloading scheduler's internal state diverges. KV transfer config used: ```json { \"kv_connector\": \"MultiConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"connectors\": [ { \"kv_connector\": \"NixlConnector\", \"kv_role\": \"kv_consumer\", \"kv_connector_extra_config\": {\"bidirectional_kv_xfer\": true} }, { \"kv_connector\": \"OffloadingConnector\", \"kv_role\": \"kv_both\", \"kv_connector_extra_config\": { \"cpu_bytes_to_use\": 214748364800, \"lazy_offload\": true } } ] } } ``` vLLM flags: `--kv-cache-dtype fp8 --enable-prefix-caching --enable-expert-parallel --max-model-len 262144` ## Analysis The assertion at `scheduler.py:975` expects `offload_keys` and `offload_block_ids` to have the same length when building CPU offload store jobs. Under heavy multi-turn load with 30 concurrent conversations accumulating context over hundreds of turns, the offloading scheduler's block tracking appears to get out of sync — likely a race or bookkeeping error when blocks are evicted and reassigned rapidly. This is replicating the workload from the [llm-d Nemotron agentic serving guide](https://github.com/llm-d/llm-d/blob/main/guides/agentic-serving/nemotron-3-ultra-550b-h200.md) with the same MultiConnector configuration.", "url": "https://github.com/vllm-project/vllm/issues/52170", "createdAt": "2026-08-13T13:46:21Z", "updatedAt": "2026-08-13T13:46:21Z", "timestamp": "2026-08-13T13:46:21Z", "metrics": { "reactions": 1, "comments": 0 }, "labels": [], "author": "bbenshab", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6c08b6b5b9b1ceced46a", "signalId": "github:vllm-project/vllm:pull_request:52164", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52164", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Attention][DSA] Take the native decode path for MTP=3 on SM90", "text": "# [Attention][DSA] Take the native decode path for MTP=3 on SM90 ## Purpose Closes #35878. The DSA indexer flattens a spec-decode batch into one single-token row per query whenever `next_n` falls outside `{1, 2}`, so with MTP=3 (`next_n = 4`) each request's KV tile is read four times instead of once. DeepGEMM's `nv_dev` branch, which vLLM already pins (`cmake/external_projects/deepgemm.cmake`), implements `next_n = 4` on SM90 through a 2-CTA multicast cluster, so Hopper no longer needs that expansion. #45322 did the equivalent for SM100; this is the SM90 case the issue asks for. Behavior matrix (\"native\" = kernel sees the real batch `(B, next_n)`; \"flatten\" = requests expanded into `B × next_n` single-token pseudo-requests): | Platform | `next_n` (= spec tokens + 1) | Before | After | Changed? | |---|---|---|---|---| | SM100 (B200/GB200) | any | native | native | — | | **SM90 (H100/H200)** | **4 (MTP=3)** | **flatten** | **native** | ✅ **this PR** | | SM90 | 1, 2 | native | native | — | | SM90 | 3, ≥ 5 | flatten | flatten | — (no kernel) | | SM12x and others | 1, 2 | native | native | — | | SM12x and others | > 2 | flatten | flatten | — (unvalidated) | Three pieces: 1. The hardcoded gate becomes `_supports_native_decode(next_n)`, which asks `native_next_n_supported()` per architecture. SM90 implements `{1, 2, 4}` — note `3` is absent, so the gate is not a simple `>=` threshold. 2. `get_paged_mqa_logits_metadata` now sizes its own scheduler slots. `fp8_fp4_paged_mqa_logits` asserts the metadata against `num_sms / num_kv_multicast`, and SM90 `next_n = 4` runs one task per 2-CTA cluster rather than per SM, so the count must be halved. The wrapper derives `next_n` from `context_lens.shape[1]` and divides internally, so `num_sms` keeps its literal meaning and callers cannot get it wrong. The metadata buffer stays sized for one slot per SM; `build()` narrows it to a prefix view using the returned tensor's own shape. 3. Legality is a property of the step, not of the configuration: the kernel is handed `max_decode_len` Q rows, not the configured `next_n`, so a batch that happens to be uniformly 3 tokens deep still flattens on SM90. ## Test Plan Unit and kernel tests: ```bash pytest tests/v1/attention/test_indexer_native_next_n.py -v pytest tests/kernels/attention/test_deepgemm_attention.py -v ``` A standalone microbenchmark (not included in this PR) timed `fp8_fp4_paged_mqa_logits` on the same inputs fed natively as `(B, next_n)` and flattened to `B × next_n` single-token rows, to size the effect the decode path is trading on. End-to-end, DeepSeek-V3.2 on 8×H200, TP8 + EP, MTP=3. The \"before\" arm is this same tree with only `vllm/v1/attention/backends/mla/indexer.py` reverted to the flattening gate, so nothing else differs between arms. Every run was checked against the startup log line to confirm which path it took: ``` before: DSA indexer decode path: use_flattening=True supports_varlen=False (next_n=4, ...) after: DSA indexer decode path: use_flattening=False supports_varlen=False (next_n=4, ...) ``` Accuracy: ```bash cd tests/evals/gsm8k pytest -s -v test_gsm8k_correctness.py \\ --config-list-file=configs/models-h200.txt -k DeepSeek-V3.2-TP ``` `DeepSeek-V3.2-TP.yaml` only reports acceptance length when `min_acceptance_length` is set, so a local copy with `min_acceptance_length: 1.0` was used to make `get_acceptance_length()` print it. Its `startup_max_wait_seconds: 1200` is also not enough on a cold JIT cache — the first start on this machine took about 24 minutes. gsm8k runs at 4096 context while `index_topk` is 2048, so its top-k is close to \"select everything\" and it is not sensitive to the indexer's numerics. MRCR was run as a long-context check where the selection is genuinely sparse: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 32768 -tp 8 \\ --enable-expert-parallel --trust-remote-code \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' python tests/evals/mrcr/mrcr_eval.py --port 8000 --num-samples 24 --max-tokens 1024 ``` Serving performance: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 --max-model-len 40960 -tp 8 \\ --enable-expert-parallel --trust-remote-code --no-enable-prefix-caching \\ --num-gpu-blocks-override 7000 \\ --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' vllm bench serve --model deepseek-ai/DeepSeek-V3.2 --dataset-name random \\ --random-input-len 32000 --random-output-len 512 --random-range-ratio 0 \\ --num-prompts $C --max-concurrency $C --ignore-eos --seed 1234 # C in 1 4 8 12 ``` Two methodology notes, because a naive sweep gives misleading numbers here: * `--num-gpu-blocks-override` pins both arms to the same KV cache. Left to the memory profiler the two arms landed on 506k vs 483k tokens on one pair of runs (and the other way round on another), which by itself changes queueing. * `--num-prompts == --max-concurrency` keeps the run to a single wave, so it is decode-bound. At 32K a 143GB×8 node holds roughly 12 concurrent requests, so higher concurrency measures the scheduler queueing, not this kernel. Every configuration was run twice so the run-to-run spread is visible next to the effect. ## Test Result ### Unit and kernel tests ``` tests/v1/attention/test_indexer_native_next_n.py ......... 9 passed tests/kernels/attention/test_deepgemm_attention.py ..... 5 passed ``` `test_deepgemm_fp8_fp4_paged_mqa_logits[2-4]` covers `next_n = 4` against the reference implementation. ### Kernel-level effect (H200, `next_n = 4`) Native `(B, next_n)` vs flattened, speedup of the paged MQA logits call alone: \"image\" | batch | 4K ctx | 16K ctx | 32K ctx | |---|---|---|---| | 1 | 0.97x | 1.03x | 1.03x | | 4 | 1.02x | 1.10x | 1.15x | | 8 | 1.02x | 1.19x | 1.33x | | 16 | 1.07x | 1.25x | 1.33x | | 32 | 1.12x | 1.27x | 1.36x | | 64 | 1.27x | 1.45x | 1.46x | The gain grows with both batch and context, which is what sharing the KV tile predicts. At batch 1 / 4K the native path is 3% slower — flattening one request into four rows gives the kernel more parallelism than it can otherwise use. That case does not appear end-to-end (see below), so the gate is left unconditional. ### End-to-end serving (32K input, 512 output) \"image\" | concurrency | mean TPOT before | after | Δ | output tok/s before | after | Δ | |---|---|---|---|---|---|---| | 1 | 16.44 ms | 16.23 ms | −1.2% | 44.61 | 45.00 | +0.9% | | 4 | 28.02 ms | 26.00 ms | −7.2% | 88.85 | 89.81 | +1.1% | | 8 | 43.43 ms | 42.54 ms | −2.0% | 108.90 | 109.84 | +0.9% | | 12 | 60.99 ms | 58.88 ms | −3.5% | 117.19 | 118.46 | +1.1% | Values are the mean of two runs. Throughput improves by 0.9–1.1% at every concurrency, and at 1, 8 and 12 that clears each arm's own run-to-run range; the concurrency-4 cell does not (its TPOT spread is ±9%, larger than the effect). There is no regression at concurrency 1, so the batch-1 kernel result above does not surface in serving. The end-to-end gain is far smaller than the kernel speedup because the paged MQA logits kernel is a small share of a 671B MoE decode step.
Raw output, concurrency 12 Before (`use_flattening=True`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 52.52 Output token throughput (tok/s): 116.99 Total token throughput (tok/s): 7428.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 61.62 Median TPOT (ms): 60.08 P99 TPOT (ms): 93.81 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.12 Median ITL (ms): 33.42 ``` After (`use_flattening=False`): ``` ============ Serving Benchmark Result ============ Successful requests: 12 Maximum request concurrency: 12 Benchmark duration (s): 51.88 Output token throughput (tok/s): 118.43 Total token throughput (tok/s): 7520.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 59.60 Median TPOT (ms): 60.80 P99 TPOT (ms): 92.96 ---------------Inter-token Latency---------------- Mean ITL (ms): 65.36 Median ITL (ms): 32.64 ```
### Accuracy \"image\" gsm8k, 1319 questions, invalid rate 0.000 on every run: | run | before | after | |---|---|---| | 1 | 0.9560 | 0.9538 | | 2 | 0.9522 | 0.9553 | | 3 | — | 0.9530 | | **mean** | **0.9541** | **0.9540** | Mean acceptance length, which is the control that matters for comparing the performance numbers at all: | run | before | after | |---|---|---| | 1 | 3.074 | 3.069 | | 2 | 3.081 | 3.079 | | 3 | — | 3.078 | | **mean** | **3.077** | **3.075** | The arms overlap on both measures. At n = 1319 one standard error on accuracy is about 0.006, and the arms differ by 0.0001. MRCR at 32K context, where the indexer's top-k is genuinely selective: | | before | after | |---|---|---| | match_ratio | 0.4873 | 0.5637 | | prefix_hit_rate | 1.0000 | 0.9583 | | n=2 / n=4 / n=8 | 0.6494 / 0.4745 / 0.3380 | 0.8091 / 0.5510 / 0.3310 | One run per arm at 24 samples over three needle buckets, so this shows no degradation rather than a gain. ### Path coverage The change adds a per-step fallback for the case where `max_decode_len` is 3, which has no SM90 kernel. Instrumented runs across gsm8k, MRCR and the serving sweep recorded over 140,000 decode steps and saw only `max_decode_len` 1 (the draft model's own decode) and 4 (the verify step) — never 3 — so the fallback stays cold in steady-state MTP decoding. Both cudagraph modes were exercised: gsm8k under `--enforce-eager`, and the serving sweep with `CUDAGraphMode.FULL_AND_PIECEWISE` capturing all 49 decode sizes. --- Not duplicating existing work: no open PR references #35878, and none of the open sparse-indexer PRs touch the SM90 `next_n` gate — #47469 is SM100 varlen, #51555 and #43327 are ROCm, #47629 and #38476 are SM8x/SM12x backends. #45322 is merged and covers SM100 only. AI assistance was used for this change and its validation.", "url": "https://github.com/vllm-project/vllm/pull/52164", "createdAt": "2026-08-13T13:06:39Z", "updatedAt": "2026-08-13T13:42:03Z", "timestamp": "2026-08-13T13:42:03Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "verified" ], "author": "zobinHuang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b595d9aa9834e5859b57", "signalId": "github:vllm-project/vllm:pull_request:52163", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52163", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend][Rust] Bound chat-template evaluation to prevent DoS (#52025)", "text": "## Purpose Fixes #52025. The Rust frontend renders a caller-supplied (or model-supplied) Jinja `chat_template` from `POST /v1/chat/completions` with **no evaluation budget**. Rendering cost is `O(N^depth)` in caller-controlled loop bounds, and minijinja's per-`range()` element limit is trivially walked around by nesting three individually-legal `range()` calls. A **116-byte** request body can occupy a request-runtime worker thread for **tens of seconds**: ```jinja {% for x0 in range(999) %}{% for x1 in range(999) %}{% for x2 in range(999) %}{% endfor %}{% endfor %}{% endfor %}ok ``` The request runtime is a bounded pool (`worker_threads = min(available_parallelism, 32)`), so a small number of such requests can stall the chat-completions plane while `/health` stays green. The request-supplied gate (`trust_request_chat_template`, default off in Python vLLM) is not yet implemented in the Rust frontend, and the **model-supplied** template path has no gate at all. This wires up minijinja's existing `fuel` feature and sets a total evaluation budget in `build_environment` (`rust/src/chat/src/renderer/hf/template.rs`). The budget bounds total work regardless of who supplied the template, so a hostile template is rejected quickly while legitimate templates — which only iterate over the request's messages/tools/documents — are unaffected. For reference, minijinja itself already refuses a single `range()` above ~1M elements; the budget here (20M fuel units) is ~20× that, well above any real chat template. This is the primary remediation (item 1) from the issue. Implementing `trust_request_chat_template` and `spawn_blocking`-ing the render (items 2 and 4) are follow-ups and out of scope here. ## Test Plan `cargo test -p vllm-chat` — added two tests in `template.rs`: - `test_chat_template_evaluation_is_bounded`: the nested-`range()` bomb from the issue is now rejected (and returns quickly instead of running for ~55s). - `test_chat_template_within_budget_renders`: a legitimate template doing non-trivial work (`range(50000)`) still renders. ## Test Result ``` $ cargo test -p vllm-chat test result: ok. 267 passed; 0 failed test result: ok. 17 passed; 0 failed test result: ok. 16 passed; 0 failed ``` All existing `vllm-chat` tests pass (the budget does not affect any real template); `cargo fmt --check` and `cargo clippy -p vllm-chat` are clean.", "url": "https://github.com/vllm-project/vllm/pull/52163", "createdAt": "2026-08-13T12:43:07Z", "updatedAt": "2026-08-13T12:47:18Z", "timestamp": "2026-08-13T12:47:18Z", "metrics": { "reactions": 4, "comments": 1 }, "labels": [ "rust" ], "author": "jaideeppyne", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:29594c1297d8cccad062", "signalId": "github:vllm-project/vllm:pull_request:51655", "event": "changed", "observedAt": "2026-08-13T16:19:22.035158Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51655", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add Muse Glimmer model support", "text": "Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context. Adds the model, its config and processor, channel-scoped reasoning and ATEM tool-call parsers, and DFlash speculative decoding support for its draft head. The model does not emit JSON tool calls and does not wrap reasoning in tags. Every turn is a sequence of channel-scoped messages, and both parsers key off that framing, so --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer are used together. The reasoning parser forces skip_special_tokens=False; without it the markers are stripped before parsing and both channels collapse into content. tool_choice=\"required\" and named tool_choice set supports_required_and_named=False so vLLM does not apply JSON guided decoding to them -- that path assumes JSON tool calls, and forcing it here either trapped the call in the reasoning channel or leaked the raw framing into content. The DFlash draft head (MuseGlimmerAssistantModel) reuses the existing qwen3_dflash implementation: same architecture, same tensors. It reads the target's residual stream at layers [1, 13, 25, 37, 49] and predicts a 16-slot block per forward. Squashed from the onyx-support integration branch (30 commits). ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51655", "createdAt": "2026-08-10T10:20:30Z", "updatedAt": "2026-08-13T11:46:25Z", "timestamp": "2026-08-13T11:46:25Z", "metrics": { "reactions": 16, "comments": 34 }, "labels": [ "documentation", "new-model", "speculative-decoding", "ready", "tool-calling", "qwen", "mrv2" ], "author": "xianbaoqian", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8844509ed5b1f480c0f9", "signalId": "github:vllm-project/vllm:pull_request:43642", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43642", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Warm up hybrid GDN/Mamba/MRoPE kernels", "text": "## Purpose Hybrid Qwen-style GDN/Mamba/MRoPE models can trigger Triton compilation during the first inference request when startup warmup does not cover the exact runtime compile keys. This revision migrates the remaining warmups to the shared kernel-owned warmup contract and removes coverage that is already provided by current upstream `main`. ## Changes - Merge the latest upstream `main` and remove duplicate warmups for zero-KV blocks, slot mapping, causal convolution, fused post-convolution, fused sigmoid update, and chunked GDN prefill. - Add kernel-owned `VllmJitKernel` implementations for packed recurrent GDN decode and MRoPE. - Route runtime launches and warmup compilation through the same owners and exact `CompileKey` definitions. - Use compile-only warmup calls under `enable_jit_warmup`; no dummy model execution or startup synchronization is added. - Cover the integer specialization buckets used by MRoPE and the static meta-parameters used by packed GDN decode. ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/model_executor/test_hybrid_gdn_mamba_mrope_warmup.py -q CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest \\ tests/kernels/core/test_mrope.py -q pre-commit run --files \\ vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py \\ vllm/model_executor/layers/rotary_embedding/mrope.py \\ vllm/model_executor/warmup/hybrid_gdn_mamba_mrope_warmup.py \\ vllm/model_executor/warmup/kernel_warmup.py \\ vllm/third_party/flash_linear_attention/ops/fused_recurrent.py \\ tests/model_executor/test_hybrid_gdn_mamba_mrope_warmup.py ``` GPU compile-only and direct numerical smoke tests were run on GPU 1 (RTX 3090). ## Test Result - Shared-contract unit tests: 3 passed. - MRoPE native-versus-CUDA tests: 12 passed. - Compile-only smoke compiled both MRoPE keys and the packed recurrent GDN key. - Direct packed recurrent GDN runtime comparison had maximum absolute error `0.007812`, within the existing test tolerance. - All applicable pre-commit hooks, including mypy, passed. The existing packed recurrent numerical pytest still fails for fp16/bf16 on both this branch and an unmodified latest-upstream worktree; fp32 passes. This is therefore not introduced by this revision. AI assistance: Codex.", "url": "https://github.com/vllm-project/vllm/pull/43642", "createdAt": "2026-05-26T04:51:32Z", "updatedAt": "2026-08-13T17:42:29Z", "timestamp": "2026-08-13T17:42:29Z", "metrics": { "reactions": 2, "comments": 11 }, "labels": [ "needs-rebase", "v1" ], "author": "lesj0610", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:d203c295106fdce4b16b", "signalId": "github:vllm-project/vllm:issue:52023", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52023", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: draft_model speculative decoding crashes at init under TP>1 when draft hidden_size > target (TRT-LLM fused allreduce+RMSNorm workspace sized from target only)", "text": "### Summary With `draft_model` speculative decoding under tensor parallelism (`TP>1`), the engine crashes at init when the **draft model's `hidden_size` is larger than the target model's**. The `fuse_allreduce_rms` compilation pass builds the FlashInfer TRT-LLM fused allreduce+RMSNorm workspace with a token budget derived from the **target** model's `hidden_size` only. When the draft model then runs its own forward under TP, its larger `hidden_size` overflows that workspace and `check_trtllm_allreduce_fusion_workspace_metadata` raises, aborting startup. `draft_hidden < target_hidden` and `TP=1` both work; the crash is specific to **`draft_hidden > target_hidden` AND `TP>1`**. ### Environment - vLLM `v0.27.1`, V1 engine, `dtype=torch.bfloat16` - 2×H100 PCIe (80GB), `tensor_parallel_size=2` - `compilation_config.pass_config.fuse_allreduce_rms: True` (default), FlashInfer allreduce backend - Full raw server logs for all arms (`collect_env` included): https://github.com/seongyun1104/depthchart/tree/39b1557/draft_model_tp_bug ### Reproduction | Arm | target (TP) | draft | draft_hidden vs target_hidden | result | |-----|-------------|-------|-------------------------------|--------| | A | Qwen3-30B-A3B (**TP=1**) | Qwen3-4B | **2560 > 2048** | OK (acceptance 83.8%, normal) | | B | Qwen3-30B-A3B (**TP=2**) | Qwen3-4B | **2560 > 2048** | **CRASH** at init | | C | Qwen3-30B-A3B (TP=2) | Qwen3-0.6B | 1024 < 2048 | OK (acceptance 73.1%, coherent) | Arm A vs B isolates the trigger to **`TP>1`**: same `draft_hidden > target_hidden` pair, only the parallelism differs. Arm C shows the same TP=2 target is fine when `draft_hidden < target_hidden`. Arm B command (greedy): ``` vllm serve Qwen/Qwen3-30B-A3B \\ --tensor-parallel-size 2 \\ --max-model-len 4096 --max-num-seqs 8 --gpu-memory-utilization 0.9 \\ --override-generation-config '{\"do_sample\": false}' \\ --speculative-config '{\"method\": \"draft_model\", \"model\": \"Qwen/Qwen3-4B\", \"num_speculative_tokens\": 1}' ``` (Qwen3-4B `hidden_size=2560` as draft; Qwen3-30B-A3B `hidden_size=2048` as target.) ### Traceback (arm B, verbatim) ``` File \".../vllm/compilation/passes/fusion/allreduce_rms_fusion.py\", line 265, in call_trtllm_fused_allreduce_norm ... check_trtllm_allreduce_fusion_workspace_metadata( File \".../flashinfer/comm/trtllm_ar.py\", line 1026, in check_trtllm_allreduce_fusion_workspace_metadata raise ValueError(error_msg) ValueError: Workspace validation failed: - token_num (8192) * hidden_dim (2560) exceeds workspace max_token_num (8192) * hidden_dim (2048). This may cause Illegal Memory Access. ``` `hidden_dim (2560)` is the draft (Qwen3-4B); the workspace budget `hidden_dim (2048)` is the target (Qwen3-30B-A3B). ### Where it comes from The fused-allreduce workspace token budget is `max_token_num = max_size // (hidden_dim * element_size)`, and `hidden_dim` is taken from the target model config in the pass. The draft model's forward under TP issues the same fused allreduce with its own (larger) `hidden_size`, exceeding the target-sized budget. Note (not a proposed fix): patching the pass `__init__` to use `max(target_hidden, draft_hidden)` did **not** resolve it — the run still reports `hidden_dim (2048)` in the validation error, so the workspace metadata that is actually checked is created upstream of that pass field. A correct fix has to reach the workspace **creation** site so its stored `hidden_dim` accounts for the draft model. ### Relationship to #50877 / #50932 Same theme (FlashInfer allreduce workspace/buffer under a draft model that adds allreduce traffic), but a **distinct backend**: - #50877 / PR #50932 — the **MNNVL** path (`trtllm_mnnvl_ar`, Lamport buffer rotation); #50932 touches `flashinfer_all_reduce.py` / `fused_allreduce_gemma_rms_norm.py` and adds an `is_buffer_size_sufficient` admission guard. - This report — the **TRT-LLM fused allreduce+RMSNorm** compilation-pass path (`allreduce_rms_fusion.py` → `flashinfer/comm/trtllm_ar.py`), which #50932 does not touch. Filing separately so the TRT-LLM path is tracked; happy to fold into #50877 if maintainers prefer a single tracking issue.", "url": "https://github.com/vllm-project/vllm/issues/52023", "createdAt": "2026-08-12T17:01:32Z", "updatedAt": "2026-08-13T17:41:35Z", "timestamp": "2026-08-13T17:41:35Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4b5ef55a5545677df0d0", "signalId": "github:vllm-project/vllm:pull_request:44488", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:44488", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix sparse BlockStored event token/hash mapping", "text": "## Purpose Fixes #44451. `BlockStored` events can become ambiguous when a KV cache group skips logical blocks, such as Mamba groups with `--mamba-cache-mode align`. Before this change, `block_hashes` and `extra_keys` only included emitted non-null blocks, while `token_ids` still covered the full logical token range. This could produce sparse events like one block hash with multiple block-sized token chunks, without enough metadata for external KV event consumers to determine which token chunk belongs to the emitted hash. This PR adds optional `BlockStored.block_offsets`. When present, `block_offsets[i]` identifies the block-sized chunk in `token_ids` corresponding to `block_hashes[i]`. Dense events keep `block_offsets=None`. ## Test Plan Unit tests: ```bash python -m pytest \\ tests/v1/core/test_prefix_caching.py::test_block_stored_event_offsets_for_null_blocks \\ tests/v1/core/test_prefix_caching.py::test_block_stored_event_offsets_for_masked_blocks \\ -q ``` End-to-end validation used a Qwen3.5 Mamba align server with KV events enabled: ```bash vllm serve /mnt/models/Qwen/Qwen3.5-4B \\ --trust-remote-code \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 1 \\ --gpu-memory-utilization 0.90 \\ --enforce-eager \\ --skip-mm-profiling \\ --limit-mm-per-prompt '{\"image\": 0, \"video\": 0}' \\ --kv-events-config '{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:8100\"}' ``` The e2e verifier subscribes to the ZMQ KV event stream, sends OpenAI-compatible completion requests, and checks that every sparse BlockStored event has reconstructable hash-to-token mapping: ```bash import threading import time from collections import Counter from typing import Any import msgspec import requests import zmq from msgspec.msgpack import Decoder class EventBatch(msgspec.Struct, array_like=True, omit_defaults=True, gc=False): ts: float events: list[Any] data_parallel_rank: int | None = None class KVCacheEvent( msgspec.Struct, array_like=True, omit_defaults=True, gc=False, tag=True ): pass class BlockStored(KVCacheEvent): block_hashes: list[Any] parent_block_hash: Any | None token_ids: list[int] block_size: int lora_id: int | None medium: str | None lora_name: str | None extra_keys: list[tuple[Any, ...] | None] | None = None group_idx: int | None = None kv_cache_spec_kind: str | None = None kv_cache_spec_sliding_window: int | None = None block_offsets: list[int] | None = None class BlockRemoved(KVCacheEvent): block_hashes: list[Any] medium: str | None group_idx: int | None = None class AllBlocksCleared(KVCacheEvent): pass class KVEventBatch(EventBatch): events: list[BlockStored | BlockRemoved | AllBlocksCleared] stats = Counter() bad_events = [] sparse_events = [] def listen(stop_event): decoder = Decoder(type=KVEventBatch) ctx = zmq.Context() sub = ctx.socket(zmq.SUB) sub.connect(\"tcp://:8100\") sub.setsockopt(zmq.SUBSCRIBE, b\"\") poller = zmq.Poller() poller.register(sub, zmq.POLLIN) while not stop_event.is_set(): if not poller.poll(500): continue frames = sub.recv_multipart() stats[\"raw_zmq\"] += 1 try: _, _, payload = frames batch = decoder.decode(payload) except Exception: stats[\"decode_error\"] += 1 continue for event in batch.events: if not isinstance(event, BlockStored): continue stats[\"block_stored\"] += 1 token_blocks = len(event.token_ids) // event.block_size hash_blocks = len(event.block_hashes) if event.kv_cache_spec_kind: stats[f\"kind:{event.kv_cache_spec_kind}\"] += 1 if token_blocks > hash_blocks: stats[\"sparse\"] += 1 record = { \"group_idx\": event.group_idx, \"kind\": event.kv_cache_spec_kind, \"block_size\": event.block_size, \"token_blocks\": token_blocks, \"hash_blocks\": hash_blocks, \"len_token_ids\": len(event.token_ids), \"block_offsets\": event.block_offsets, } sparse_events.append(record) ok = ( event.block_offsets is not None and len(event.block_offsets) == hash_blocks and all(0 <= off < token_blocks for off in event.block_offsets) ) if not ok: bad_events.append(record) def send_requests(): base_url = \"http://:8000\" model = \"/mnt/models/Qwen/Qwen3.5-4B\" url = f\"{base_url}/v1/completions\" lengths = [500, 527, 529, 800, 1055, 1057, 1584, 2112, 2500, 3168] salts = [\"same-salt\", \"tenant-A\", \"tenant-B\"] for salt in salts: for n in lengths: prompt = f\"{salt}\\n\" + \" \".join(f\"tok{i}\" for i in range(n)) r = requests.post( url, json={ \"model\": model, \"prompt\": prompt, \"temperature\": 0, \"max_tokens\": 16, }, timeout=120, ) r.raise_for_status() def main(): stop_event = threading.Event() t = threading.Thread(target=listen, args=(stop_event,), daemon=True) t.start() time.sleep(1) send_requests() time.sleep(3) stop_event.set() t.join(timeout=2) print(\"Summary:\", dict(stats)) print(\"Sparse events:\") for event in sparse_events: print(event) if bad_events: print(\"BAD sparse events:\") for event in bad_events: print(event) raise SystemExit(1) if not sparse_events: print(\"No sparse events observed.\") raise SystemExit(2) print(\"PASS: sparse BlockStored events are reconstructable.\") if __name__ == \"__main__\": main() ``` ## Test Result ``` Unit tests pass. ``` Before the fix, reverting the code reproduces ambiguous sparse Mamba events: BAD sparse events: ``` {'group_idx': 0, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': None} {'group_idx': 1, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': None} {'group_idx': 2, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 2, 'hash_blocks': 1, 'len_token_ids': 1056, 'block_offsets': None} ``` After the fix, sparse Mamba events include offsets and pass validation: ``` {'group_idx': 0, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} {'group_idx': 1, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} {'group_idx': 2, 'kind': 'mamba', 'block_size': 528, 'token_blocks': 3, 'hash_blocks': 1, 'len_token_ids': 1584, 'block_offsets': [2]} PASS: sparse BlockStored events are reconstructable. ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/44488", "createdAt": "2026-06-04T04:28:25Z", "updatedAt": "2026-08-13T17:40:02Z", "timestamp": "2026-08-13T17:40:02Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "v1" ], "author": "Li-brua", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:ad39bfb9459e3cfdb36d", "signalId": "github:vllm-project/vllm:pull_request:44969", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:44969", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Gating more ROCm tests", "text": "Add 22 AMD mirrors for existing CI test groups as the Stage D gating expansion. ### `mi250_1` - Cudagraph - Docker Build Metadata - Kernels Mamba Test - Kernels Helion Test - Language Models Test (PPL) - Language Models Test (MTEB) ### `mi250_2` - Plugin Tests (2 GPUs) ### `mi300_1` - Model Runner V2 Spec Decode - PyTorch Compilation Unit Tests - Speculators Correctness - Language Models Test (Extended Generation) - Entrypoints Integration (Responses API) - Entrypoints Integration (Multimodal) - Async Engine, Inputs, Utils, Worker ### `mi300_2` - Model Runner V2 Distributed (2 GPUs) - Distributed Model Tests (2 GPUs) ### `mi300_4` - V1 e2e (4xH100-4xMI300) - LoRA TP (Distributed) ### `mi355_1` - Entrypoints Unit Tests - Multi-Modal Processor - Multi-Modal Models (Extended Generation 2) - Multi-Modal Models (Extended Generation 3)", "url": "https://github.com/vllm-project/vllm/pull/44969", "createdAt": "2026-06-09T06:56:07Z", "updatedAt": "2026-08-13T17:38:30Z", "timestamp": "2026-08-13T17:38:30Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "rocm", "ready", "needs-rebase", "ci/build", "v1", "nvidia" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:166923dd1eebba21cc83", "signalId": "github:vllm-project/vllm:pull_request:52197", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Support DSpark configs with `architectures=DSparkDraftModel` + `model_type=qwen3`", "text": "## Purpose Generic Qwen `DSparkDraftModel` is now normalized to `Qwen3DSparkModel` so models like https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark can now run on vLLM ## Test Plan ## Test Result Tested gsm8k with ``` vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned75 -tp=4 --spec-model RadixArk/Qwen3.8-2.4T-A95B-DSpark --spec-method dspark --spec-tokens 7 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52197", "createdAt": "2026-08-13T17:37:50Z", "updatedAt": "2026-08-13T17:38:26Z", "timestamp": "2026-08-13T17:38:26Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "qwen" ], "author": "mgoin", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:7d74baf519f3aea04b22", "signalId": "github:vllm-project/vllm:pull_request:52070", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52070", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf][Spec Decode] Disable EAGLE cache-hit block drop for always-K=0 dynamic DSD", "text": "## Purpose Opt-in skip of EAGLE-style last-block drop on prefix-cache hits when dynamic speculative decoding is **always K=0**. Default is off, so stock cache-hit behavior and observed greedy tokens are unchanged. Always-K=0 DSD (`num_speculative_tokens_per_batch_size=[[1,16,0]]`) produces no draft tokens, but `use_eagle=True` still drops the last matched KV block on a prefix-cache hit. That inflates scheduled-token volume on a long-context warm workload (stock: 4 steps / ~24k tokens per burst vs nospec ~1 step). This PR keeps `use_eagle=True` and only disables that drop when the operator sets `disable_eagle_cache_drop_for_k0=True` **and** all of: prefix caching on, no KV connector, `use_eagle()` (EAGLE / EAGLE3 / MTP / DFlash / DSpark), dense lookup `max(lookup[1:])==0`. Related: #49548 (always-K=0 slice only; not a universal fix). This is a **performance tradeoff**, not a correctness fix. Greedy outputs can change when the flag is on (measured divergence at generated pos 22 multi-request / pos 67 single-request vs stock K=0). ## Test Plan ```bash .venv/bin/python -m pytest \\ tests/v1/core/test_eagle_cache_drop_k0.py \\ tests/v1/core/test_mamba_align_chunk_split.py \\ tests/v1/core/test_prefix_caching.py::test_eagle_enabled_removes_last_block \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_partial_blocks \\ tests/v1/core/test_prefix_caching.py::test_eagle_with_sliding_window \\ tests/v1/core/test_prefix_caching.py::test_prefill_hybrid_model_combinations_eagle \\ tests/v1/core/test_prefix_caching.py::test_eagle_swa_alignment_caches_extra_block \\ tests/v1/spec_decode/test_dynamic_sd.py -q ``` Enable (example): ```python speculative_config={ \"method\": \"mtp\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [[1, 16, 0]], \"disable_eagle_cache_drop_for_k0\": True, } # plus enable_prefix_caching=True; no KV connector ``` ## Test Result CPU unit tests above: **64 passed** (2026-08-12, local). ThinkingCap-Qwen3.6-27B-FP8, dual RTX 5090 TP=2, always-K=0 `[[1,16,0]]`, ~49k prompt with prefix-cache hit, 8 sessions × 2 rounds × 128 tokens, inproc. GPU validation used commit `da159d447d` overlaid into the production venv. The only extra local change was an uncommitted scheduler JSONL helper for the burst census; it does not change scheduling decisions. **Primary comparison (this flag, one outing):** C0 **82.6 t/s** (n=1) vs B2 **136.7 t/s** (n=3, stdev 0.3), about **1.65×**. Mixed C3 `[[1,2,2],[3,16,0]]` does not enable the flag. **Burst census (same outing):** | Config | Warm steps/burst | Warm Σ tok/burst | |---|---:|---:| | Stock always-K=0 (C0) | 4 | 24,238 | | This change (B2, flag on) | 2 | 11,429 | | Coarse `use_eagle=False` (B1, earlier run) | 2 | 11,429 | B2 matches B1’s 4→2 / 24k→11k shape. `use_eagle` stays True. **Earlier local prototype (not this flag; do not mix with the numbers above):** same workload family, util 0.90, one-rep: C0 89.4 t/s, local B2 (`VLLM_SCHED_WARM_BATCH`) 147.0 t/s, B1 147.7 t/s. Nearby nospec ~178 t/s. Residual vs nospec is not attributed. Greedy identity vs stock K=0 with the flag on: **FAIL** at pos 22 (multi) / pos 67 (single), 1 rep each. **Small accuracy probe** (not GSM8K): 8 grade-school integer problems, greedy, ~2k-token shared stem cached first so the last-block drop actually runs. C0 **7/8**, B2 **8/8**. The seven unambiguous items matched. The eighth is an ambiguous wording trap (capacity 80 vs 80−15=65); C0 answered 65, B2 answered 80. Do **not** read 8/8 vs 7/8 as a quality win. Cliff check only: the flag did not break the clear math items. No named model eval (GSM8K / `tests/evals/`) was run. ## Why this is not a duplicate No open PR implements an always-K=0 opt-in skip of `drop_eagle_on_cache_hit`. **#50897** is the general correctness approach (successor-aware hashes so the last block can be kept without changing greedy tokens). It is **not** a substitute we can cite for this workload today: - 43 files; **merge-conflicting** as of 2026-08-11. Collaborator review asked to **split** it (too pervasive; still keeps the old drop path, so core complexity goes up). A rebase alone does not address that. No LGTM. - We have **not** measured #50897 on the always-K=0 ~49k warm-burst bench. Same speed as this flag is a hypothesis. This PR is a small, default-off option for that measured always-K=0 volume defect. If successor-aware hashing lands and removes the defect, **deprecate and remove this flag**. The two do not conflict today (off by default; no hash-format change). Not discussed on #50897. | Related | Difference | |---|---| | #50897 | Correctness-preserving hash redesign (see above). Not merge-ready as-is; unmeasured here. | | #51769 | Warns when EAGLE-style methods cost a large prefix-cache hit. Diagnostic only. | | #51295 | Hybrid attention miss/corruption from eagle drop. Different bug. | | #48375 / #45614 | Mamba eagle-drop plumbing / hit accuracy. Not DSD K=0 scheduling volume. | | #51466 / #51575 | Separate DSD work; #51466 was withdrawn after a negative policy experiment. Neither is bundled. | KV-connector / Mooncake Store eagle-drop paths are **out of scope** (gate requires `connector is None`). ## AI disclosure AI assistance was used to draft the implementation and tests. A human (Greg) is accountable for the PR and will review every changed line before merge. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52070", "createdAt": "2026-08-12T23:52:41Z", "updatedAt": "2026-08-13T17:35:53Z", "timestamp": "2026-08-13T17:35:53Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "Suppressor72", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1dae0bf1eec14d72ceea", "signalId": "github:vllm-project/vllm:pull_request:49652", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49652", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Fix autoregressive draft decode capture with dynamic SD", "text": "## Purpose Fixes #48494. Dynamic speculative decoding (DSD) derives the decode query lengths for each scheduled speculative-token count from the manager's configured `decode_query_len`: ```text num_new_sampled_tokens = decode_query_len - max_speculative_tokens runtime_query_len = scheduled_speculative_tokens + num_new_sampled_tokens ``` This derivation is valid when `decode_query_len` includes the maximum number of draft tokens, as it does for target verification and autoregressive draft prefill. It is not valid for autoregressive draft decode: every draft step processes exactly one token per request and dispatches the CUDA graph with `num_tokens == num_reqs` and `uniform_token_count == 1`. For example, with `max_speculative_tokens=3`, applying the DSD derivation to the draft decode manager's `decode_query_len=1` produces query lengths `1`, `-1`, and `-2` for scheduled K values `3`, `1`, and `0`. The non-positive descriptors eventually reach `InputBatch.make_dummy` and fail: ```text assert 0 < num_reqs <= num_tokens ``` This PR: - adds `use_dynamic_decode_query_len` to `CudaGraphManager`, defaulting to `True` to preserve existing behavior; - disables dynamic query-length derivation only for the autoregressive draft decode manager, whose query length is fixed at one; - keeps dynamic query-length derivation enabled for target verification, autoregressive draft prefill, DFlash, and all other existing callers; and - adds a production-wiring regression test for a representative `K=3/1/0` DSD schedule. This keeps full CUDA graph execution enabled. It does not clamp invalid shapes, swallow the assertion, or fall back to eager execution. Duplicate-work checks found no open PR referencing #48494 and no existing autoregressive Dynamic-SD CUDA Graph fix. Related PR #47737 skips non-positive query lengths for DSpark's dynamic draft manager when a K=0 tier disables drafting. That is complementary: the autoregressive draft decode manager has a fixed query length of one for every draft step and must not derive its shape from the DSD schedule at all. In particular, filtering non-positive derived lengths can leave it with no FULL decode graph when the runtime schedule does not contain the configured upper-bound K. PR #47131 implements D-cut verification-depth pruning and is also unrelated to this capture-shape bug. ## Test Plan ### Unit and static checks ```bash .venv/bin/python -m pytest -q \\ tests/v1/spec_decode/test_dynamic_sd_cug.py \\ tests/v1/cudagraph/test_cudagraph_manager.py .venv/bin/ruff check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py .venv/bin/ruff format --check \\ vllm/v1/worker/gpu/cudagraph_utils.py \\ vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py \\ tests/v1/spec_decode/test_dynamic_sd_cug.py git --no-pager diff --check ``` The regression test covers: - a valid K=3/1/0 runtime schedule below the configured upper-bound K=4; - dynamic draft-prefill query lengths; - fixed draft-decode query length `1`; and - full-CUDA-graph dispatch for every batch size from 1 through 8. ### GPU red/green reproduction Environment: - 2x NVIDIA A800, tensor parallel size 2; - V2 model runner; - Qwen3.5 architecture with native `Qwen3_5MTP`; - GDN attention path; - full CUDA graphs; and - DSD schedule `K=3/1/0`. ### Real-weight end-to-end smoke test The patched branch was also tested with the public `Qwen/Qwen3.5-0.8B-Base` checkpoint, including its native MTP weights: ```bash VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-0.8B-Base \\ --tensor-parallel-size 2 \\ --max-model-len 1024 \\ --max-num-seqs 8 \\ --gpu-memory-utilization 0.50 \\ --dtype bfloat16 \\ --language-model-only \\ --speculative-config '{ \"method\": \"mtp\", \"model\": \"Qwen/Qwen3.5-0.8B-Base\", \"num_speculative_tokens\": 3, \"num_speculative_tokens_per_batch_size\": [ [1, 2, 3], [3, 4, 1], [5, 8, 0] ], \"draft_tensor_parallel_size\": 2 }' \\ --compilation-config '{ \"mode\": 3, \"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"cudagraph_capture_sizes\": [8] }' ``` The same checkpoint and request set were then run with `--enforce-eager` as a numerical reference. ## Test Result ### Unit and static checks ```text Base: vllm-project/vllm@1479bd9e9 Environment: Python 3.11.2, torch 2.11.0+cu130, 2x NVIDIA A800-SXM4-40GB pytest: 6 passed ruff check: All checks passed ruff format --check: 3 files already formatted pre-commit: 10 applicable Python hooks passed git diff --check: passed ``` The pre-commit run covered Ruff check/format, typos, mypy 3.10, SPDX headers, root lazy imports, filenames, forbidden imports, new `torch.cuda` API usage, and boolean context-manager checks. ### GPU red/green result Before this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/3 failure: InputBatch.make_dummy -> AssertionError ``` After this patch: ```text target FULL capture: 3/3 draft prefill FULL capture: 3/3 draft decode FULL capture: 1/1 server: Application startup complete ``` ### Real-weight result - 9/9 requests succeeded across concurrency 1, 3, and 5, exercising scheduled K values 3, 1, and 0. - Speculative-decoding metrics confirmed that the MTP path generated and accepted draft tokens. - For the patch-active K=3 and K=1 tiers, all four full-CUDA-graph results matched eager results field-for-field. Repeating the eager run produced the same four results. - K=0 does not execute autoregressive draft decode and was therefore excluded from the patch-path numerical comparison. - Neither the full-CUDA-graph nor eager log contained `AssertionError`, `InputBatch.make_dummy`, or an `ERROR` record. The end-to-end smoke test used Qwen3.5 native MTP rather than the Gemma4 checkpoint from #48494. Both use the shared `AutoRegressiveSpeculator` draft decode manager fixed by this PR. This test validates the control flow and small-model numerical behavior; it does not make a large-model performance claim. ## AI assistance OpenAI Codex assisted with root-cause analysis, code drafting, and test drafting. I reviewed the changes and validation results, and this PR is ready for maintainer review.", "url": "https://github.com/vllm-project/vllm/pull/49652", "createdAt": "2026-07-24T02:00:33Z", "updatedAt": "2026-08-13T17:35:51Z", "timestamp": "2026-08-13T17:35:51Z", "metrics": { "reactions": 3, "comments": 11 }, "labels": [ "bug", "speculative-decoding", "v1", "nvidia", "mrv2" ], "author": "CZT0", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:891763b660e6873e5bd6", "signalId": "github:vllm-project/vllm:issue:49986", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:49986", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Perf] DSD arms pay a large baseline tax vs no-spec under production defaults; PIECEWISE override identified as one factor", "text": "## Motivation While benchmarking `num_speculative_tokens_per_batch_size` (Dynamic SD) schedules on H100 NVL 94GB (`prithivMLmods/gemma-4-31B-it-qat-FP8` target + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` draft, vLLM `c5d967c23`) I found that **every arm using a `speculative_config` pays a substantial throughput tax vs no-spec under production defaults**, and the gap is large enough at short ctx that it dominates the aggregate spec-bench signal for schedules that lower K on the short-ctx majority. Position-balanced 2-trial (T1 order: C→N→S→A, T2 reverse: A→S→N→C), cache-wipe + cold-start-burn methodology, `prefix_repetition` c=256, 3 warmup + 3 measure per cell. | ctx | no-spec (K=0) | static K=3 | 3-item batch schedule (A′) | 2D 6-cell schedule (C′) | |---:|---:|---:|---:|---:| | 400 | **2711.7** | 2139.8 (−21%) | 1875.6 (**−31%**) | 1890.7 (−30%) | | 900 | 1987.1 | 1838.9 (−7%) | 1453.5 (−27%) | 1874.6 (−6%) | | 1900 | 1815.2 | 1822.4 (~0%) | 1416.6 (−22%) | 1848.2 (+2%) | | 4000 | 1535.9 | 1692.8 (+10%) | 1232.8 (−20%) | 1680.4 (+9%) | Two observations: 1. **The gap is largest at short ctx**, where a K=0 tier should be nearly free (drafter is loaded but the verifier is not asked to check any drafted tokens). At ctx=400, the DSD-mode arms are 20-31% slower than no-spec. 2. **The gap narrows as ctx grows**, and the 2D schedule overtakes no-spec at ctx ≥ 1900. The mechanism is memory-bound decode amortization — but the crossover would happen much earlier if the baseline tax at short ctx were smaller. ## Identified factor: PIECEWISE override Server logs confirm the DSD arms are downgraded from `FULL_AND_PIECEWISE` to `PIECEWISE`: ``` WARNING vllm.py:829 Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` The trigger is in `vllm/config/vllm.py::_maybe_override_dynamic_sd_cudagraph_mode` (lines 824-841): ```python def _maybe_override_dynamic_sd_cudagraph_mode(self) -> None: if ( speculative_config is None or not speculative_config.uses_dynamic_speculative_decoding() or not self.compilation_config.cudagraph_mode.has_full_cudagraphs() or self.use_v2_model_runner ): return self.compilation_config.cudagraph_mode = CUDAGraphMode.PIECEWISE ``` So all DSD schedules (including trivial `[[1, 512, 3]]` static schedules) lose full CUDA graphs unless `VLLM_USE_V2_MODEL_RUNNER=1` is set. The no-spec baseline retains full graphs, which likely explains part of the −31% gap at short ctx. **But cudagraph is not the only factor.** In an eager-mode control (`enforce_eager=True`, both arms lose graphs equally) at ctx=1900, static K=3 was still 6% slower than no-spec (0.94×), not faster as the mechanism would predict. So there is a residual DSD-mode overhead beyond the cudagraph-mode difference — probably drafter forward on K=0 steps, spec bookkeeping, admission cost — that needs to be decomposed. ## Proposed decomposition + fix path 1. **Decompose the tax** into: (a) PIECEWISE downgrade, (b) drafter forward on K=0 steps, (c) spec scheduling bookkeeping, (d) tier-boundary ramp effects (a small fraction of steps at c≈192-256 fire lower tiers than the aggregate suggests, per the ramp-leak observation in RFC #48627 §Motivation). 2. **True K=0 fast path**: when a DSD schedule selects K=0 for a given (bs, ctx), route through the no-spec path entirely for that step (skip drafter forward, skip spec accounting). This should recover most of the short-ctx gap without touching cudagraph capture. 3. **K-keyed FULL cudagraph capture**: since a declarative schedule enumerates the K set at startup (unlike learned per-step K predictors), the (batch bucket × K)-shape graphs are statically enumerable. Capturing them removes the need for the PIECEWISE downgrade in the DSD path — the \"resource-neutral\" property claimed for declarative schedules becomes literal once graphs are keyed on K. ## Repro Config: `prithivMLmods/gemma-4-31B-it-qat-FP8` + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant`, H100 NVL 94GB, driver 580, CUDA 13.0, vLLM `c5d967c23` (branch `feat/dsd-2d-ctx-schedule` of #48944, but the tax is present with any DSD schedule including the merged `num_speculative_tokens_per_batch_size` from #32374). ```bash # static K=3 arm (reproduces the DSD-mode override): vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 \\ --speculative-config '{\"model\":\"\",\"num_speculative_tokens\":4,\"num_speculative_tokens_per_batch_size\":[[1,512,3]]}' # no-spec baseline: vllm serve --gpu-memory-utilization 0.90 --max-model-len 8192 # Bench (either arm): vllm bench serve --model --port 8000 \\ --dataset-name prefix_repetition --prefix-repetition-prefix-len 400 \\ --prefix-repetition-suffix-len 96 --prefix-repetition-num-prefixes 1 \\ --prefix-repetition-output-len 100 \\ --num-prompts 256 --max-concurrency 256 --ignore-eos \\ --percentile-metrics ttft,tpot,itl ``` Full raw data (server logs, per-cell measurements, snapshot deltas) available on request; happy to share the tarball or push to a public repo if that's easier for review. ## Context This came out of the measurement decomposition on #48944 — full 6-arm, 4-ctx table and methodology are in that PR's comment thread. The 2D ctx-axis schedule in #48944 is one specific instance of a DSD schedule; the baseline tax observed here applies to all DSD-mode arms (including the pre-existing `num_speculative_tokens_per_batch_size` API from #32374), so this issue is orthogonal to that PR and worth tracking separately. Happy to prototype the K=0 fast path as a follow-up PR if this direction sounds right — would appreciate a sanity check from @ekagra-ranjan (as the DSD API author) and @benchislett before I start.", "url": "https://github.com/vllm-project/vllm/issues/49986", "createdAt": "2026-07-27T13:04:39Z", "updatedAt": "2026-08-13T17:35:41Z", "timestamp": "2026-08-13T17:35:41Z", "metrics": { "reactions": 0, "comments": 10 }, "labels": [], "author": "seongyun1104", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8df98118508bea9df595", "signalId": "github:vllm-project/vllm:issue:49548", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:49548", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Performance]: Dynamic speculative decoding (num_speculative_tokens_per_batch_size) causes catastrophic aggregate-throughput collapse under concurrency at the batch-size threshold (MTP, V1/PIECEWISE)", "text": "### Report of performance regression Enabling **dynamic speculative decoding** via `num_speculative_tokens_per_batch_size` produces two effects on a Qwen3.5-122B MTP (k=2) setup: 1. **Expected:** the documented `FULL_AND_PIECEWISE → PIECEWISE` cudagraph downgrade, which costs ~14% single-stream (59.6 → 50.0 tok/s). This is fine and matches the warning. 2. **Unexpected:** a **catastrophic aggregate-throughput collapse under concurrency** that looks like a pathological stall, *not* just \"no speculation at high batch.\" With the schedule `[[1,4,2],[5,512,0]]` (MTP k=2 for batch ≤ 4, spec **disabled** for batch ≥ 5), 8 concurrent 180-token requests dropped from **~232 tok/s aggregate** (static `num_speculative_tokens: 2`) to **24–157 tok/s**, with **wall-clock 40–60 s** for a workload that completes in **~6.3 s** under the static config — same model, same prompts, same token counts. Single-stream under the *same* dynamic config was a coherent 50 tok/s (the expected PIECEWISE level), so the collapse is specific to concurrency crossing the schedule's batch-size threshold. The naive expectation is that at batch ≥ 5 (K=0) throughput should fall back to roughly the **non-spec** decode rate. Instead it is ~1.5–10× *worse* than a plain non-spec run would be. That gap is what this issue is about. #### Environment - vLLM **0.23.1rc1.dev1302+ge765bbc97** (eugr/spark-vllm nightly) - **NVIDIA GB10 / DGX Spark**, sm_121a, unified memory (single device) - Model: **Qwen3.5-122B-A10B INT4 (AutoRound)**, native **MTP** speculator, `num_speculative_tokens: 2` - Attention backend: **FLASH_ATTN** - `enforce_eager=false`, cudagraph_mode requested `FULL_AND_PIECEWISE` - `VLLM_USE_V2_MODEL_RUNNER` **not** set (V1 runner) #### Speculative config Static (baseline, ~232 tok/s agg8, ~6.3 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2} ``` Dynamic (collapses to 24–157 tok/s, 40–60 s wall): ```json {\"method\": \"mtp\", \"num_speculative_tokens\": 2, \"num_speculative_tokens_per_batch_size\": [[1, 4, 2], [5, 512, 0]]} ``` #### Numbers | Metric | static `k=2` | dynamic `[[1,4,2],[5,512,0]]` | |---|---:|---:| | Single-stream tok/s | 59.6 | 50.0 (expected PIECEWISE downgrade) | | Aggregate tok/s, 8 concurrent × 180 tok | ~232 | **24–157** | | Wall-clock, 8 concurrent | ~6.3 s | **40–60 s** | | Output token counts | identical | identical | #### Startup warning (as documented / expected) ``` Dynamic speculative decoding changes the target verification length at runtime. Overriding cudagraph_mode from FULL_AND_PIECEWISE to PIECEWISE for reliability. Use VLLM_USE_V2_MODEL_RUNNER=1 if you want to use full CUDA graphs. ``` We have **not** yet tested `VLLM_USE_V2_MODEL_RUNNER=1` with the dynamic schedule (there is a separate open report, #48494, that V2 + `num_speculative_tokens_per_batch_size` + MTP crashes at speculator decode capture, so we held off). ### Misc discussion on performance — code-reading hypotheses We can't attach a profiler run yet (the box is a shared production node), so we read `main` (`f83de6d`) to make this actionable. Candidate mechanisms, strongest first: **H1 — At K=0 the MTP drafter still runs a full draft-model forward every step.** `SpecDecodeBaseProposer.propose` short-circuits on K=0 only **after** the first-pass draft forward has already executed: - `vllm/v1/spec_decode/llm_base_proposer.py:580-590` runs `self.model(**model_kwargs)` (the MTP-layer forward), - then `vllm/v1/spec_decode/llm_base_proposer.py:606-618` returns an empty tensor with the explicit comment *\"The prefill forward pass above already ran to keep the drafter KV cache in sync.\"* The runner also has **no K==0 guard** — the drafter is invoked whenever `spec_config is not None` and `input_fits_in_drafter` (`vllm/v1/worker/gpu_model_runner.py:4570-4601` and the MTP branch at `:5096-5212`, which calls `drafter.propose(num_speculative_tokens=0, ...)`). So \"spec disabled at batch ≥ 5\" is **not** equivalent to a non-spec run: every decode step at high batch still pays a full MTP forward plus all input prep (`set_inputs_first_pass`, attention-metadata build) for zero drafts. Combined with the per-step spec-decode overhead already noted in #47277, this alone can erase — and invert — the expected fallback throughput. **H2 — Dynamic SD disables the uniform-spec padding that preserves full cudagraphs, forcing PIECEWISE + variable per-step shapes.** The decode-padding path that pads to uniform spec size is explicitly gated off whenever a dynamic schedule is present: `vllm/v1/core/sched/scheduler.py:822-838` (`(self.num_spec_tokens > 0 and self.dynamic_sd_lookup is None)`). This is the documented `FULL_AND_PIECEWISE → PIECEWISE` downgrade (`vllm/config/vllm.py:819-836`). Under concurrency, per-step query lengths vary as K flips at the batch-4↔5 boundary and as requests enter/leave, so the target forward (`gpu_model_runner._determine_batch_execution_and_padding`, `vllm/v1/worker/gpu_model_runner.py:3876`) and the drafter forward (`vllm/v1/spec_decode/llm_base_proposer.py:1780`) see a churn of shapes, each dispatched/padded separately with attention metadata rebuilt every step. The dispatcher pads up rather than recapturing, so this isn't naive recapture — but PIECEWISE replay + eager glue over a constantly-shifting shape set, at batch 8, on a 273 GB/s unified-memory device, compounds. **H3 — K is keyed off the raw scheduled-request count, so the threshold thrashes.** `vllm/v1/core/sched/scheduler.py:1116-1120` selects `num_spec_tokens_to_schedule = self.dynamic_sd_lookup[len(num_scheduled_tokens)]`. `len(num_scheduled_tokens)` counts **all** scheduled requests including prefills/chunks, so a single prefill joining 4 decodes bumps the count to 5 and silently disables spec for the whole step. As the running set oscillates around the threshold, K flips 2↔0 step-to-step, changing the scheduled verify-token count and therefore both forward shapes, defeating cudagraph reuse and forcing repeated pad/dispatch/metadata rebuilds. This interacts badly with H2. **H4 — KV lookahead is over-reserved at K=0.** `num_lookahead_tokens` is fixed to the static `num_spec_tokens` at init (`vllm/v1/core/sched/scheduler.py:234, 246-258`) and passed to `allocate_slots` as `effective_lookahead_tokens` (`:886-889, 912-925`) — it is **not** lowered to the dynamic K. So at batch 8 with K=0 the scheduler still reserves `num_spec_tokens` (=2) lookahead slots per request every step. Reserved lookahead slots are normally transient, but they still count against the free-block admission check, so under KV pressure this could trigger earlier can't-schedule/preemption events than a true non-spec run — which would present exactly as a throughput stall. (Lowest-confidence; flagging for completeness.) H1 and H2 are the most defensible and, together, would explain why K=0-at-high-batch is materially worse than plain non-spec rather than equal to it. ### Offer We have a **GB10 / DGX Spark (sm_121a)** available and are happy to run follow-up diagnostics if a maintainer wants them — e.g. a torch profiler / nsys trace of the 8-concurrent dynamic run, `vllm:spec_decode_*` counters at batch ≥ 5, per-step cudagraph dispatch mode logging, or an A/B with the H1 wasted-forward path short-circuited. Just say which knobs/logging you'd like enabled. ### Your current environment ```text vLLM 0.23.1rc1.dev1302+ge765bbc97 (eugr/spark-vllm nightly) NVIDIA GB10 / DGX Spark, sm_121a, 128 GB unified memory (~273 GB/s) Model: Qwen3.5-122B-A10B INT4 AutoRound, native MTP, num_speculative_tokens=2 Attention backend: FLASH_ATTN (collect_env.py output can be attached on request from the box) ``` ### Before submitting a new issue... - [x] Searched existing/past issues. Closest are #47277 (native MTP per-step overhead, static K, single-stream), #42505 (DFlash slower at concurrency > 8, static K), #49369 (DSpark aggregate halved, static K), and #48494 (`num_speculative_tokens_per_batch_size` + MTP crashes full-cudagraph capture under V2). None cover this dynamic-schedule **concurrency-stall** signature (single-stream fine, aggregate collapses at the batch threshold under V1/PIECEWISE).", "url": "https://github.com/vllm-project/vllm/issues/49548", "createdAt": "2026-07-23T07:25:02Z", "updatedAt": "2026-08-13T17:35:41Z", "timestamp": "2026-08-13T17:35:41Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [], "author": "tobby168", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bbd09c759637e4f9b3d4", "signalId": "github:vllm-project/vllm:pull_request:51466", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51466", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Fix DSD K-lookup to count sampling-only requests, preventing K thrashing at batch boundaries", "text": "> [!IMPORTANT] > **Withdrawn after controlled follow-up (2026-08-13).** This PR correctly > demonstrates that total-request and sampling-request lookup can choose > different K values, but I had not established that sampling-only was the > correct policy or that it reduced K thrashing. A later 25-pair controlled > experiment found no target workload meeting the predeclared benefit bar, > heavy-prefill workloads favored the existing total-count policy in every > pair, and complete-trace K transitions were 50 with sampling-only versus 40 > with total-count. I am closing the PR and retaining the original description > below as historical context. See the closure comment for details. ## Description The dynamic speculative decoding (DSD) K-lookup in `scheduler.py` uses `len(num_scheduled_tokens)` to index into `dynamic_sd_lookup`. This counts **all** scheduled requests, including mid-prefill/chunked-prefill requests. When prefills are mixed with decodes, they temporarily inflate the batch size and cause K to thrash between values as requests enter and leave prefill — a contributing factor to the catastrophic aggregate-throughput collapse reported in #49548. ### Example With schedule `[[1, 2, 2], [3, 16, 0]]` (K=2 for batch ≤ 2, K=0 for batch ≥ 3): | Step | Requests | Old count | Old K | New count | New K | |---|---|---|---|---|---| | 2 decode + 1 prefill (512 tok) | 3 | 3 | **0** | 2 (decode-only) | **2** | | 2 decode + 1 prefill (1 tok chunk) | 3 | 3 | **0** | 2 (sampling-only) | **2** | The old code flips K from 2→0 every time a prefill joins 2 decodes, then back to 2 when it leaves. Each flip changes the verify-token count, forward shape, and CUDA graph dispatch. ### Fix Count only requests that will **sample this step** (decode or final-prefill-chunk that reaches sampling), using the same phase-transition predicate the scheduler uses elsewhere: ```python req.num_computed_tokens + num_tokens >= req.num_tokens + req.num_output_placeholders ``` This correctly handles: - Pure decode requests - Mid-prefill chunked requests (excluded — will not sample) - Final prefill chunks that reach sampling (included — will sample) - 1-token prefill chunks that are NOT final (excluded) - Spec-padded decode requests (impossible in DSD mode — spec padding is gated on `dynamic_sd_lookup is None`) If no requests reach sampling (all mid-prefill), falls back to `len(num_scheduled_tokens)` to preserve original behavior. ## Benchmark Dual RTX 5090, TP=2, Qwen3.6-35B-A3B-Uncensored-FP8, MTP k=2, `VLLM_USE_V2_MODEL_RUNNER=1` + #49652 to preserve FULL CUDA graphs. | Scenario | Before (±stdev) | After (±stdev) | Δ | |---|---|---|---| | 4-session × 256 tok | 195 ± 164 t/s | **190 ± 49 t/s** | Variance −70% | | 8-session × 256 tok | 308 ± 67 t/s | **412 ± 163 t/s** | +34% mean | | Single-stream 32k | 285 ± 3 t/s | **289 ± 1 t/s** | Unchanged | The variance reduction confirms K thrashing is eliminated. The multi-session mean remains below no-spec baseline under long context — the cause is under investigation (see our correction comments on #49986 and #49548). The gap is context-length dependent and its root cause has not yet been identified. ## Complementary to #49652 This fix targets the scheduler-side K-selection pathology (#49548 H3). PR #49652 fixes the worker-side MRV2 CUDA graph capture crash (#48494). Both are needed for functional DSD with FULL CUDA graphs on MRV2. ## Test - `py_compile vllm/v1/core/sched/scheduler.py` — PASS - `pytest test_dynamic_sd_sampling_only_k_lookup` — 1 passed (8/8 edge cases) - Live runtime verified on dual RTX 5090 with Qwen3.6-35B-A3B FP8 + MTP + MRV2 + DSD Note: vLLM native scheduler tests require full dev install (`pip install -e .`). The standalone test validates the core predicate logic. Fixes part of #49548. ## AI Assistance Disclosure AI tools (codex CLI) were used in the development of this PR. All changes were reviewed and tested by the human submitter (Greg Weyer).", "url": "https://github.com/vllm-project/vllm/pull/51466", "createdAt": "2026-08-08T00:14:53Z", "updatedAt": "2026-08-13T17:35:34Z", "timestamp": "2026-08-13T17:35:34Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "speculative-decoding" ], "author": "Suppressor72", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:379c19731de7f4f3da2d", "signalId": "github:vllm-project/vllm:pull_request:52196", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52196", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Mamba][Quantization] Add Stage 1 (CPU-only, reference) int8 quantization utilities for SSM state", "text": "## Purpose Adds a self-contained, pure-PyTorch reference implementation of per-channel dynamic int8 quantize/dequantize helpers for the Mamba/SSM recurrent state (`vllm/model_executor/layers/mamba/state_quant.py`), plus a `QuantizedSSMState` wrapper modeling the read/compute/write pattern a real integration would use each decode step. **Motivation:** vLLM already ships FP8/TurboQuant/NVFP4 quantization for the attention KV cache (see `STR_DTYPE_TO_TORCH_DTYPE` in `vllm/utils/torch_utils.py`) because that cache is read and rewritten every decode step and is memory-bandwidth bound. The Mamba/SSM state (conv + temporal state, used by Mamba1/2, GatedDeltaNet, KDA, ShortConv and every hybrid model built on them - Jamba, NemotronH, Zamba2, Qwen3.5, FalconH1, etc.) has the exact same profile, but `MambaDType` (`vllm/config/cache.py`) only allows `float32`/`float16`/`bfloat16` today, and `MambaStateDtypeCalculator` (`vllm/model_executor/layers/mamba/mamba_utils.py`) has no quantize/dequantize step anywhere in it. Interestingly, `STR_DTYPE_TO_TORCH_DTYPE` already contains `\"int8\"`/`\"fp8*\"` entries (reused from KV-cache dtype strings), and `MambaBase.bind_kv_cache` (`vllm/model_executor/layers/mamba/abstract.py`) already slices the raw per-block byte page and reinterprets it per declared dtype/shape - so a smaller-dtype state would already be sized correctly by the existing allocator. What's missing is the actual quantize-before-write / dequantize-before-compute step, since the selective-scan recurrence itself must run in floating point. ## Scope / what this is NOT This change is intentionally self-contained and is **NOT** wired into `MambaDType`, `CacheConfig`, or any real mixer/kernel (`mamba_mixer.py`, `mamba_mixer2.py`, the `gdn`/`kda`/`short_conv` variants). Doing that safely means touching the compiled CPU/Triton/CUDA selective-scan kernels and every per-architecture mixer forward pass, which I don't think can be responsibly done through source-browsing/PR authorship alone without a real local test run on the actual kernels. See the module docstring in `state_quant.py` for the full list of deferred follow-up work (Literal/config plumbing, per-mixer wiring, fused kernel). I'm opening this as a numerically-verified building block / RFC-style prototype rather than a claim that quantized Mamba state is fully usable today. ## Test Plan Added `tests/kernels/mamba/test_state_quant.py`: CPU-only pytest tests covering quantize/dequantize round-trip error bounds, int8 value range, the all-zero-state edge case, rejecting non-int8 input to dequantize, the `QuantizedSSMState` wrapper's round-trip and in-place update, and numerical drift of a simulated diagonal recurrence when the state is quantized after every step. ## Test Result I have visually verified the pasted source is syntactically well-formed, but I have not run this test suite myself in a real Python/PyTorch environment. Please run `pytest tests/kernels/mamba/test_state_quant.py` before relying on this. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described above (int8 quantization utilities for the Mamba/SSM state, as a Stage 1 prototype/RFC). - [x] A test plan is included above. - [ ] Test results are NOT included (not yet run by the author - see Test Result section above for why). - [x] This is a breaking-change-free, additive-only change (two new files; no existing files modified).
", "url": "https://github.com/vllm-project/vllm/pull/52196", "createdAt": "2026-08-13T17:34:41Z", "updatedAt": "2026-08-13T17:35:17Z", "timestamp": "2026-08-13T17:35:17Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "Laalinibh", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:b6644516b374898e456c", "signalId": "github:vllm-project/vllm:pull_request:43107", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:43107", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core][WIP] Check for GPU<->CPU sync during CI", "text": "vLLM now uses asynchronous scheduling by default and in the majority of cases. Performance relies on the absence of any gpu<->cpu synchronizations on the main cuda stream, but such syncs can be opaque and it is easy for them to creep in accidentally. This change adds a `VLLM_GPU_SYNC_CHECK` env var which enables `torch.cuda.set_sync_debug_mode` for the model forward pass and sampler, so that we can easily check for such syncs. I'm trying first to enable it globally in the CI to flush out syncs that need to be fixed or where they are unavoidable and the check needs to be suppressed. A new `gpu_sync_allowed()` context manager can be used to wrap known syncs (i.e. temporary or unavoidable). This PR replaces https://github.com/vllm-project/vllm/pull/40561 - rebased now that man of the exposed \"low hanging\" sync fixes have been merged to main.", "url": "https://github.com/vllm-project/vllm/pull/43107", "createdAt": "2026-05-19T15:45:29Z", "updatedAt": "2026-08-13T17:34:59Z", "timestamp": "2026-08-13T17:34:59Z", "metrics": { "reactions": 3, "comments": 23 }, "labels": [ "rocm", "speculative-decoding", "ready", "ci/build", "v1", "multi-modality", "qwen", "kv-connector", "nvidia", "ready-run-all-tests", "quantization", "mistral", "cohere" ], "author": "njhill", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ca37bc26159b00ea7c8a", "signalId": "github:vllm-project/vllm:pull_request:50594", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50594", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Fix MiniMax M3 prompt reasoning initialization", "text": "## Purpose Fixes #46042. Fixes #50549. MiniMax-M3 adaptive reasoning can leak a leading `` into streaming `delta.content`. The existing initialization scans the entire rendered prompt with `is_reasoning_end()`. Because the prompt can contain paired `` markers as instructions or examples, the parser may incorrectly conclude that reasoning has already ended before generation starts. This change: - adds a prompt-boundary-specific reasoning-state hook while preserving the existing behavior by default - initializes MiniMax-M3 from `thinking_mode`: `enabled` starts in reasoning, `disabled` starts after reasoning, and `adaptive` lets generated markers determine the state - uses the hook in Chat Completions, the Responses API, streaming parsing, and structured-output initialization - preserves the deferred `adaptive=None` state through the structured-output gate instead of falling back to a full-prompt scan - scopes generated-marker checks to output tokens so instructional prompt markers cannot activate the grammar early - constrains unresolved adaptive sampling to grammar-valid tokens plus every decoded reasoning-marker tokenization - replays adaptive direct-content tokens through the grammar, including ambiguous prefixes and speculative draft state - respects the final reasoning boundary when continuing an existing assistant message, including explicit open continuations under disabled mode, and scopes continuation marker scans to that final message only - preserves open and closed `thinking` content-part state during continuation - propagates incomplete Responses reasoning items to frontend and engine parsers with the same request-local state - clears continuation-only parser state and refreshes the frontend parser after Responses built-in tool turns rerender the prompt - forwards request-local reasoning mode and prompt state through offline `LLM.chat()` and `enqueue_chat()` into the synchronous engine - keeps batch reasoning parsers request-local so continuation state cannot leak between conversations - initializes each engine-side structured-output parser from its prompt exactly once - adds focused regressions for instructional markers, adaptive marker boundaries, structured output, and all MiniMax-M3 thinking modes ## Related work and scope - #45718 fixed split-token MiniMax-M3 marker parsing, but not prompt-boundary state initialization. - #50152 identified the same full-prompt scan failure and demonstrated the production symptom, but closed without merging. - #48550 also addresses prompt-mode initialization while adding broader speculative-decoding and tool-parser changes. - This PR limits the fix to prompt-boundary reasoning initialization across online and offline chat paths and Responses, plus the structured-output state transitions needed to preserve it. ## Test Plan - Run focused parser and structured-output regressions: `uv run pytest -q tests/reasoning/test_minimax_m3_reasoning_parser.py tests/v1/structured_output/test_reasoning_structured_output.py` - Run offline chat prompt-state regression: `uv run pytest -q tests/entrypoints/llm/test_offline_reasoning_state.py` - Run batch prompt-state regressions: `uv run pytest -q tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py -k batch_forwards_prompt_reasoning_state` - Run Responses continuation regressions: `uv run pytest -q tests/entrypoints/openai/responses/test_serving_responses.py` - Run static validation on the changed Python files. - Serve `MiniMaxAI/MiniMax-M3` with the patch and send adaptive and enabled streaming chat-completion requests, including a structured response format. - Confirm that reasoning markers never appear in `delta.content`, reasoning remains in the reasoning field, and the output grammar starts only after generated reasoning ends or advances the first direct adaptive content token. ## Test Result Passed: - MiniMax-M3 reasoning parser: 44 tests passed - reasoning-aware structured output: 21 tests passed - offline chat prompt-state forwarding: 1 test passed - batch prompt-state forwarding: 3 tests passed - Responses serving: 23 tests passed, 1 expected xfail - `ruff check`: all checks passed - `ruff format`: all changed files formatted - Python `compileall` - `git diff --check` The 92 focused tests were run in a CPU-only temporary environment with `VLLM_TARGET_DEVICE=empty`; the repository-wide GPU CI has not run yet. Live MiniMax-M3 evaluation is pending because this workspace has no GPU/model deployment. It should be completed before merge. ## AI assistance OpenAI Codex assisted with implementation and regression-test drafting. The commits include `Assisted-by` trailers. The submitter requested the changes after reviewing the proposed scope.", "url": "https://github.com/vllm-project/vllm/pull/50594", "createdAt": "2026-07-31T16:34:24Z", "updatedAt": "2026-08-13T17:34:44Z", "timestamp": "2026-08-13T17:34:44Z", "metrics": { "reactions": 2, "comments": 24 }, "labels": [ "bug", "structured-output", "frontend", "tool-calling" ], "author": "jinbagi", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9f4c6c9826782aa1693b", "signalId": "github:vllm-project/vllm:pull_request:52168", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52168", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Restore multimodal support on the plain \"vllm\" throughput backend", "text": "## Purpose `vllm bench throughput --backend vllm` crashes on any multimodal dataset (e.g. ASR/Whisper) with: ValueError: Multi-modal content is not supported on backend 'vllm'; use one of []. Regression from #50981. `get_requests()` computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handle: ```python mm_backends = (\"vllm-chat\",) if args.backend == \"vllm-chat\" else () ``` This is circular — it only lets the already-selected backend through the gate, and only if it's literally \"vllm-chat\". So --backend vllm always computes mm_backends=() and self-rejects, even though _run_vllm_requests (the function that executes --backend vllm requests) already forwards multi_modal_data straight into LLM.generate — it was always multimodal-capable, same as vllm-chat. Only hf (asserts prompts must be str) and mii (text-only) genuinely can't carry multimodal content, so the allowlist should be static rather than derived from the chosen backend: ```python mm_backends = (\"vllm\", \"vllm-chat\") ``` ## Test Plan - Added test_get_requests_allows_multimodal_on_plain_vllm_backend to tests/benchmarks/test_throughput_cli.py, mirroring the existing test_get_requests_resolves_mmvu pattern: stubs ASRDataset (network-free), parses --backend vllm --dataset-name hf --dataset-path openslr/librispeech_asr, asserts get_requests() resolves without raising. - Ran the exact vllm bench throughput command from the bug report end-to-end on real hardware (Intel XPU B70) before and after the fix. - ruff check / ruff format on both changed files. ## Test Result - New test: fails with the reported ValueError against pre-fix code, passes against post-fix code. - python -m pytest tests/benchmarks/test_throughput_cli.py -k \"allows_multimodal_on_plain_vllm_backend or resolves_mmvu\" → 2 passed. - End-to-end on B70: pre-fix reproduces the ValueError verbatim; post-fix completes cleanly (Throughput: 20.96 requests/s, 5386.74 total tokens/s). - ruff check: all checks passed. ruff format --diff: no changes. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52168", "createdAt": "2026-08-13T13:27:39Z", "updatedAt": "2026-08-13T17:34:35Z", "timestamp": "2026-08-13T17:34:35Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug", "performance" ], "author": "mganczarenko", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a8d20085126c64e07029", "signalId": "github:vllm-project/vllm:pull_request:52195", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52195", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Add Stage 1 (CPU-only, reference) BitNet ternary quantization backend", "text": "## Purpose Adds an initial CPU-only, reference implementation of a BitNet b1.58-style ternary ({-1, 0, 1}) weight quantization backend for linear layers (`vllm/model_executor/layers/quantization/bitnet.py`), registered through the existing `register_quantization_config` out-of-tree plugin mechanism (no core files touched). There is real, currently-unclaimed demand for this: #33142 (open feature request), and #18213 / #17279 (closed as \"not planned\" due to inactivity, not rejected on merit). Weight quantization: absmean ternary rule per BitNet b1.58 (Eq. 7), packed 2 bits/value (4 values/byte). ## Scope / what this is NOT This is intentionally a correctness-first Stage 1: online (load-time) quantization of a bf16/fp16 checkpoint into the ternary representation, unpack-then-matmul execution path. It does **not** include a fused Triton/CUDA kernel, reading pre-packed ternary checkpoints, or end-to-end validation against a real BitNet checkpoint (e.g. microsoft/bitnet-b1.58-2B-4T) - all flagged here as follow-up work for a later PR, not included in this change. ## Test Plan Added `tests/quantization/test_bitnet.py`: CPU-only pytest tests covering the pack/unpack round trip, rejection of non-2D input, equivalence of `BitNetLinearMethod.apply` against a manual dequantize+matmul reference, and the no-bias path. ## Test Result I have visually verified the pasted source is syntactically well-formed, but I have not run this test suite myself in a real Python/PyTorch environment, nor exercised it against a real checkpoint or on GPU. Please run `pytest tests/quantization/test_bitnet.py` and a smoke import test before relying on this - I'd appreciate maintainer/CI verification given how this was authored. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR is described above (prototype BitNet ternary quantization backend, addressing #33142). - [x] A test plan is included above. - [ ] Test results are NOT included (not yet run by the author - see Test Result section above for why). - [x] This is a breaking-change-free, additive-only change (new file + new OOT-registered quant method; no existing files modified).
", "url": "https://github.com/vllm-project/vllm/pull/52195", "createdAt": "2026-08-13T17:33:34Z", "updatedAt": "2026-08-13T17:34:13Z", "timestamp": "2026-08-13T17:34:13Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "quantization" ], "author": "Laalinibh", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:13a7942be2d4c9151600", "signalId": "github:vllm-project/vllm:pull_request:50620", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50620", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][NIXL] Include transfer mode (push/pull) in the compatibility hash", "text": "#### Overview: Include the NIXL transfer mode (push vs pull) in the connector so a push (WRITE) connector and a pull (READ) connector can never be paired, and so an external router can distinguish them. Follow-up to #49230 (now merged), addressing review feedback from @iyastreb ([#49230 thread](https://github.com/vllm-project/vllm/pull/49230#discussion_r3686841923), [this PR's thread](https://github.com/vllm-project/vllm/pull/50620#discussion_r3702697050)). #### Details: The push (`NixlPushConnector`, WRITE) and pull (`NixlConnector`, READ) connectors use incompatible transfer protocols, but nothing prevented them from being paired across prefill/decode, and the transfer mode wasn't visible to an external router. **Compatibility hash (worker side):** - `NixlBaseConnectorWorker._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorWorker`, threaded into `compute_nixl_compatibility_hash`. - Add `transfer_mode` to the compatibility-hash factors so mismatched connectors are rejected early at handshake with a clear message. - Bump the NIXL connector version **6 → 7** and document the new factor. **Router support (scheduler side):** - `NixlBaseConnectorScheduler._TRANSFER_MODE = \"pull\"`, overridden to `\"push\"` in `NixlPushConnectorScheduler`. - Emit `transfer_mode` in the `kv_transfer_params` returned by both the pull and push `request_finished` paths, so an external router can route pull vs push producers. Consumed by [vllm-project/router#187](https://github.com/vllm-project/router/pull/187). **Not a duplicate:** this transfer-mode work was split out of #49230 (which fixed the speculative-config factors) to keep that PR scoped; no other open PR addresses mixing push/pull connectors. **Tests run:** - `.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_connector.py -k \"transfer_mode_changes_compatibility_hash or scheduler_advertises_transfer_mode or speculative_config_compatibility_hash\" -q` — **passed**. - `pre-commit run --files ` — all applicable hooks **passed**. - GPU-dependent handshake tests were not run locally (macOS, no CUDA); relying on CI. AI assistance was used to investigate, implement, and test this change. Every changed line was reviewed by the human submitter before submission. #### Where should the reviewer start? - `vllm/distributed/kv_transfer/kv_connector/v1/nixl/metadata.py` — `compute_nixl_compatibility_hash` (new `transfer_mode` factor + version bump). - `base_worker.py` / `push_worker.py` — worker `_TRANSFER_MODE`. - `base_scheduler.py` / `pull_scheduler.py` / `push_scheduler.py` — scheduler `_TRANSFER_MODE` emitted into `kv_transfer_params`. - `tests/v1/kv_connector/unit/test_nixl_connector.py` — hash test, scheduler-mode test, and the pull `kv_transfer_params` assertion.", "url": "https://github.com/vllm-project/vllm/pull/50620", "createdAt": "2026-07-31T19:41:03Z", "updatedAt": "2026-08-13T17:32:37Z", "timestamp": "2026-08-13T17:32:37Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "bug", "documentation", "ready", "kv-connector" ], "author": "tzulingk", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d372cb6d3a908d7a15ec", "signalId": "github:vllm-project/vllm:pull_request:52194", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52194", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[LoRA][MoE] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts", "text": "# [LoRA][MoE] Fix PEFT 0.18+ `target_parameters` LoRA loading for 3D MoE experts ## Summary PEFT 0.18+ can train a LoRA against a **fused 3D MoE expert parameter** via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`) rather than against an `nn.Module`. When it does, it reads the parameter `[E, dim1, dim2]` positionally as `[E, in=dim1, out=dim2]`. vLLM's `add_lora_w13` / `add_lora_w2` punica kernels use the opposite `nn.Linear` `[out, in]` convention, so per expert the trained delta is the transpose of what vLLM expects: `ΔW_vllm = ΔW_peft^T`. On stock vLLM this is **not** a silent no-op — `_stack_moe_lora_weights` reshapes the adapter with the wrong dimension semantics and `set_lora` raises a shape mismatch that kills the engine. Serving a real Qwen3.5-35B-A3B (`Qwen3_5MoeForConditionalGeneration`) expert LoRA fails in `vllm/lora/layers/fused_moe.py`: ``` RuntimeError: The size of tensor a (512) must match the size of tensor b (2048) at non-singleton dimension 2 ``` (512 = `moe_intermediate_size`, 2048 = `hidden_size`.) ## Fix 1. **`peft_helper.py`** — parse the `target_parameters` list from `adapter_config.json`. 2. **`lora_model.py`** — propagate `target_parameters` onto `LoRAModel` (constructor, `clone()`, and `from_lora_tensors`). 3. **`model_manager.py`** — in `_stack_moe_lora_weights`, detect the PEFT 3D layout and, when present, swap `lora_A`/`lora_B` and transpose the trailing two dims **after** the existing EP-aware expert slicing (so the fix composes with expert parallelism). ### Why the layout must be *detected*, not assumed `create_dummy_lora_weights` builds dummy MoE LoRA buffers in the **native** layout that shares the same flat shape as the PEFT layout, so an unconditional transpose would corrupt native/dummy adapters. Detection uses: - **Primary signal:** `target_parameters` names a fused expert weight (`experts.gate_up_proj` / `experts.down_proj` / `experts.w13_weight` / `experts.w2_weight`). - **Fallback** (older adapters that don't record `target_parameters`): native `lora_a` trailing dim equals `hidden_size`; the PEFT layout does not. ## Testing All results below were produced against this branch's **source** (editable install, `VLLM_USE_PRECOMPILED=1 uv pip install -e .`) on an H200 — not a monkeypatch over a released wheel. ### Unit tests (added in this change) ``` python -m pytest -v \\ tests/lora/test_peft_helper.py::test_peft_helper_target_parameters_direct \\ tests/lora/test_lora_manager.py::test_target_parameters_indicates_3d_lora \\ tests/lora/test_lora_manager.py::test_from_lora_tensors_propagates_target_parameters \\ tests/lora/test_peft_helper.py::test_peft_helper_pass \\ tests/lora/test_lora_manager.py::test_from_lora_tensors ``` → **10 passed** (the discriminator test is parametrized over 6 cases). The last two are pre-existing tests on adjacent code, included as a regression check. ### End-to-end numerical proof (Qwen3.5-35B-A3B expert LoRA) Bug reproduced: serving the raw PEFT adapter on the **unpatched** loader crashes in `set_lora` with the `512 vs 2048` error above. With the fix, three arms are compared by **chosen-token logprobs** (see caveat): base (no adapter), an independent offline-converted oracle (native layout, self-checks `max|ΔW_vllm − ΔW_peft^T| = 0.000e+00` over 320 experts), and the raw PEFT adapter loaded directly. | prompt | [effect] oracle-vs-base max\\|Δlogprob\\| | [equiv] raw-vs-oracle max\\|Δlogprob\\| | |---|---|---| | chest pain dx | 0.2808 | 0.0000 | | night blindness | 0.1846 | 0.0000 | | neonatal meningitis | 0.1028 | 0.0000 | | diabetic antihypertensive | 0.1601 | 0.0000 | | PKU enzyme | 0.0758 | 0.0000 | - **[effect]** the adapter demonstrably changes output (one prompt even flips the greedy answer ordering). - **[equiv]** the directly-loaded raw adapter is **bit-identical** to the offline-converted oracle across all 32 tokens × 5 prompts, no greedy-path divergence → the in-loader transpose matches the independently-verified transform. **VERDICT: PASS.** **Testing caveat (why logprobs, not decoded text):** for an experts-only (8-of-256, rank-32) adapter, greedy decoded text is a false-negative trap — the adapter shifts logits substantially yet often does not flip the greedy argmax on short prompts, so text can look identical to base even when the adapter is correctly applied. The verdict is therefore numerical. ## Model evaluation This change is the difference between a hard crash and a correctly-applied adapter for the affected adapter class; the numerical equivalence-vs-oracle result above is the output-affecting evidence. Broader `tests/evals/` accuracy runs can be added if reviewers want them. ## AI assistance disclosure AI assistance (Claude) was used to author this change. A human submitter has reviewed every changed line and run the tests above.", "url": "https://github.com/vllm-project/vllm/pull/52194", "createdAt": "2026-08-13T17:29:22Z", "updatedAt": "2026-08-13T17:32:29Z", "timestamp": "2026-08-13T17:32:29Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [], "author": "feixiangpeng", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:f0288b75b527b0677098", "signalId": "github:vllm-project/vllm:pull_request:43449", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:43449", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Formatting] Collapse multi-line arg lists where possible", "text": "This is a very large PR because it adjusts a ruff formatting rule which affects a lot of files. This might be partially personal preference but I find it much cleaner when lists / method defs are collapsed onto a single line when they fit. It reduces the lines of code and how much vertical real estate you have to scan to understand things. And just looks cleaner imo. I can understand the case for vertically formatted lists in that it's easier to see quickly the length and delineated items, but feel that the cons outweigh the pros. This formatting change reduces vLLM LoC by 32,000. The actual change is to `pyproject.toml`, adding `skip-magic-trailing-comma = true`: ```toml [tool.ruff.format] docstring-code-format = true skip-magic-trailing-comma = true [tool.ruff.lint.isort] split-on-trailing-comma = false ```` Anyhow please feel free to vote on this! P.S. I guess merging this all in one go may introduce a lot of conflicts to existing PRs, so perhaps we could apply incrementally across the codebase.", "url": "https://github.com/vllm-project/vllm/pull/43449", "createdAt": "2026-05-22T20:57:47Z", "updatedAt": "2026-08-13T17:32:13Z", "timestamp": "2026-08-13T17:32:13Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "needs-rebase", "ci/build", "v1", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "rust", "kimi", "cohere" ], "author": "njhill", "state": "open", "assignees": [ "sangstar" ], "change": "new" } }, { "id": "event:1aeacb4aca2af487df2d", "signalId": "github:vllm-project/vllm:pull_request:43327", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:43327", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Add per-call decode budget to sparse-MLA indexer", "text": "## Purpose The ROCm AITER sparse-MLA indexer decode path allocates a fresh fp32 logits tensor on every call to `rocm_fp8_paged_mqa_logits`: `(heads, B*next_n, max_model_len)` on the stage1 3D path, or `(B*next_n, max_model_len)` on the gfx942/gfx950 fused 2D path. With long context and a large running batch the per-call working set is multi-GiB; across ~60 layers per decode forward the caching allocator fragments and OOMs. Prefill had the same problem and got `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` plus chunking in #36178. This is the decode mirror. Two pieces, decode-only, ROCm-only: - the fp32 logits buffer is now drawn from a per-process workspace keyed by (device, shape) and refilled with `-inf` between calls, so no per-layer alloc/free of multi-GiB tensors; - `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` (default 4096) bounds the per-call working set. When a single call would exceed the budget the batch is split into sub-batches that fit and copied into the full output. The aiter kernel does not consume `schedule_metadata` on ROCm, so the metadata doesn't need to be re-derived per chunk. Set to 0 to restore the legacy unbounded behaviour. Prefill is not touched. The CUDA path is not touched (this file only runs on ROCm via the AITER sparse-MLA backend). Both `_ON_GFX942` and `_ON_GFX950` are covered, plus the stage1 3D else-branch that older arches still hit. Related: #36178 is the prefill-side mirror (same shape of fix, opposite forward-pass phase). #41002 is in flight in the same area, happy to rebase on top of it after it merges. ## Test Plan 1. Numeric equivalence test added at `tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py`. Skipped on non-ROCm and when the AITER `paged_mqa_logits` module is not importable, so it sits with the rest of the ROCm-only kernel tests. On a ROCm gate it asserts the chunked path returns bit-identical logits to the full-batch call. ```bash pytest -s -v tests/v1/attention/test_rocm_aiter_mla_sparse_decode.py ``` 2. Synthetic micro-bench on a single MI355X (gfx950), 60 back-to-back calls into `rocm_fp8_paged_mqa_logits` with a pre-allocated paged kv at `B=1024, next_n=1, heads=64, head_dim=128, block_size=64, max_model_len=131072`. Stress shape, picked so the 3D per-call working set is well above any reasonable budget. Measured peak `torch.cuda.max_memory_allocated()` minus the paged-kv baseline, sweeping `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` against the legacy code. 3. End-to-end re-run of the GLM-5.1-FP8 customer repro that motivated this: ```bash vllm bench serve --random-input-len 1 --random-output-len 2 --num-prompts 1024 \\ --ignore-eos --temperature 0 --seed 0 \\ --gpu-memory-utilization 0.9 -tp 8 --enforce-eager ``` ## Test Result Indexer working set at the high-water mark, 60-layer loop: | setting | indexer working set @ peak | | -------------------------------------------------- | -------------------------: | | legacy (no patch) | 33792 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB` unset (= 4096) | ~4160 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=2048` | ~2080 MiB | | `VLLM_SPARSE_INDEXER_DECODE_MAX_MB=1024` | ~1040 MiB | The 33.8 GiB legacy delta is exactly `heads * B * next_n * max_model_len * 4` bytes for the 3D stage1 buffer; chunking brings it down to `chunk_b * heads * next_n * max_model_len * 4` plus a one-shot 2D accumulator. Wall time across all four settings stays between 1.09 and 1.24 s for the 60-call loop, the only overhead is one `copy_` per chunk into the persistent output buffer. End-to-end on GLM-5.1-FP8, the decode-time HIP OOM that originally fired at ~24 GiB per layer no longer occurs; 1017/1024 prompts succeed, the 7 remaining failures are bench-client side (\"never received a valid chunk\") and clear with `--max-concurrency 256`.", "url": "https://github.com/vllm-project/vllm/pull/43327", "createdAt": "2026-05-21T15:29:18Z", "updatedAt": "2026-08-13T17:31:29Z", "timestamp": "2026-08-13T17:31:29Z", "metrics": { "reactions": 3, "comments": 7 }, "labels": [ "rocm", "v1" ], "author": "reger-men", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:0ef446a5e855d262a313", "signalId": "github:vllm-project/vllm:pull_request:45802", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:45802", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine", "text": "## Purpose Add token-aware reasoning token counting for the Streaming Parser Engine and surface the count through OpenAI-compatible usage fields. - Adds `completion_tokens_details.reasoning_tokens` to usage responses. - Propagates token counts through the parser engine pipeline: `TokenIDScanner -> IncrementalLexer -> StreamingParserEngine -> SemanticEvent`. - Counts only `REASONING_CHUNK` tokens, excluding reasoning boundary terminals such as `` / ``. - Reuses the engine-based streaming parser path for non-streaming parsing when output token IDs are available, so streaming and non-streaming reasoning token accounting use the same token-aware boundaries. - Ensures parser instances are scoped per output choice in chat completion handling. - Adds coverage for parser-engine token counting, chat completion usage, and Responses API reasoning token accounting. ## Test Plan ## Test Result ``` curl http://localhost:8001/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -H \"Authorization: Bearer EMPTY\" \\ -d '{ \"model\": \"\", \"messages\": [ { \"role\": \"user\", \"content\": \"北京什么天气？\" } ],\"chat_template_kwargs\":{\"enable_thinking\": true}, \"tools\": [ { \"type\": \"function\",\"strict\":true, \"function\": { \"name\": \"get_weather\", \"description\": \"Get current weather for a city\", \"parameters\": { \"type\": \"object\", \"properties\": { \"city\": { \"type\": \"string\" } }, \"required\": [\"city\"] } } } ], \"tool_choice\": \"auto\", \"stream\": false }' {\"id\":\"chatcmpl-80b2a0ffa82be55a\",\"object\":\"chat.completion\",\"created\":1782098609,\"model\":\"/mnt/data4/models/Qwen/Qwen3.5-35B-A3B-FP8\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":null,\"refusal\":null,\"annotations\":null,\"audio\":null,\"function_call\":null,\"tool_calls\":[{\"id\":\"chatcmpl-tool-ac88f083f78ccffb\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\": \\\"北京\\\"}\"}}],\"reasoning\":\"用户询问北京的天气情况。我需要使用get_weather工具来获取北京的当前天气信息。\\n\"},\"logprobs\":null,\"finish_reason\":\"tool_calls\",\"stop_reason\":null,\"token_ids\":null,\"routed_experts\":null}],\"service_tier\":null,\"system_fingerprint\":\"vllm-0.23.1rc1.dev243+gb529bfd6c-tp2-4b4f7693\",\"usage\":{\"prompt_tokens\":277,\"total_tokens\":323,\"completion_tokens\":46,\"prompt_tokens_details\":null,\"completion_tokens_details\":{\"reasoning_tokens\":18}},\"prompt_logprobs\":null,\"prompt_token_ids\":null,\"prompt_text\":null,\"kv_transfer_params\":null} ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/45802", "createdAt": "2026-06-16T09:31:21Z", "updatedAt": "2026-08-13T17:29:02Z", "timestamp": "2026-08-13T17:29:02Z", "metrics": { "reactions": 2, "comments": 8 }, "labels": [ "frontend", "ready", "tool-calling" ], "author": "chaunceyjiang", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:aa751d03639a8966fae1", "signalId": "github:vllm-project/vllm:pull_request:38484", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:38484", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Build] Add SM121 (DGX Spark / GB10) to published build targets", "text": "## Summary - Add compute capability 12.1 (SM121) to `TORCH_CUDA_ARCH_LIST` in all build configurations: release wheel CI, Dockerfile, docker-bake.hcl, versions.json, and FlashInfer AOT build - `CMakeLists.txt` already lists 12.1 in `CUDA_SUPPORTED_ARCHS` for CUDA 12.8+ and 13.0+, and all kernel-specific arch lists (CUTLASS, Marlin FP8, MoE, NVFP4) already include 12.1 — but the build scripts never passed 12.1 in `TORCH_CUDA_ARCH_LIST`, so the intersection filtering excluded it from all published artifacts - This is the minimal change needed so that published wheels and Docker images include SM121 kernels for DGX Spark / GB10 users out of the box ## Context The NVIDIA GB10 GPU in DGX Spark reports compute capability 12.1 (SM121). Without native kernels in published builds, users must either build from source or rely on PTX JIT compilation via the 12.0+PTX forward-compat path (slower first-run, no guarantee of correctness for all kernels). **Upstream PyTorch fix:** The CMake auto-detection bug that corrupts `12.1` → `12.1(2.0)` when building PyTorch from source on SM121 hardware has fixes in review: [pytorch/pytorch#174065](https://github.com/pytorch/pytorch/pull/174065) and [pytorch/pytorch#173754](https://github.com/pytorch/pytorch/pull/173754). I independently reproduced and verified both fixes on DGX Spark hardware (see comments on those PRs). PyTorch cu130 aarch64 nightly wheels already ship with `12.0+PTX` which covers SM121 via JIT. **Relationship to existing PRs:** [#31740](https://github.com/vllm-project/vllm/pull/31740) includes these build config changes among 27 files of broader SM121 platform support, but has been `CONFLICTING` since January. This PR extracts just the build target additions — 5 files, 6 lines changed — to unblock SM121 in published artifacts independently. Addresses #36821. ## Test evidence Tested on NVIDIA DGX Spark: | Component | Value | |-----------|-------| | GPU | NVIDIA GB10 | | Compute capability | 12.1 (SM121) | | Architecture | aarch64 | | OS | Ubuntu 24.04.4 LTS | | CUDA | 13.0, V13.0.88 | | Driver | 590.48.01 | Confirmed that `CMakeLists.txt` already supports 12.1 in `CUDA_SUPPORTED_ARCHS` and all kernel-specific arch lists (`SCALED_MM_ARCHS`, `FP4_ARCHS`, `MLA_ARCHS`, `CUTLASS_MOE_DATA_ARCHS`, `MARLIN_FP8_ARCHS`, `MARLIN_MOE_FP8_ARCHS`). The only gap was the `TORCH_CUDA_ARCH_LIST` in build scripts. ## Files changed | File | Change | |------|--------| | `.github/workflows/scripts/build.sh` | Add `10.0 12.0 12.1+PTX` to release wheel arch list | | `docker/Dockerfile` | Add `12.1` to both `torch_cuda_arch_list` ARGs (lines 191, 804) | | `docker/docker-bake.hcl` | Add `12.0 12.1` to default arch list | | `docker/versions.json` | Match Dockerfile ARG update | | `tools/flashinfer-build.sh` | Add `12.1` to CUDA 13.0+ FlashInfer AOT arch list | ## Test plan - [ ] Verify `docker buildx bake --print` shows 12.1 in resolved `torch_cuda_arch_list` - [ ] Verify wheel build with CUDA 13.0 includes SM121 kernels (`cuobjdump` check) - [ ] Run vLLM inference on DGX Spark with published image (no `--enforce-eager` needed)", "url": "https://github.com/vllm-project/vllm/pull/38484", "createdAt": "2026-03-29T18:15:26Z", "updatedAt": "2026-08-13T17:28:39Z", "timestamp": "2026-08-13T17:28:39Z", "metrics": { "reactions": 1, "comments": 8 }, "labels": [ "needs-rebase", "ci/build", "nvidia" ], "author": "JCorners68", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:e818097cceef6cc385df", "signalId": "github:vllm-project/vllm:pull_request:42773", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:42773", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] laguna xs gsm8k test", "text": "## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42773", "createdAt": "2026-05-15T19:41:52Z", "updatedAt": "2026-08-13T17:26:20Z", "timestamp": "2026-08-13T17:26:20Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "ready", "needs-rebase" ], "author": "mgoin", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:fb4d31de11638d444d48", "signalId": "github:vllm-project/vllm:pull_request:51280", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51280", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][CI] Solidify entrypoint LLM lifecycle", "text": "- Replace direct `LLM(...)` construction throughout `tests/entrypoints` with the shared `VllmRunner` lifecycle. - Add an `ExitStack`-backed runner factory for tests that need one long-lived runner or several concurrent runners. - Consolidate multimodal, structured-output, offline-mode, collective-RPC, pooling, and weight-transfer cleanup onto the complete runner shutdown path. - Defer ROCm VRAM settling while multiple runners coexist, then release runner references and perform one bounded wait after every engine exits. - Preserve direct-constructor behavior where required, including pooling model length and chunked-prefill defaults. This addresses AMD CI instability caused by entrypoint tests constructing `LLM` directly and relying on partial or ad hoc cleanup. Those paths could leave engine processes or lazily reclaimed ROCm allocations alive long enough to starve a later model startup. Routing construction through `VllmRunner` guarantees engine shutdown, Dynamo reset, distributed cleanup, and bounded VRAM settling even when a test fails. The multi-engine GPU-utilization case requires deferred settling because no individual runner can reach baseline while its peers remain live.", "url": "https://github.com/vllm-project/vllm/pull/51280", "createdAt": "2026-08-06T16:25:21Z", "updatedAt": "2026-08-13T17:24:04Z", "timestamp": "2026-08-13T17:24:04Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "rocm", "structured-output" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:51fad9150ac20a01c482", "signalId": "github:vllm-project/vllm:pull_request:51520", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51520", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RL] Add nccl-m2n sharding-aware weight transfer (worker side)", "text": "## Purpose Adds `nccl_m2n`, the inference-side half of a sharding-aware weight transfer backend built on [NCCL M2N](https://github.com/NVIDIA/nccl-extensions), per [RFC #46439](https://github.com/vllm-project/vllm/issues/46439). The broadcast NCCL backend assumes both sides hold the same layout, so a trainer that shards its parameters (FSDP, EP) must all-gather each one to a full tensor before sending. `nccl_m2n` moves each parameter with a single `reshard` that redistributes it between two disjoint meshes inside one communicator — trainer ranks `[0, T)`, inference workers `[T, T + N)`, which is exactly the interval vLLM's existing `rank_offset` convention already produces. The trainer sends its local shards and never materializes a full tensor. This PR is the **worker side only**. Each worker receives the whole tensor and hands it to `load_weights`, which is what the broadcast backend does for every parameter — so the two are directly comparable, and the destination layout is the same for every parameter, derived from the rank split with no negotiation. An external trainer can drive it today through the existing four-phase control plane (HTTP or Ray). Two follow-ups, already written and stacked behind this: 1. the in-tree trainer engine (`M2NWeightSource` / `DTensorModuleSource`, `M2NTrainerWeightTransferEngine`) plus a runnable FSDP → TP example; 2. per-parameter destination layouts, so a worker receives only its own shard. Selected with `--weight-transfer-config '{backend: nccl_m2n}'`. Default behavior is unchanged, and the runtime is imported lazily, so vLLM is unaffected unless the backend is selected. ## Not a duplicate Checked per `AGENTS.md`: ```bash gh pr list --repo vllm-project/vllm --state open --search \"46439 in:body\" # none gh pr list --repo vllm-project/vllm --state open --search \"m2n\" # none relevant gh pr list --repo vllm-project/vllm --state open --search \"weight transfer backend\" ``` No open PR references RFC #46439 or adds an m2n backend. The two adjacent ones solve a different problem: - **#40828 (WPI)** — a Kubernetes-native zero-copy transport for moving weights from external trainers. It is a transport substitution; it does not reshard between differing parallelism layouts, which is the entire point here. - **#43375 (RDT weight sync, draft)** — a WIP prototype for RDT sharded transfer. Overlapping in goal, but a different transport and still a prototype with no test plan. If it lands first, the two backends coexist in the factory registry the same way `nccl` / `ipc` / `sparse_nccl` do. ## Test Plan ```bash python -m pytest tests/distributed/test_weight_transfer_m2n.py -v ``` The suite covers how a transfer is *described*, not the transfer itself: layout encoding (including the size-1-axis spelling of full replication), the dtype and tensor-rank limits, init/update-info validation, and backend registration. Those are the failures worth catching cheaply — a bad layout or a ragged plan puts the two sides into mismatched collectives, which hangs rather than raises, so everything is validated during `init_transfer_engine` before any rendezvous. The transfer path needs the `nccl-extensions` runtime and multiple GPUs, so it is exercised out of tree against a real trainer. ## Test Result - `ruff check` / `ruff format --check` — clean on all touched files. - Layout invariants and the replicated-destination derivation verified against a stubbed `torch`, confirming both sides compute the same destination layout independently. - `pytest tests/distributed/test_weight_transfer_m2n.py` — **14/14 passed** (first run; this suite had never been executed before). Python 3.12.13, pytest 9.1.1, vLLM editable install on branch `nccl-m2n-worker` @ `1e7b332a6`. ## Model evaluation This PR has only the worker side change. We will need the trainer side change as well to make an end-to-end model run. The latter will be submitted in a stack PR. ## Known limitations Inherited from the m2n build and documented in `docs/training/weight_transfer/m2n.md`: - tensor rank 1..3; no fp4; `Partial` placements unsupported - the trainer's device mesh must cover a contiguous rank interval from 0, 1-D or 2-D - m2n caps per-destination-shard fan-in at 16 source shards (`MAX_SOURCES`, a compile-time bound in `reshard_limits.h`; raising it needs a library rebuild). This is not a rank limit in general — with a sharded destination each shard is fed by only a few sources. It binds here because a *replicated* destination is a single shard fed by every source shard, so a trainer that shards its parameters is limited to 16 ranks until the sharded-destination follow-up lands. A replicated trainer is unaffected at any scale. - requires `nccl-extensions` (NCCL ≥ 2.30.5) and `VLLM_NCCL_SO_PATH` pointing at the same `libnccl.so` that `libnccl_m2n.so` was linked against — torch bundles an older NCCL, so this is load-bearing rather than cosmetic. ## AI assistance This PR was written with AI assistance (Claude Opus 5 via Claude Code). Every line has been reviewed by the submitter, who understands and can defend the change end to end.", "url": "https://github.com/vllm-project/vllm/pull/51520", "createdAt": "2026-08-08T21:29:42Z", "updatedAt": "2026-08-13T17:23:55Z", "timestamp": "2026-08-13T17:23:55Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "needs-rebase" ], "author": "kwen2501", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a98bb0181ade8df0e920", "signalId": "github:vllm-project/vllm:pull_request:50729", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50729", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Mamba] Fix overlapping state copy race", "text": "[PR #30877](https://github.com/vllm-project/vllm/pull/30877) introduced generic Mamba state copies, and [PR #40172](https://github.com/vllm-project/vllm/pull/40172) added the fused GPU copy. A speculative-decode convolution-state shift can copy within the same physical block with overlapping source and destination ranges. Parallel memcpy-style loads/stores do not provide memmove ordering, which explains the intermittent [AMD CI failure](https://buildkite.com/vllm/amd-ci/builds/11556/list?sid=019fbc8d-6c5e-4172-83cb-f45a49946645&tab=output). This version is rebased over [PR #49436](https://github.com/vllm-project/vllm/pull/49436): - Temporal state copies retain #49436's 3D grid and 16-way tiled copy. - DS convolution copies use stable row-to-lane ownership while iterating tokens low to high. - SD convolution copies retain `_memcpy_u64_tiled(..., NUM_TILES=1)` for distinct physical blocks and exact self-copies, and use an ordered token copy only for a true same-physical-block left overlap. - Conv copies remain single-CTA (`tile_idx > 0` returns), so no grid-wide barrier is assumed or required. - The generic fallback is unchanged. MRv1 and MRv2 already share this fused copy kernel for the relevant hybrid/speculative/aligned path; routing every MRv1 configuration through the MRv2 lifecycle would break unsupported/custom state layouts such as ReplaySSM and would not cover the CPU fallback. - Existing overlap assertions derive expected values from untouched snapshots. The broad Python-vs-fused golden uses disjoint pages so the generic memcpy helper is not treated as a memmove oracle. ## Validation - `HIP_VISIBLE_DEVICES=0 .venv/bin/python -m pytest -q -s tests/v1/worker/test_mamba_utils.py`: **37 passed** - `uvx pre-commit run --files vllm/v1/worker/mamba_utils.py tests/v1/worker/test_mamba_utils.py`: **passed** - The original failing revision reproduced locally on MI355; the rebased implementation passed the same snapshot checks. - Qwen3.5-0.8B MTP checks across 8×MI355, MRv1/MRv2, SD/DS, and batch 1/8 produced matching output hashes. A counterbalanced device-kernel benchmark on MI355 covered both runners, SD/DS, same-page and distinct-page copies, and batch 1/8. For a 24-layer Qwen3.5-shaped state copy: | Path | Batch | Result vs current main | |---|---:|---:| | SD same-page, MRv1 | 1 / 8 | +14.7 µs / +1.1% | | SD same-page, MRv2 | 1 / 8 | +18.0 µs / +1.2% | | SD distinct-page, both runners | 1 / 8 | -2.5% / +1.7–1.8% | | DS, both runners and mappings | 1 / 8 | 95.5–98.5% lower latency | The SD same-page cost is the overlap-safe ordering; the common distinct-page path retains the tiled memcpy. MRv1 and MRv2 copy-kernel latency is effectively equivalent because they already execute the same fused mechanism. This PR was developed with assistance from OpenAI Codex.", "url": "https://github.com/vllm-project/vllm/pull/50729", "createdAt": "2026-08-02T05:57:32Z", "updatedAt": "2026-08-13T17:22:35Z", "timestamp": "2026-08-13T17:22:35Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "bug", "ready" ], "author": "AndreasKaratzas", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:87b4dac94ffb19195517", "signalId": "github:vllm-project/vllm:pull_request:51338", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "text", "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51338", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RFC][Bug Fix][Spec Decode] Require explicit speculative methods", "text": "This PR removes speculative-method guessing and makes the method an explicit contract. ## Problem On current `main`, vLLM guesses the speculative method from model path/name substrings such as `eagle3`, `dflash`, and `dspark`. A DFlash or DSpark checkpoint at a neutral training path can therefore be treated as the generic autoregressive `draft_model`, while renaming or moving the same checkpoint changes serving behavior. For example: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15}' ``` Previously, this could silently select `draft_model`. In the original end-to-end Qwen3-8B + DFlash reproduction, the misdetected path produced acceptance length 1.02; adding explicit `\"method\": \"dflash\"` produced 3.88. Serving a specialized checkpoint through the wrong proposer is worse than requiring one extra field, so this PR now fails closed. ## Design - A method is required for every explicit speculative configuration. It can be supplied as `\"method\"` in `--speculative-config` or with `--spec-method`. - The explicit method selects the proposer and any structural loader normalization. vLLM does not replace or second-guess it from the model path, checkpoint architecture, structural `model_type`, or other metadata. - Checkpoint metadata is used only after method selection for implementation details, such as choosing the embedded DeepSeek-V4 DSpark loader for `method=\"dspark\"` and defaulting its proposal depth from `dspark_block_size`. - Known checkpoint schemas that explicitly declare their algorithm may populate the method before `SpeculativeConfig` is constructed. The speculators format uses `speculators_model_type`; PEagle is normalized to `method=\"eagle3\"` with parallel drafting. - `num_speculative_tokens` remains optional when the selected checkpoint format declares a proposal depth, such as speculators `speculative_tokens`, `n_predict`, or DSpark `dspark_block_size`. Otherwise the existing clear error remains. - DFlash and DSpark parallel-drafting behavior is derived from the explicit method. - Explicit shorthand values are folded into the config before checkpoint-declared defaults, so `--spec-method`, `--spec-model`, and `--spec-tokens` follow the same precedence rule as JSON fields. For an embedded DeepSeek-V4 checkpoint, the same checkpoint metadata is therefore handled according to the requested method: `draft_model` keeps the ordinary model loader, `mtp` uses `DeepSeekV4MTPModel`, and `dspark` uses `DSparkDraftModel`. The metadata does not choose among them. ## Breaking change Configurations that relied on implicit selection must add a method. Before: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` After: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"method\": \"dflash\", \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` For a generic autoregressive draft model, use `\"method\": \"draft_model\"`. Legacy EAGLE/EAGLE3, Medusa, MLP-speculator, n-gram, and custom-proposer configurations must likewise state their method. A speculators-format checkpoint used as the top-level model still works without an explicit `--speculative-config`, because its schema declares both the algorithm and proposal depth. ## Validation - End-to-end reproduction recorded before this design change: - misdetected `draft_model`: acceptance length 1.02 - explicit `dflash`: acceptance length 3.88 - This PR now rejects the missing-method configuration instead of silently serving it autoregressively. - Explicit method, schema default, loader selection, and callable override tests: 18 passed. - Full draft HF override module: 11 passed. - DeepSeek EAGLE loader compatibility regression: 1 passed. - Earlier focused config, DSpark, engine-argument, scheduler, and structured-output suites: 50 passed. - Full `tests/test_config.py` before the final refinement: 179 passed; 2 failed only because this environment cannot access gated `meta-llama/Meta-Llama-3-8B-Instruct` (HTTP 403). - All pre-commit hooks passed on every changed file, including Ruff, mypy, Markdown lint, SPDX, DCO sign-off, and configuration validation. - GPU model evaluation was not rerun for this follow-up; the serving change is fail-fast configuration and loader-selection behavior, and the earlier DFlash measurement above is retained as the motivating model result.
Commands run ```bash .venv/bin/python -m pytest -q \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/test_config.py \\ -k 'method_selects_draft_override or callable_overrides_reach_the_draft_config or arch_mapping_applies_before_callable_override or composed_override_is_picklable or explicit_method_selects_deepseek_v4_loader or speculative_config_requires_method or speculators_format' .venv/bin/python -m pytest -q \\ 'tests/config/test_model_arch_config.py::test_draft_model_arch_config[eagle618/deepseek-v3-random-eagle618/eagle-deepseek-v3-random-eagle-True]' .venv/bin/python -m pytest -q tests/config/test_speculative_draft_hf_overrides.py .venv/bin/pre-commit run --files ```
## Contribution notes - This is not a duplicate of #47490: that PR preserves auto-detection and validates explicit-method mismatches; this PR removes automatic method selection and fails when an explicit configuration omits the method. - This is not a duplicate of #44566: that PR only propagates token depth from a speculators-format draft; this PR changes method-selection semantics while preserving declared token defaults. - This is not a duplicate of #52165: that PR detects embedded DeepSeek-V4 DSpark metadata and uses it for automatic routing; this PR makes the explicit method authoritative and uses metadata only to configure the selected loader. - AI assistance was used. I reviewed every changed line and ran the validation listed above.", "url": "https://github.com/vllm-project/vllm/pull/51338", "createdAt": "2026-08-07T00:53:10Z", "updatedAt": "2026-08-13T17:20:56Z", "timestamp": "2026-08-13T17:20:56Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "documentation", "new-model", "structured-output", "speculative-decoding", "rust" ], "author": "WindChimeRan", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:01ddeee4bb5348fd77c9", "signalId": "github:vllm-project/vllm:pull_request:40337", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:40337", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring", "text": "## Summary Replaces the vanilla padded-bmm MaxSim path (introduced in #35330, re-enabled on GPU in #38620) with vendored flash-maxsim Triton kernels for ColBERT / ColPali document scoring. Addresses feature request #38282. ## Why this isn't a duplicate - Issue #38282 explicitly asks for this integration; no other open PR addresses it (searched `is:pr is:open flash maxsim`). - PR #38620 re-enabled GPU MaxSim infrastructure but explicitly deferred the kernel integration (\"only cleans up the API to make integration easier\") — this PR is the promised follow-up. - PR #35330 introduced the vanilla `compute_maxsim_score_batched` that this PR replaces. ## Approach Three scoring paths with automatic fallback: 1. **Zero-copy** (default for GPU, d ≥ 16, no chunked prefill, default pooling params): - Project the full `hidden_states` once via `TokenEmbeddingPoolerHead.project_batch()`. - Each doc request's pooler output is a `[L, d]` view into `projected_batch` — no copy, no extra allocation. - `flash_maxsim_rerank_direct` reads each doc at `(offset, length)` directly from the projected tensor. - The `[B, Lq, Ld]` score matrix is never materialized; the fused kernel tiles through SRAM. 2. **Flash-packed** (fallback when zerocopy disabled or pooling params use matryoshka / `use_activation=False`): - `torch.cat` all docs once, call `flash_maxsim_packed` with `cu_seqlens`. - No per-doc padding; kernel skips padding tokens via `cu_seqlens`. 3. **Vanilla** (fallback for CPU tensors, embedding dim < 16 (Triton `tl.dot` minimum), or `VLLM_FORCE_VANILLA_MAXSIM=1`): - Original sub-batched padded-bmm — unchanged. Triton autotune keys are bucketed (`{32, 64, 128, 256, 512, 1024, 2048, 4096}` for Lq/Ld, `next_pow2(d)` for embed dim). `LateInteractionRunner.__init__` pre-compiles 72 bucket combinations so no autotune fires on first request; failures during warmup are logged and cleanup is in `finally`. ## Files **New** (vendored Triton kernels — `vllm/v1/pool/flash_maxsim/`): - `flash_maxsim.py` — main forward kernel (hardware-specific autotune configs) - `flash_maxsim_rerank.py` — zero-copy rerank kernel (reads scattered offsets) - `flash_maxsim_varlen.py` — packed cu_seqlens kernel - `flash_maxsim_advanced.py` — q-reuse / split-K variants - `__init__.py` **Modified**: - `vllm/v1/pool/late_interaction.py` — `compute_maxsim_score_batched` dispatches to flash when available - `vllm/v1/worker/gpu/pool/late_interaction_runner.py` — kernel warmup, `_score_zerocopy`, `has_pending_docs` - `vllm/v1/worker/gpu_model_runner.py` — zerocopy branch in `_pool()` with fallback on chunked prefill / matryoshka / `use_activation=False` - `vllm/model_executor/layers/pooler/tokwise/heads.py` — `project_batch()` method (fp16 matmul, fp32 cast of small output) **Tests / demos**: - `tests/v1/worker/test_late_interaction_runner.py` — updated to d=32 (Triton requires K≥16) - `tests/v1/worker/bench_flash_maxsim.py` — detailed kernel benchmark - `tests/v1/worker/demo_flash_maxsim.py` — copy-paste kernel/memory comparison - `tests/v1/worker/demo_oom_resilience.py` — live server OOM demo ## Environment toggles - `VLLM_DISABLE_ZEROCOPY=1` — disable zero-copy, use flash-packed - `VLLM_FORCE_VANILLA_MAXSIM=1` — disable flash entirely, use vanilla bmm - `VLLM_FLASH_MAXSIM_WARMUP_D=` — extend warmup to an extra embedding dim ## Results (A100 80GB, ColBERT) ### Kernel-level on variable-length docs | Workload | Vanilla | Flash-Packed | Zero-Copy | |----------|---------|-------------|-----------| | B=1K, Ld=10-100 | 42.6ms | 0.58ms | **0.05ms** (~788×) | | B=5K, Ld=10-100 | 220.3ms | 2.16ms | **0.08ms** (~2,830×) | | B=10K, Ld=10-100 | 437.6ms | 4.04ms | **0.14ms** (~3,034×) | | B=10K, Ld=50-500 | 472.9ms | 4.22ms | **0.53ms** (~892×) | | ColPali B=1K, Ld≈1030 | 40.4ms | 0.72ms | **0.22ms** (~184×) | Most of vanilla's time (>90%) on varlen workloads is Python-side padding / fp32 cast / allocation, not bmm. The fused kernel eliminates that bookkeeping. ### E2E `/v1/score` (real embeddings, concurrent clients, best-of-3) | Docs/req | Conc | Flash TPS | Vanilla TPS | ΔTput | ΔP95 | |----------|------|-----------|-------------|-------|------| | 500 | 8 | 6.2/s | 5.2/s | **+19%** | **+19%** | | 500 | 16 | 6.0/s | 5.2/s | **+17%** | +7% | | 1000 | 4 | 3.0/s | 2.6/s | **+16%** | +13% | | 2000 | 4 | 1.6/s | 1.3/s | **+23%** | **+24%** | | 5000 | 1 | 0.5/s | 0.4/s | **+22%** | +18% | | 5000 | 2 | 0.6/s | 0.5/s | **+15%** | +19% | ### Correctness - Kernel vs fp32 bmm: `max_abs_err = 4e-6` - E2E 5K real docs: `max_abs_diff < 0.001`, zero pairs > 0.001 off, **top-3 rankings identical** Small tail-ranking noise at positions 4-5 reflects fp16 tensor-core nondeterminism on scores within 5e-4 of each other. ## Test plan - [x] `pytest tests/v1/worker/test_late_interaction_runner.py -v` → 4 passed - [x] `python tests/v1/worker/demo_flash_maxsim.py` → kernel speedups & memory savings displayed - [x] `python tests/v1/worker/bench_flash_maxsim.py` → full kernel benchmark - [x] 5K real ColBERT docs via `/v1/score` with `VLLM_FORCE_VANILLA_MAXSIM` vs flash → score parity verified - [x] `pre-commit run` ruff clean on all changed files - [ ] CI on `main` ## Notes for reviewers - The zerocopy path falls back to the normal pooler when `pooling_params.dimensions` is set (matryoshka) or `use_activation=False`, because `project_batch` normalizes before truncation — matryoshka would produce non-unit vectors. Falls back on chunked prefill for the same reason (partial cache). - `flash_maxsim_advanced.py` (q-reuse / split-K variants) is vendored but not dispatched from the default path. Kept for potential follow-ups. - Warmup takes ~160s cold (first launch on a host, no Triton disk cache) and ~15s warm; model load is ~80s either way. ## AI assistance Development of this PR used AI assistance (Claude). Every changed line was reviewed by the submitter; benchmarks were run end-to-end by the submitter against both paths on real A100 hardware with real ColBERT embeddings.", "url": "https://github.com/vllm-project/vllm/pull/40337", "createdAt": "2026-04-20T08:50:30Z", "updatedAt": "2026-08-13T17:17:58Z", "timestamp": "2026-08-13T17:17:58Z", "metrics": { "reactions": 3, "comments": 30 }, "labels": [ "frontend", "v1", "mrv2", "verified" ], "author": "roipony", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4b3a63fe6c518f238227", "signalId": "github:vllm-project/vllm:issue:52137", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52137", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: split local/external prefix-cache hits in `prompt_tokens_details`", "text": "### 🚀 The feature, motivation and pitch ## Background With distributed prefix caching (KV connectors such as Mooncake / lmcache), cache hits in a single prefill can come from two different sources: - **local**: hits on this instance's GPU prefix cache; - **external**: hits fetched from a remote KV store / another instance via the KV connector. The scheduler already tracks `num_local_cached_tokens` / `num_external_cached_tokens` separately in `PrefillStats`, but `OutputProcessor` only forwards their sum (`num_cached_tokens`), so the breakdown is lost before it reaches the OpenAI serving layer — users cannot tell, per request, where cache hits came from. Exposing this information per-request may also be handy for pricing purposes. ## Current behavior With `--enable-prompt-tokens-details` enabled, the response currently looks like: ```json \"prompt_tokens_details\": { \"cached_tokens\": 3627, \"created_cache_tokens\": 0, \"multimodal_tokens\": null } ``` ## Expected behavior Add two optional keys, `local_cached_tokens` / `external_cached_tokens`, alongside the existing fields: ```json \"prompt_tokens_details\": { \"cached_tokens\": 2320, \"created_cache_tokens\": 0, \"multimodal_tokens\": null, \"local_cached_tokens\": 1216, \"external_cached_tokens\": 1104 } ``` Invariant: `cached_tokens == local_cached_tokens + external_cached_tokens`. ## Requirements - Strictly additive change: existing fields (`cached_tokens` / `created_cache_tokens` / `multimodal_tokens`) must be preserved untouched; - New fields default to `None` and must not affect existing output when the flag is off or the values carry no meaning; - Cover both streaming and non-streaming paths for chat and completion endpoints, and also benefit offline `LLM()` users. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52137", "createdAt": "2026-08-13T09:18:49Z", "updatedAt": "2026-08-13T17:17:15Z", "timestamp": "2026-08-13T17:17:15Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "feature request" ], "author": "fataswellassad", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9b71adc4f1de0197b2b2", "signalId": "github:vllm-project/vllm:issue:37736", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:37736", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[CI Failure]: Gemma3 OOMs with transformers backend", "text": "### Test group mi250_1: Multi-Modal Models (Standard) 2: qwen3 + gemma ### Describe the failing test This is not exactly a test failure, but it has been recommended to investigate further the OOMing event of Gemma3, which is a 4B model. The intuition here is that the fake tensor that is used for profiling is large enough that exceeds the 64 GB of MI250 GPUs. However, it has been suggested that this is still weird. ### 📝 History of failing test https://github.com/vllm-project/vllm/pull/37610#issuecomment-4102286515 ### CC List. @DarkLight1337 (for transparency)", "url": "https://github.com/vllm-project/vllm/issues/37736", "createdAt": "2026-03-21T05:32:29Z", "updatedAt": "2026-08-13T17:16:19Z", "timestamp": "2026-08-13T17:16:19Z", "metrics": { "reactions": 0, "comments": 8 }, "labels": [ "rocm", "unstale", "ci-failure" ], "author": "AndreasKaratzas", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:92e012e77356bdaf345c", "signalId": "github:vllm-project/vllm:pull_request:51351", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51351", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Declare EAGLE cache-peek capability for KV cache managers", "text": "## Summary - declare whether each KV cache manager can safely peek past an EAGLE/MTP candidate boundary - resolve EAGLE cache-peek groups through the shared KV cache spec registry - exclude recurrent and otherwise unsupported cache managers from the fallback - fail closed when an unsupported group is explicitly annotated - apply the shared capability in the core, Mooncake store, and offloading paths Fixes #50630. ## Root cause When no KV cache group was explicitly annotated for EAGLE, several paths conservatively treated every group as an EAGLE group. This is safe only for managers that can drop the extra matched cache unit. Recurrent Mamba state cannot be rewound this way, so each caller had to exclude Mamba manually. The new manager capability makes this invariant explicit and defaults to unsupported. Full-attention and sliding-window managers opt in. ## Duplicate-work check Before opening this PR, issue #50630 and its comments were checked. Open PR searches by issue number and by the EAGLE/cache-peek/Mamba area found no PR addressing the same change. ## Tests Linux, Python 3.10.12: - Focused regression tests: `10 passed, 14 warnings in 2.71s` - Related test files: `109 passed, 14 warnings in 21.33s` - `ruff check`: passed - `ruff format --check`: 9 files already formatted - `git diff --check`: passed The warnings are existing PyTorch `torch.jit.script_method` deprecation warnings. No model evaluation or GPU benchmark was run because this is a control-plane KV-cache capability and boundary-selection change covered by unit tests. ## AI assistance AI assistance was used for test design, and PR drafting. The human submitter reviewed the changes and ran the reported tests.", "url": "https://github.com/vllm-project/vllm/pull/51351", "createdAt": "2026-08-07T03:36:18Z", "updatedAt": "2026-08-13T17:15:26Z", "timestamp": "2026-08-13T17:15:26Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "needs-rebase", "kv-connector" ], "author": "xinyuran", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:7ad8a4bb922b2d1c0716", "signalId": "github:vllm-project/vllm:pull_request:52193", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52193", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "diagnostic patch created covering all workspace creation", "text": "## Purpose Checks #52023 ## Test Plan This is still in WIP, it is for logs from reporter. ## Test Result WIP ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52193", "createdAt": "2026-08-13T17:14:42Z", "updatedAt": "2026-08-13T17:15:17Z", "timestamp": "2026-08-13T17:15:17Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia" ], "author": "khushali9", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:c91f14d85e959b575ea9", "signalId": "github:vllm-project/vllm:pull_request:50883", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50883", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][KV Offload] Scale UniformTypeKVCacheSpecs groups by DCP", "text": "## Purpose #49964 gated the DCP scaling of a group's token span on `isinstance(group.kv_cache_spec, AttentionSpec)` so that Mamba groups stay unscaled. `UniformTypeKVCacheSpecs` is a container rather than an `AttentionSpec` subclass, so DSA models — whose indexer and MLA layers merge into a single uniform group — lost the DCP factor too: - `build_offloading_config` reports `tokens_per_block = block_size` (64), - `resolve_kv_cache_block_sizes` still returns `block_size * dcp` (128) for a single group, so enabling CPU offload under DCP dies at startup: ``` AssertionError: tokens_per_block=64 not divisible by tokens_per_hash=128. Hybrid models (e.g. Mamba+Attention) need --enable-prefix-caching to align block sizes. ``` Reported by @Leoyzen on GLM-5.2, TP8/DCP2, `fp8_ds_mla`, MTP, `OffloadingConnector`: https://github.com/vllm-project/vllm/pull/46514#issuecomment-5166406963 The scaling is now keyed off a helper that unwraps the container, so Mamba groups stay unscaled and every attention group keeps its DCP span. Same call site is fixed in `resolve_kv_cache_block_sizes` for multi-group configs. Related: #50823 fixes the other half of the same container-not-unwrapped family (block table width). ## Test Three tests, one per call site plus the hybrid case, next to the Mamba tests from #49964: - `offloading_connector/test_config.py::test_dcp_scales_uniform_type_attention_group_blocks` - `offloading_connector/test_config.py::test_dcp_scales_uniform_type_group_alongside_mamba` - `core/test_kv_cache_utils.py::test_resolve_block_sizes_scales_uniform_type_group_by_dcp` Verified fail→pass in a container built from the current nightly (`0.26.1rc1.dev255+g5e35a6f4f`); each test fails on unpatched `main` with the reported assert (`tokens_per_block=16 not divisible by tokens_per_hash=32`) or a wrong span, and the two suites pass with the fix: ``` python3 -m pytest tests/v1/core/test_kv_cache_utils.py \\ tests/v1/kv_connector/unit/offloading_connector/test_config.py -q # 115 passed ``` Config-level change, no effect on model output; no eval run. AI assistance (Claude) was used for this work; all changes reviewed.", "url": "https://github.com/vllm-project/vllm/pull/50883", "createdAt": "2026-08-03T15:41:58Z", "updatedAt": "2026-08-13T17:13:23Z", "timestamp": "2026-08-13T17:13:23Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "needs-rebase", "kv-connector" ], "author": "drakosha", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:3621fe7892af09bb501f", "signalId": "github:vllm-project/vllm:pull_request:51551", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51551", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][MLA] Add guarded gfx942 FP8 context prefill", "text": "## Summary Adds an opt-in gfx942 FP8 context-prefill path for Kimi-K3 MLA using the D192/V128 kernel from ROCm/aiter#4645. It also supports checkpoint-specific static per-head scales through a versioned external artifact, removing dynamic amax reductions while preserving the existing attention boundary. ## Background The existing AITER FP8 ASM kernels do not cover Kimi-K3's asymmetric D192 Q/K and D128 V shape on gfx942. ROCm/aiter#4645 adds a packed variable-length Gluon implementation and static per-head quantization for this shape. ## What this PR does - Enables the AITER FP8 path only with `VLLM_ROCM_KIMI_K3_FP8_PREFILL=1`. - Restricts dispatch to BF16 Kimi-K3 MLA on gfx942 with 12 heads, D192 Q/K, and D128 V. - Quantizes Q once per context pass and reuses its per-head scale across context segments. - Collects layer-owned Q/K/V maxima after warmup in eager calibration mode. - Writes atomic TP-rank shards and merges them into a versioned `safetensors` artifact. - Validates model revision, TP/PP topology, cache mode, dimensions, dtype, layer coverage, and finite positive descales before compilation. - Uses a one-kernel static quantizer when an artifact is configured. Missing or incompatible artifacts fail instead of silently falling back. - Keeps the BF16 path for unsupported shapes and short contexts. ## Validation Model: `moonshotai/Kimi-K3`, TP8, MI325X. - Full five-shot gsm8k: 0.9606 exact match with 0.0054 stderr. - Needle retrieval: 9 of 9 passed at 128K, 512K, and 1,047,552 tokens with needle depths 10%, 50%, and 90%. - Deterministic completion sanity checks: 3 of 3 passed. - Matched 1,047,552-token dynamic TTFT: 434.65 s, 404.80 s, and 404.83 s. - Matched static TTFT: 431.38 s, 402.44 s, and 402.44 s. - Mean static TTFT improved by 0.64%. The two warm repeats improved by 0.59%. - AITER synthetic gfx942 tests: 9 passed. - vLLM artifact and configuration tests: 8 passed. ## Notes / scope This PR depends on ROCm/aiter#4645. The feature remains disabled by default. Calibration and validation prompt sets are separate, and calibration is eager-only. ## Test plan - [x] Full gsm8k, five-shot, 1319 samples - [x] 128K, 512K, and one-million-token needle retrieval - [x] Three matched one-million-token dynamic and static repeats - [x] API sanity checks - [x] Artifact compatibility and merge tests", "url": "https://github.com/vllm-project/vllm/pull/51551", "createdAt": "2026-08-09T10:16:12Z", "updatedAt": "2026-08-13T17:10:51Z", "timestamp": "2026-08-13T17:10:51Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "rocm", "needs-rebase", "nvidia", "kimi", "k3" ], "author": "maeehart", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:878d4ee9d340f371904f", "signalId": "github:vllm-project/vllm:issue:41623", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:41623", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: Decode Context Parallelism (`--decode-context-parallel-size`) output drift and gibberish in v0.21.0 and latest nightly", "text": "### Your current environment
The output of python collect_env.py ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch : N/A XPU used to build PyTorch : N/A ============================== Python Environment ============================== Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.15.134+release+2.10.0r8-amd64-x86_64-with-glibc2.35 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 CUDA_MODULE_LOADING set to : GPU models and configuration : GPU 0: NVIDIA A100-SXM4-80GB GPU 1: NVIDIA A100-SXM4-80GB GPU 2: NVIDIA A100-SXM4-80GB GPU 3: NVIDIA A100-SXM4-80GB GPU 4: NVIDIA A100-SXM4-80GB GPU 5: NVIDIA A100-SXM4-80GB GPU 6: NVIDIA A100-SXM4-80GB GPU 7: NVIDIA A100-SXM4-80GB Nvidia driver version : 590.48.01 cuDNN version : Could not collect HIP runtime version : N/A MIOpen runtime version : N/A Is XNNPACK available : True ============================== CPU Info ============================== Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 43 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: AuthenticAMD Model name: AMD EPYC 7742 64-Core Processor CPU family: 23 Model: 49 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 2 Stepping: 0 Frequency boost: enabled CPU max MHz: 2250.0000 CPU min MHz: 1500.0000 BogoMIPS: 4500.26 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es Virtualization: AMD-V L1d cache: 4 MiB (128 instances) L1i cache: 4 MiB (128 instances) L2 cache: 64 MiB (128 instances) L3 cache: 512 MiB (32 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-63 NUMA node1 CPU(s): 64-127 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled Vulnerability Spec rstack overflow: Mitigation; SMT disabled Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.0.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.19.0.56 [pip3] nvidia-cudnn-frontend==1.18.0 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-cufile==1.15.1.6 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.0 [pip3] nvidia-cutlass-dsl==4.4.2 [pip3] nvidia-cutlass-dsl-libs-base==4.4.2 [pip3] nvidia-ml-py==13.595.45 [pip3] nvidia-nccl-cu13==2.28.9 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvshmem-cu13==3.4.5 [pip3] nvidia-nvtx==13.0.85 [pip3] pyzmq==27.1.0 [pip3] torch==2.11.0+cu130 [pip3] torch_c_dlpack_ext==0.1.5 [pip3] torchaudio==2.11.0+cu130 [pip3] torchvision==0.26.0+cu130 [pip3] transformers==5.7.0 [pip3] triton==3.6.0 [conda] Could not collect ============================== vLLM Info ============================== ROCM Version : Could not collect vLLM Version : 0.20.1rc1.dev91+ga749a33d8 (git sha: a749a33d8) vLLM Build Flags: CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS SYS 0-63 0 N/A GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 NODE NODE 64-127 1 N/A GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 NODE NODE 64-127 1 N/A GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 NODE NODE 64-127 1 N/A GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X NODE NODE 64-127 1 N/A NIC0 SYS SYS SYS SYS NODE NODE NODE NODE X PIX NIC1 SYS SYS SYS SYS NODE NODE NODE NODE PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 ============================== Environment Variables ============================== NVIDIA_VISIBLE_DEVICES=GPU-f810eb6d-56e1-1937-6319-dc03bb5381c2,GPU-d6e4c7e1-a414-78e0-4c9e-7d3d1a6f97a5,GPU-972f9bbe-6f43-b12f-2ad3-ca8c2d44328a,GPU-238b6428-8034-d89d-ff91-d9b7d0dc6cbd,GPU-5c2d9c0d-7b82-9b66-9a0b-30238270fac3,GPU-bad1cbfa-554c-22c7-ccb8-e24d4a26883e,GPU-4e92be98-9b5d-af97-0e44-1a50cc7971ea,GPU-b22f4dec-4abb-6dcb-c1aa-52a2a705758e NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576 TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=13.0.2 VLLM_ENABLE_CUDA_COMPATIBILITY=0 VLLM_ENGINE_READY_TIMEOUT_S=86400 LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 PYTORCH_NVML_BASED_CUDA_CHECK=1 TORCHINDUCTOR_COMPILE_THREADS=1 TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root ```
### 🐛 Describe the bug GPU: 8x A100 `vllm serve moonshotai/Kimi-K2.6 --port 5000 --host 0.0.0.0 --download-dir /workspace/.cache/huggingface/hub --api-server-count 8 --tensor-parallel-size 8 --decode-context-parallel-size 8 --trust-remote-code --enable-chunked-prefill --enable-prefix-caching --max-num-seqs 32 --gpu-memory-utilization 0.95 --max-model-len 262144 --enable-auto-tool-choice --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --mm-processor-cache-gb 8 --mm-processor-cache-type shm` Decode Context Parallelism (DCP, `--decode-context-parallel-size`) produces unrelated gibberish output in latest nightly. This is a regression. Happens somewhere between `a749a33d8d05acdd3ab346bd3f0c6b5c9c80474f` (works well) `01d4d1ad375dc5854779c593eee093bcebb0cada` (gibberish output). Only when `--decode-context-parallel-size` is set, Kimi-K2.6 leads to completely unrelated gibberish. Problematic commit is between the above commits. ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/41623", "createdAt": "2026-05-04T10:57:26Z", "updatedAt": "2026-08-13T17:08:51Z", "timestamp": "2026-08-13T17:08:51Z", "metrics": { "reactions": 0, "comments": 20 }, "labels": [ "bug" ], "author": "ehfd", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:16f779e7b0c1bbe7c2d8", "signalId": "github:vllm-project/vllm:pull_request:51704", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51704", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec", "text": "## Purpose Part of the KV-cache layout standardization series (RFC #42082). **Stacked on #51612** — the diff shown includes it until that lands and this retargets `main`. Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants in MLA spec overrides. This PR makes specs plain data and moves each packed format to the backend/component that owns it: - `AttentionSpec` gains two optional packing fields `num_head_slots` (H in standard layout) and`state_content_bytes` (C in standard layout). Page sizes derive uniformly as `num_heads × storage_block_size × content`. > NOTE: i dont love the num_head_slots name but this should hopefully be temporary until we can [fully refactor the backends](https://github.com/vllm-project/vllm/issues/42449); i.e. separate allocation from connector and kv-cache manager concerns - `AttentionBackend.customize_spec(spec)` is a temporary hook until we can move to having the backend return the spec instead of the layer (see: https://github.com/vllm-project/vllm/issues/42449) Prepares the ground for the layout-standardization PR ([[6/N]](https://github.com/vllm-project/vllm/pull/51718)), where these fields describe the standardized `[B, H, N, C]` page and the AITER backends publish separate K/V head groups through the same hook. --- AI-assisted (Claude); reviewed by submitter", "url": "https://github.com/vllm-project/vllm/pull/51704", "createdAt": "2026-08-10T16:20:40Z", "updatedAt": "2026-08-13T17:08:18Z", "timestamp": "2026-08-13T17:08:18Z", "metrics": { "reactions": 2, "comments": 23 }, "labels": [ "rocm", "ready", "nvidia", "ready-run-all-tests", "mrv2", "kimi", "k3" ], "author": "LucasWilkinson", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:30784b4be6caf2de74d5", "signalId": "github:vllm-project/vllm:pull_request:42792", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:42792", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[WIP][Model Runner V2] support spec decode + mamba align prefix caching", "text": "## Accuracy Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' \\ --default-chat-template-kwargs '{\"enable_thinking\": false}' ``` **Results** ``` | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |---------|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k_cot| 3|flexible-extract| 8|exact_match|↑ |0.7998|± |0.0110| | | |strict-match | 8|exact_match|↑ |0.7968|± |0.0111| ``` ## Performance Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\ -tp 2 -dp 1 \\ --enable-prefix-caching \\ --mamba-cache-mode align \\ --max-num-seqs 64 \\ --attention-config '{\"use_trtllm_attention\": 0}' \\ --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' ``` **Results** | Metric | #42406 | This PR | Delta | |--------|-----------|-----------|-------| | **General** | | | | | Successful requests | 512 | 512 | — | | Failed requests | 0 | 0 | — | | Max request concurrency | 16 | 16 | — | | Benchmark duration (s) | 119.40 | 113.97 | -4.5% | | Total input tokens | 2,167,806 | 2,271,087 | +4.8% | | Total generated tokens | 262,144 | 262,144 | — | | Request throughput (req/s) | 4.29 | 4.49 | +4.7% | | Output token throughput (tok/s) | 2,195.48 | 2,300.20 | +4.8% | | Peak output token throughput (tok/s) | 1,553.00 | 1,040.00 | -33.0% | | Peak concurrent requests | 28.00 | 28.00 | — | | Total token throughput (tok/s) | 20,351.05 | 22,227.97 | +9.2% | | **Time to First Token** | | | | | Mean TTFT (ms) | 282.49 | 105.66 | -62.6% | | Median TTFT (ms) | 145.19 | 84.13 | -42.1% | | P99 TTFT (ms) | 3,800.32 | 995.73 | -73.8% | | **Time per Output Token** | | | | | Mean TPOT (ms) | 6.64 | 6.68 | +0.6% | | Median TPOT (ms) | 6.59 | 6.61 | +0.3% | | P99 TPOT (ms) | 10.03 | 7.95 | -20.7% | | **Inter-token Latency** | | | | | Mean ITL (ms) | 16.35 | 17.04 | +4.2% | | Median ITL (ms) | 10.58 | 15.83 | +49.6% | | P99 ITL (ms) | 105.52 | 38.90 | -63.1% | | **End-to-end Latency** | | | | | Mean E2EL (ms) | 3,677.74 | 3,521.25 | -4.3% | | Median E2EL (ms) | 3,528.08 | 3,468.17 | -1.7% | | P99 E2EL (ms) | 8,904.19 | 4,462.88 | -49.9% | | **Speculative Decoding** | | | | | Acceptance rate (%) | 73.88 | 78.34 | +4.46pp | | Acceptance length | 2.48 | 2.57 | +3.6% | | Drafts | 105,785 | 102,119 | -3.5% | | Draft tokens | 211,570 | 204,238 | -3.5% | | Accepted tokens | 156,310 | 160,003 | +2.4% | | Position 0 acceptance (%) | 83.86 | 85.68 | +1.82pp | | Position 1 acceptance (%) | 63.90 | 71.00 | +7.10pp |", "url": "https://github.com/vllm-project/vllm/pull/42792", "createdAt": "2026-05-15T22:55:32Z", "updatedAt": "2026-08-13T17:08:05Z", "timestamp": "2026-08-13T17:08:05Z", "metrics": { "reactions": 1, "comments": 2 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "TheEpicDolphin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e925eb049033815b4409", "signalId": "github:vllm-project/vllm:pull_request:52175", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52175", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix Cohere ChatV2 citation and tool handling issues", "text": "## Purpose Fix issues ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [X] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52175", "createdAt": "2026-08-13T14:21:42Z", "updatedAt": "2026-08-13T17:07:51Z", "timestamp": "2026-08-13T17:07:51Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "frontend", "needs-rebase", "tool-calling", "cohere" ], "author": "andrewbcohere", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:9f7fa0b3d2b85ab6914e", "signalId": "github:vllm-project/vllm:issue:42932", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:issue:42932", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: vLLM wheel version mismatch", "text": "### Your current environment vLLM wheel that is getting installed is different from that of the branch used. Example Docker is generated using the vllm branch - 0.19.0 , https://github.com/vllm-project/vllm/tree/releases/v0.19.0 But while building the wheel/docker, it generates the wheel which has version 0.19.1 pip3 list -l | grep vllm vllm 0.19.1.dev0+g2a69949bd.d20260417.rocm713 Same applies for vLLM branch - 0.19.1 as well vLLM branch used --- https://github.com/vllm-project/vllm/tree/releases/v0.19.1 Wheel that is generated has the version as pip3 list -l | grep vllm vllm 0.19.2.dev3+g24efb8904.d20260514.rocm713 ### 🐛 Describe the bug There are no crashes or faults, This mismatch should not happen ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/42932", "createdAt": "2026-05-18T06:18:15Z", "updatedAt": "2026-08-13T17:07:51Z", "timestamp": "2026-08-13T17:07:51Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "bug" ], "author": "jvadakat", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:43a336d796a8d74e1a44", "signalId": "github:vllm-project/vllm:pull_request:52136", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52136", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Add `pydocstyle` to the `ruff` rules", "text": "Enables the `D` ruleset in ruff and fixes the resulting violations across the codebase. ### Config `D100`–`D107` are ignored, so this does **not** require a docstring on anything that lacks one today. Six more rules are ignored because their remaining violations were either impossible to autofix or actively wrong to autofix: | Rule | Why skipped | |---|---| | `D205`, `D400`, `D415` | All treat a summary sentence that merely *wraps* as summary-plus-body. The autofix appends periods to list items. | | `D401`, `D404` | Flag idiomatic noun-phrase docstrings on properties and attributes. | | `D301` | Churn-only (`r\"\"\"` prefixes). | Plus `ignore-var-parameters = true` (no `*args`/`**kwargs` descriptions required) and a `tests/**` ignore for `D417`, since test params are mostly pytest fixtures. ### Changes The bulk is autofix output for the mechanical rules (`D209`, `D212`, `D200`, `D202`, `D413`, and friends). On top of that, by hand: - **~290 `Args:` entries** written across 88 files for `D417`, mostly in `fused_moe`, attention backends, and multimodal models. Several were format bugs rather than missing docs: numpy-style `Parameters`/`----------` blocks that ruff's Google parser can't read at all. - **807 trailing periods** for `D400`/`D415`, applied only where `D205` doesn't co-fire, so no list items were punctuated. - **37 `E501` regressions** fixed. These were introduced by the `D212` autofix collapsing wrapped summaries onto line 1 and pushing them past 88 chars. - A handful of one-offs: `D210`, `D402`, `D418`, `D419`. Incidental fixes picked up along the way: typos (`videoand`, `output_dytpe`, `the m odel`) and stale docs naming parameters that no longer exist.", "url": "https://github.com/vllm-project/vllm/pull/52136", "createdAt": "2026-08-13T09:16:21Z", "updatedAt": "2026-08-13T17:07:32Z", "timestamp": "2026-08-13T17:07:32Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "performance", "new-model", "rocm", "structured-output", "frontend", "tpu", "intel-gpu", "speculative-decoding", "ci/build", "multi-modality", "tool-calling", "llama", "qwen", "deepseek", "cpu", "gpt-oss", "kv-connector", "nvidia", "quantization", "mrv2", "mistral", "build-docs", "kimi", "k3", "cohere" ], "author": "hmellor", "state": "open", "assignees": [ "sangstar" ], "change": "updated" } }, { "id": "event:438108f2f59445c2fb2f", "signalId": "github:vllm-project/vllm:pull_request:51415", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51415", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Manual act quant fusion llama", "text": "Starts the `ActivationQuantFusionPass` manual-fusion migration (RFC #43224, specific tracker https://github.com/vllm-project/vllm/issues/43501) on the producer side of the `QuantizedActivation` contract (#44260). Adds `maybe_fused_act_quant`: given an activation and the linear it feeds, it emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kernel when the linear advertises a consumable `input_quant_key`, and falls back to the plain activation otherwise. Llama's MLP forward calls it for `down_proj` — the model-code change is one line. When manual fusion fires the `silu_and_mul` pattern is already consumed, so the compiler `ActivationQuantFusionPass` finds nothing and the two never double-fuse. Only `(SiluAndMul, kFp8StaticTensorSym)` is registered for now; other schemes/activations are one table row each. Not a duplicate: builds on the landed contract (#44260) and is the activation-quant counterpart to the AR+RMSNorm manual fusion in #45855; no open PR covers the producer side. ## Test Smoke tested on `RedHatAI/Llama-3.2-1B-Instruct-FP8` (default compiled + CUDA-graph path): all 16 `mlp.down_proj` layers take the fused branch (`input_quant_key == kFp8StaticTensorSym`) and generations are coherent — the `QuantizedActivation` flows through the compiled `down_proj.forward` into `apply_weights` with no graph break. TP2 coherence and an A/B vs. the compiler pass are follow-ups. AI assistance (Claude) was used; all changed lines are human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/51415", "createdAt": "2026-08-07T15:44:04Z", "updatedAt": "2026-08-13T17:07:15Z", "timestamp": "2026-08-13T17:07:15Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "ready", "llama", "quantization" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:f89200e48c4204af8861", "signalId": "github:vllm-project/vllm:pull_request:52139", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52139", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][ROCm][CI] Give the AITER MLA decode metadata stub its MLA dims", "text": "# Purpose `tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py::test_persistent_decode_metadata_matches_fp8_golden` fails on main with `AttributeError: 'types.SimpleNamespace' object has no attribute 'q_lora_rank'`. Two jobs report it, the dedicated AITER MLA job and the sharded `kernels/attention` job, but it is the same test. \"[Model] Add native Dots3 NOTE multimodal support\" (#51255), changed `MLACommonMetadataBuilder.__init__` to read the MLA latent dimensions off the layer in `static_forward_context` instead of calling `get_mla_dims(self.model_config)`, so that hybrid MLA models can use different dimensions per KV cache group. The builder now also sizes its chunked-prefill workspace from `kv_lora_rank + qk_rope_head_dim` rather than from `model_config.get_head_size()`. This test stubs that layer with a `SimpleNamespace` holding only `prefill_backend`, so the builder raises on the first dimension it asks for. #51255 updated the same stub in the sibling test `test_rocm_aiter_mla_causal_verify_mask.py` and in the MLA backend tests, but missed this file, which uses the identical pattern. # Test Plan The change adds the five MLA dimensions to the stub. They are taken from `get_mla_dims(vllm_config.model_config)`, which is what the builder itself used before #51255, so the stub cannot drift from the `deepseek-ai/DeepSeek-R1` configuration the rest of the test is built around. The sibling test hardcodes the same values as module constants because it also builds tensors from them; this one does not need them anywhere else. On gfx950 (MI355X), with AITER: ``` pytest -v -s tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py \\ tests/kernels/attention/test_rocm_aiter_mla_causal_verify_mask.py ``` The repository was also searched for any other test stubbing a layer the same way; these two are the only ones. # Test Result Before the change the decode metadata test fails with the `AttributeError` above. After it, both tests pass. The assertion the test exists for, that the builder's persistent decode metadata matches the golden recomputed with explicit fp8 dtypes, is reached and holds; nothing about the coverage changes. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52139", "createdAt": "2026-08-13T09:47:40Z", "updatedAt": "2026-08-13T17:06:16Z", "timestamp": "2026-08-13T17:06:16Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "rocm" ], "author": "stefankoncarevic", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8e6fb2dd28d33648831b", "signalId": "github:vllm-project/vllm:pull_request:51855", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51855", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[K3] support recoverssm for K3", "text": " ## Purpose This PR adds ReplaySSM speculative decoding for the NVIDIA Kimi-K3 KDA path on Model Runner V2. It is intended for Kimi-K3 serving with DSpark and supports `mamba_cache_mode=align` prefix caching. Without ReplaySSM, KDA speculative decoding materializes a full recurrent state for every speculative position. ReplaySSM keeps one checkpoint and compact per-token records, then reconstructs the state at the accepted position after every verify. Under the tested TP8 Kimi-K3 configuration, this increases the effective cache capacity by 10.97% at the same 46.48 GiB cache-memory budget. ## Design ### Kimi-K3-specific V2 integration The implementation is contained in the Kimi-K3 KDA metadata, layer, and ReplaySSM operator paths. The metadata builder gives speculative rows one KDA checkpoint slot and creates a model-specific post-sampling committer. The V1 integration from the initial implementation was removed because Kimi-K3 does not run on Model Runner V1. ReplaySSM is selected through `--use-replayssm`. The regular KDA prefill and non-speculative decode paths are unchanged. ### Verify records FP32 corrections KDA verify reads the checkpoint without modifying it and computes the normal speculative outputs. For each speculative token it records: - the correction vector `u` in FP32; - the key and raw gate in the activation dtype. Recording `u` removes `v` and `beta` from the commit path and avoids recomputing the state-dependent correction. FP32 is used for `u` to preserve reconstruction accuracy; keys and gates remain BF16 in the evaluated configuration. ### Direct accepted-position reconstruction After verification determines the accepted length, one Triton kernel reconstructs the required checkpoint directly. For the KDA recurrence: ```text S_t = S_{t-1} * exp(g_t) + u_t outer k_t ``` The accepted state is: ```text S_p = S_0 * exp(sum(g_1 ... g_p)) + sum_j u_j outer (k_j * exp(sum(g_{j+1} ... g_p))) ``` The commit kernel evaluates this expression in reverse token order. It loads the original state once and does not recursively update or write intermediate states. The loop bound is the runtime accepted length rather than the complete eight-token DSpark activation window. In align mode, the same launch also reconstructs the next prefix-cache boundary when the accepted range crosses it. Convolution history is compacted to the accepted position and to the boundary using the same commit plan. ### Cache layout Each KDA page contains: ```text (convolution state, KDA checkpoint, FP32 correction records, key/gate records) ``` Replay records are transient and are not included in prefix-cache state copies. The checkpoint and convolution state remain the prefix-cache boundary state. ## Scope - NVIDIA Kimi-K3 KDA - Model Runner V2 - speculative decoding, validated with Kimi-K3-DSpark and seven draft tokens - `mamba_cache_mode=align` prefix caching This PR does not add a generic ReplaySSM interface to shared attention or model state classes, and it does not add Mamba2 support. ## Test Plan and Results ### Unit tests ```bash .venv/bin/python -m pytest tests/models/kimi_k3/test_kda.py -v ``` Result: `69 passed`. The 18 ReplaySSM cases cover: - the actual eight-token DSpark window; - partial and full acceptance lengths; - direct final-state and align-boundary reconstruction; - both convolution-state layouts; - request-index remapping; - gates with and without a lower bound; - cache shape and dtype derivation. ```bash .venv/bin/pre-commit run --files vllm/model_executor/layers/mamba/mamba_utils.py vllm/models/kimi_k3/nvidia/kda.py vllm/models/kimi_k3/nvidia/ops/replayssm.py tests/models/kimi_k3/test_kda.py ``` Result: all hooks passed. ```bash .venv/bin/pre-commit run mypy-3.12 --all-files --hook-stage manual ``` Result: passed for Python 3.12. ### End-to-end accuracy Controlled setup: - Kimi-K3 target and Kimi-K3-DSpark draft model, TP8 on two GB300 nodes; - seven greedy draft tokens and block rejection sampling; - align-mode Mamba cache, prefix caching, FP8 MLA cache, FlashInfer MLA, and fastsafetensors; - identical datasets, prompts, scorers, and serving parameters for ReplaySSM ON and OFF. | GSM8K, 1,319 samples | ReplaySSM ON | OFF | ON - OFF | |---|---:|---:|---:| | Flexible extract | 96.3609% | 96.1334% | +0.2274 pp | | Strict match | 96.0576% | 95.9060% | +0.1516 pp | The paired 95% confidence intervals were `[-0.4549, +0.9098]` pp for flexible extract and `[-0.6065, +0.9098]` pp for strict match. Both runs completed all 1,319 requests with zero request errors. Independently initialized greedy serving runs are not bitwise deterministic, so the paired score comparison is used instead of exact generated-text equality. | MRCR, 40 long prompts run cold and warm | ReplaySSM ON | OFF | |---|---:|---:| | Cold score | 0.74879 | 0.66791 | | Warm score | 0.74879 | 0.66791 | | Cold/warm exact-output consistency | 100% | 97.5% | | Warm prefix-cache token hit rate | 89.13% | 90.50% | The MRCR prompts ranged from 16,757 to 33,772 tokens and contained 991,216 prompt tokens per pass. ReplaySSM completed both passes without request errors. Total DSpark acceptance was 53.3514% with ReplaySSM and 53.4062% without it, a difference of -0.0548 percentage points. GSM8K-only acceptance was 50.0899% with ReplaySSM and 50.0336% without it. ### Memory capacity Both variants used a fixed 46.48 GiB cache-memory budget. | Metric | ReplaySSM ON | OFF | Difference | |---|---:|---:|---:| | Effective cache capacity | 2,856,170 tokens | 2,573,777 tokens | +10.97% | | Max 262k-request concurrency | 10.895x | 9.818x | +10.97% | ### Nsight Systems profile The profile used batch 32, 128 forced output tokens, and 80 captured decode steps per GPU on the same two-node TP8 setup. The aggregate results below include eight initial small-batch steps. | KDA work per decode step | Initial recurrent ReplaySSM commit | This PR | |---|---:|---:| | Verify | 3.115 ms | 3.540 ms | | Commit | 4.793 ms | 1.464 ms | | Commit auxiliaries | 0.081 ms | 0.081 ms | | Total | 7.989 ms | 5.085 ms | The direct reconstruction makes the commit kernel 3.27x faster and reduces total ReplaySSM KDA time by 36.34%. The FP32 correction write increases verify time by 13.66%, but commit time falls by 69.45%. Native non-ReplaySSM KDA took 2.772 ms per step in the corresponding OFF trace. The 72 steady batch-32 steps each verified 256 tokens. Their CUDA-graph model-forward GPU wall time was 65.208 ms with ReplaySSM and 64.873 ms without it, a 0.335 ms or 0.52% difference. Kernel residence exceeds wall time because kernels on concurrent streams overlap. | ReplaySSM ON steady forward kernel residence | Time per step | Share | |---|---:|---:| | DeepGEMM and nvJet GEMM | 38.760 ms | 56.5% | | MoE routing and packing | 8.121 ms | 11.8% | | SiTU activation | 7.682 ms | 11.2% | | TP collectives | 6.136 ms | 8.9% | | ReplaySSM verify | 3.791 ms | 5.5% | | KDA convolution and normalization | 0.938 ms | 1.4% | | MLA attention kernels | 0.519 ms | 0.8% | | Other kernels | 2.658 ms | 3.9% | | Total kernel residence | 68.603 ms | 100% | ReplaySSM verify runs inside model forward and replaces the native KDA recurrent kernel. Its steady kernel residence was 3.791 ms versus 3.009 ms for native KDA, an increase of 0.782 ms. The smaller 0.335 ms forward wall-time difference reflects kernel overlap and run-to-run variation. Commit runs after sampling determines the accepted length, so it is not part of model forward. Across the 72 steady batch-32 steps it took 1.711 ms per step and varied with the accepted length. | Post-sampling ReplaySSM commit | Time per step | Share | |---|---:|---:| | FP32 correction state reconstruction | 1.618 ms | 94.5% | | Convolution-state compaction | 0.081 ms | 4.7% | | Commit-plan preparation | 0.0069 ms | 0.4% | | Align bookkeeping | 0.0061 ms | 0.4% | | Total | 1.711 ms | 100% | Adding the steady model-forward difference and post-sampling commit gives an estimated ReplaySSM GPU critical-path cost of 2.046 ms per batch-32 decode step, before common sampling and postprocessing work. ## Duplicate-work check The following searches were run against open vLLM PRs: ```bash gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM Kimi K3\" gh pr list --repo vllm-project/vllm --state open --search \"KDA speculative decoding\" gh pr list --repo vllm-project/vllm --state open --search \"ReplaySSM in:title,body\" ``` No other open PR implements this Kimi-K3 KDA integration. Related work is materially different: - #49847 targets Mamba2; - #49887 targets Gated DeltaNet; - #50140 is a WIP FlashInfer ReplaySSM kernel; - #42792 adds generic Mamba align-prefix speculative support rather than Kimi-K3 KDA ReplaySSM. ## AI assistance AI assistance was used for implementation, tests, profiling analysis, model evaluation analysis, and this PR description. The submitting human must review every changed line, reproduce the relevant tests, and understand and defend the design end-to-end. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan and exact commands. - [x] Unit, end-to-end accuracy, memory, acceptance, and performance results. - [x] Duplicate-work check and AI-assistance disclosure.
**BEFORE SUBMITTING, PLEASE READ **", "url": "https://github.com/vllm-project/vllm/pull/51855", "createdAt": "2026-08-11T17:49:54Z", "updatedAt": "2026-08-13T17:03:24Z", "timestamp": "2026-08-13T17:03:24Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "performance", "needs-rebase", "mrv2", "verified", "kimi", "k3" ], "author": "ZJY0516", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:bd5b31763c09ed1d8b0a", "signalId": "github:vllm-project/vllm:pull_request:52192", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52192", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add tuned fused_moe config for NVIDIA GB10 (Nemotron-3-Super shape, E=512 N=2688)", "text": "**Problem.** No `E=512,N=2688,device_name=NVIDIA_GB10` config exists: every Nemotron-3-Super serve on a DGX Spark falls back to the default heuristic and logs \"Using default MoE config. Performance might be sub-optimal!\" Second config for this device, first for this shape (#45949 covers E=512,N=512 fp8). **What's in the file.** Decode keys M=1,2,4,8 tuned on hardware; rows at M=16,32,64,256,1024,2048,4096 pinned to the default heuristic's own per-M choices. The pins are required: `try_get_optimal_moe_config` snaps to the nearest key with no fallback, and a tuned-keys-only ladder measured −2.35% on pp2048 (prefill-scale M landing on a small-M config); the pinned rows make large-M lookups exactly-stock. The ladder stops at M=8 because the stock tuner cannot complete this shape on unified memory (companion issue to follow — [#MEMORY-ISSUE]); extension is mechanical. **Evidence.** - Kernel-level A/B (CUDA-event, same boot): +2.3% to +5.2% per decode key, **+3.5% geomean** over the default heuristic. - Serve-level paired A/B (12 boots): pp2048 −0.59% [−1.74, +0.55] (n=32/arm); serve decode a wash (+0.09% [−2.56, +2.75]). We claim the kernel-level win only, not an end-to-end speedup. - The startup warning clears with the config present (verified in the leg-B serve log). - Environment: vLLM 83ad767ee, driver 610.57.04, GB10 121 GB unified.", "url": "https://github.com/vllm-project/vllm/pull/52192", "createdAt": "2026-08-13T17:02:39Z", "updatedAt": "2026-08-13T17:03:18Z", "timestamp": "2026-08-13T17:03:18Z", "metrics": { "reactions": 0, "comments": 1 }, "labels": [ "nvidia" ], "author": "maxspevack", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:c8238b215ce297b69d30", "signalId": "github:vllm-project/vllm:pull_request:51647", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51647", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Pad non-aligned AITER MLA heads", "text": "## Summary - pad non-16-aligned ROCm AITER MLA query heads to the next supported multiple of 16 and slice padding from the output - size dense and sparse persistent metadata for the padded launch shape - enable Kimi-K3 TP4's 24 heads/rank to use AITER MLA instead of falling back to Triton MLA ## Performance 8x MI355X, Kimi-K3 TP4/DP2/EP8, 100,000 input tokens (95,911 shared prefix + 4,089 suffix), OSL 1024, one API server, three runs per cohort. | Cohort | Triton MLA TPOT | AITER padded TPOT | TPOT change | Triton output tok/s | AITER output tok/s | Throughput change | |---|---:|---:|---:|---:|---:|---:| | C16 | 62.696 ms | 41.013 ms | -34.6% | 205.40 | 286.48 | +39.5% | | C24 | 76.061 ms | 46.816 ms | -38.5% | 256.32 | 373.99 | +45.9% | Source base: vLLM `61c1dd0966fd8d91954003cf6b7ab746448a46f8`, AITER `ca68b4f3501762c15c550cb920a5516e9710cf89`. ## Test plan - [x] targeted ROCm AITER MLA tests: 32 passed, 2 skipped - [x] Ruff check and format check on all changed files - [x] GSM8K-100: 99/100, 0 malformed outputs, 0 invalid numeric answers - [x] TP4/DP2/EP8 full-graph server startup and 100K C16/C24 decode benchmark Made with [Cursor](https://cursor.com)", "url": "https://github.com/vllm-project/vllm/pull/51647", "createdAt": "2026-08-10T09:20:08Z", "updatedAt": "2026-08-13T17:02:23Z", "timestamp": "2026-08-13T17:02:23Z", "metrics": { "reactions": 2, "comments": 13 }, "labels": [ "rocm", "verified" ], "author": "LiuYinfeng01", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:2515d1562c6594a13b43", "signalId": "github:vllm-project/vllm:pull_request:47392", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:47392", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][MoE] Plumb swigluoai activation into FlashInfer b12x MoE", "text": "## Purpose Models with a clamped SwiGLU-OAI MoE activation (`swigluoai_uninterleave`, e.g. [nvidia/MiniMax-M3-NVFP4](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)) cannot use the FlashInfer b12x MoE backend: `FlashInferB12xExperts` handles only silu and relu2. This matters on SM120/SM121: FlashInfer TRT-LLM MoE requires SM100-family GPUs and FlashInfer CUTLASS lacks this activation, so such models have no FlashInfer MoE path there. FlashInfer supports `swigluoai_uninterleave` on this kernel since flashinfer-ai/flashinfer#3744; this PR adds it to `FlashInferB12xExperts` and passes `swiglu_alpha/beta/limit` to the `B12xMoEWrapper`, gated on the installed FlashInfer supporting them. Behavior with the current pin (0.6.13, which predates the kernel support) is unchanged; a later pin bump enables these models on b12x with no further vLLM changes. The unit test now drives the production weight-processing path instead of hand-preparing weights, covers silu and swigluoai, and uses a relative-error check that actually fails on a wrong gate/up ordering or activation (the old tolerances passed under any kernel behavior). This guards the whole b12x weight pipeline, not just the new activation. Not a duplicate of #47001, which widens b12x `_supports_activation` without plumbing the activation into the kernel call. ## Test Plan ``` pytest tests/kernels/moe/test_flashinfer_b12x_moe.py ``` Needs SM120 and FlashInfer at/past flashinfer-ai/flashinfer#3744 for the swigluoai cases; they skip on the current pin. ## Test Result RTX 5080, FlashInfer at the #3744 merge commit: 48 passed with the fix. Without the fix, all 24 swigluoai cases fail (relative error 0.60 vs the 0.45 bound) and the 24 silu cases pass. On pre-#3744 FlashInfer the swigluoai cases skip and silu passes. Developed with AI assistance (Claude Code).", "url": "https://github.com/vllm-project/vllm/pull/47392", "createdAt": "2026-07-02T05:02:35Z", "updatedAt": "2026-08-13T17:02:11Z", "timestamp": "2026-08-13T17:02:11Z", "metrics": { "reactions": 1, "comments": 5 }, "labels": [ "bug", "needs-rebase", "nvidia" ], "author": "yichengj0", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:7108358b2bfcab05bd44", "signalId": "github:vllm-project/vllm:issue:52071", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52071", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug]: speculative decoding under pipeline parallelism produces wrong output with --no-async-scheduling", "text": "### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Reproduced at `pipeline_parallel_size` 2, 4 and 8, with two different speculative methods and two model families. Code references are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug With `--no-async-scheduling`, speculative decoding under pipeline parallelism produces wrong output — or a CUDA fault — instead of being rejected or working. Turning async scheduling back on fixes it. ### Root cause `next_decode_eligible_step` is assigned in exactly one place in the tree: ``` vllm/v1/core/sched/async_scheduler.py:49 request.next_decode_eligible_step = self.current_step + self.pp_size ``` initialised in one place: ``` vllm/v1/request.py:167 self.next_decode_eligible_step = 0 ``` and read in exactly one place: ``` vllm/v1/core/sched/scheduler.py:509 if self.current_step < request.next_decode_eligible_step: # V2+PP+async: enforce `pp_size` steps between same-req decodes # to match worker-side sampled-tokens broadcast slot ring cadence. ``` The base `Scheduler` — which is what `--no-async-scheduling` selects — reads that field and never sets it. It stays `0`, the guard never fires, and nothing keeps a request's decodes `pp_size` steps apart. The worker-side sampled-token broadcast ring is then read out of phase. ### What that produces, measured A probe at the point where a step's input ids are built (`combine_sampled_and_draft_tokens`), PP=2: ``` last_sampled=[16925] drafts=[16925, 16925, 16925] healthy last_sampled=[0] drafts=[0, 0, 0] ~25% of spec steps ``` Token id 0 is embedded as the anchor. On the model under test token 0 is `!`, and the generated text fills with `!`. The same skew has a second face — the step gets scheduled without the accepted token that its drafts hang off: ``` healthy: hidden=(4,1024) logits_indices=[0, 1, 2, 3] qlen=4 num_logits=4 ndraft=3 failing: hidden=(3,1024) logits_indices=[-1, 0, 1, 2] qlen=3 num_logits=4 ndraft=3 ``` `logits_start = query_end - num_logits = 3 - 4 = -1`, so `hidden_states[input_batch.logits_indices]` either wraps to the last row silently (torch semantics) or trips: ``` Assertion `ind >=0 && ind < ind_dim_size && \"vectorized gather kernel index out of bounds\"` failed ``` The same skew also reaches `_prepare_prefill_inputs_kernel` in `v1/worker/gpu/spec_decode/autoregressive/speculator.py`, where `query_len -= num_rejected` can reach 0 and the unmasked `tl.store(draft_input_ids_ptr + query_start - 1, ...)` writes out of bounds — Triton adds negative indices to the base pointer rather than wrapping. ### Reproduced with - `method: mtp` on GLM-4.5-Air (AWQ), PP=4 - `method: dspark` on Kimi-K3, PP=8 - PP=2 on a 16-layer synthetic target ### Controls Two controls that rule out the obvious alternatives, same tree, same session: ``` speculation OFF, async on vs async off : outputs IDENTICAL Qwen3-0.6B, PP=1 vs PP=2, no speculation : outputs IDENTICAL ``` So the pipeline path is greedy-stable on its own, and the scheduler mode alone changes nothing. The defect appears only when speculation and PP are combined with async scheduling disabled. Also worth noting: with async scheduling ON, plain decode throughput on the same config was ~1.9× better than with it off (94.6 → 48.8 ms/token TPOT, GLM-4.5-Air PP=4, concurrency 1), so the flag is expensive in its own right. ### Suggested fix Any of these beats the current behaviour, in descending order of preference: 1. Set the cadence in the base scheduler too, so the guard works regardless of scheduling mode. 2. Reject `speculative_config` + `pipeline_parallel_size > 1` + `--no-async-scheduling` at config time, with a message naming the cadence. 3. At minimum, document that speculative decoding under PP requires async scheduling. Silently emitting `!` is the worst of the available outcomes, and it is what happens today. ### Before submitting a new issue... - [x] I searched the tracker (several phrasings) and did not find this reported. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/issues/52071", "createdAt": "2026-08-12T23:52:42Z", "updatedAt": "2026-08-13T17:01:20Z", "timestamp": "2026-08-13T17:01:20Z", "metrics": { "reactions": 0, "comments": 6 }, "labels": [ "kimi" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:b745c523050ad7f6f8a4", "signalId": "github:vllm-project/vllm:pull_request:51665", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51665", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix weight tying", "text": "Fixes #51063. vLLM decides whether to tie `lm_head` purely from `tie_word_embeddings` and never looks at the checkpoint, so a config claiming tied weights against a checkpoint with a real `lm_head` silently discards it and generates gibberish. `ModelConfig.maybe_untie_word_embeddings` now checks the safetensors metadata and builds the model untied so the head loads; after loading, `torch.equal` re-ties it if it was just a redundant copy (as in `Qwen/Qwen3-0.6B`) or warns and keeps it if it genuinely differs. This matches `PreTrainedModel.tie_weights`. The reverse case, untied config with no head in the checkpoint, now ties and warns instead of leaving the head uninitialised. That makes the per-model workarounds redundant. `AutoWeightsLoader` now skips weights whose target parameter aliases one already loaded, which removes ~50 hand-rolled `lm_head` skips and also covers the ~20 models that tie with no skip at all. Tying itself is now expressed one way. `self.lm_head = self.lm_head.tie_weights(self.model.embed_tokens)` is the only form that dispatches through `quant_method.tie_weights`, so the other two are converted to it: - `self.lm_head.weight = self.model.embed_tokens.weight` (33 sites) bypassed the quant method entirely, so it was wrong for quant methods that repack. The `ParallelLMHead` already exists at that point, so this is a one-line swap with no structural change. - `self.lm_head = self.model.embed_tokens` (23 sites) built no `ParallelLMHead` at all in the tied branch. These now always build one and tie it. Two consequences worth review: a transient vocab x hidden allocation, and the quant config is now consulted for the `lm_head` prefix where it previously never was. - Five models hardcoded tying with `assert config.tie_word_embeddings` (`gemma`, `gemma2`, `commandr`, `cohere2_moe`, `mpt`). Untying at config time would trip these, so the asserts are dropped and an untied config keeps using the embeddings exactly as it does today. Those five still have no real `lm_head`, so they cannot honour one in a checkpoint; left for a follow-up. So `AutoWeightsLoader` now skips weights whose target parameter aliases one already loaded. That removes ~50 hand-rolled `lm_head` skips and also covers the ~20 models that tie with no skip at all.", "url": "https://github.com/vllm-project/vllm/pull/51665", "createdAt": "2026-08-10T11:38:44Z", "updatedAt": "2026-08-13T16:59:14Z", "timestamp": "2026-08-13T16:59:14Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "speculative-decoding", "llama", "qwen", "gpt-oss", "mistral", "kimi", "k3", "cohere" ], "author": "hmellor", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:b07a4af44b23602ed04c", "signalId": "github:vllm-project/vllm:pull_request:52191", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52191", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CPU] Support FP16/BF16 persisted GDN state on AMX", "text": "## Purpose BF16/FP16 GDN state reduces the aligned block size used by hybrid prefix caching and therefore reduce the prefix-caching glassjaw on CPU AMX backends. ## Test Plan Server ```bash vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \\ --kv-cache-dtype {auto|fp8} \\ --mamba-ssm-cache-dtype {auto|bfloat16} \\ --language-model-only \\ --enable-prefix-caching \\ ``` Client ```bash KV_CACHE_DTYPE=auto MAMBA_SSM_CACHE_DTYPE=auto for PREFIX_LEN in 640 1152 2176 3200 3456 3500 4000 4352; do curl -fsS -X POST http://127.0.0.1:8000/reset_prefix_cache vllm bench serve \\ --model Qwen/Qwen3.6-35B-A3B-FP8 \\ --num-prompts 512 \\ --max-concurrency 128 \\ --dataset-name prefix_repetition \\ --prefix-repetition-prefix-len \"$PREFIX_LEN\" \\ --prefix-repetition-suffix-len 200 \\ --prefix-repetition-output-len 128 \\ --prefix-repetition-num-prefixes 1 done ``` ## Test Result BF16 GDN state throughput improvement over auto/FP32 GDN state, measured as the geometric mean across all eight prefix lengths: | KV cache dtype | Output-token throughput | | --- | ---: | | `auto` | +35.18% | | `fp8` | +37.51% | Align mode uses a common block size for attention KV and recurrent state. In this run, the effective blocks were: | KV cache dtype | Auto GDN block | BF16 GDN block | | --- | ---: | ---: | | `auto` | 1152 tokens | 640 tokens | | `fp8` | 2176 tokens | 1152 tokens | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52191", "createdAt": "2026-08-13T16:57:49Z", "updatedAt": "2026-08-13T16:58:27Z", "timestamp": "2026-08-13T16:58:27Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "cpu" ], "author": "tianmu-li", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:cbe99c6abe18a0497695", "signalId": "github:vllm-project/vllm:pull_request:52190", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52190", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage)", "text": "# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage) ## Purpose Kimi-K3's model classes carry no `@support_torch_compile` and vLLM logs: ``` `torch.compile` is turned on, but the model /path/to/Kimi-K3 does not support it. Please open an issue on GitHub if you want it to be supported. ``` Because there is no inductor post-grad graph, **none of the fusion passes ever run for K3** — even though they are enabled in the config and reported as active at startup: ``` INFO [config/compilation.py] Enabled custom fusions: norm_quant, act_quant, allreduce_rms, mla_dual_rms_norm ... 'pass_config': {..., 'fuse_mla_dual_rms_norm': True, 'fuse_allreduce_rms': True, ...} ``` In particular `MLADualRMSNormFusionPass` already knows how to fuse the paired MLA `q_a_layernorm` + `kv_a_layernorm` into AITER's `fused_qk_rmsnorm` kernel, but it never sees a graph. Confirmed by instrumenting `VllmFusionPatternMatcherPass.__call__`: zero pass invocations and zero pattern dumps on an otherwise fully-configured run. This PR enables compilation so the existing passes can do their job. Two of the three changes are **correctness fixes** that compilation exposes rather than optimizations. ## Changes 1. **`vllm/models/kimi_k3/amd/linear.py`** — add `@support_torch_compile` to `KimiLinearModel`. Its `__init__(self, *, vllm_config, prefix)` and `forward` signatures already match the shape the decorator expects (same as `DeepseekV2Model`). 2. **`vllm/models/kimi_k3/amd/kda.py` + `vllm/config/compilation.py`** — wrap the KDA conv1d + recurrent/chunk core in a `kimi_kda_attention_core` custom op declaring `mutates_args=[\"core_attn_out\"]`, and add `vllm::kimi_kda_attention_core` to the default `splitting_ops`. **This is required for correctness, not perf.** `KimiGatedDeltaNetAttention._forward` writes its result in place into a caller-allocated `core_attn_out` and returns `None`, and was not a registered custom op. Under fullgraph capture the call therefore has no compiler-visible effect and the entire linear-attention path is eliminated: with only the decorator applied, **all KDA kernels for all 69 KDA layers stop executing** (K3 is 24 MLA + 69 KDA), while the surrounding `o_proj` and MoE keep running on the uninitialized `torch.empty` buffer. It fails silently — no exception, no graph-break warning, just degenerate output. Follows the existing `vllm::qwen_gdn_attention_core` pattern. 3. **`vllm/models/kimi_k3/amd/ops/attn_res.py`** — register the attention-residual Triton launcher as `vllm::kimi_attn_res` with `mutates_args=[\"blocks\"]`. Two problems, one op. First, the launcher picks its tile shape with a host-side branch on the token count: ```python if num_tokens >= 256 or num_blocks <= 1: block_l, num_warps = 1, 4 else: block_l, num_warps = 4, 8 ``` Dynamo traced this, evaluated it on the large profile-run batch and emitted the dynamic-shape guard `s72 >= 256`. vLLM's wrapper drops guards, so the artifact specialized for the `>= 256` branch was silently reused for small decode batches. Second, with `block_write_idx >= 0` the kernel stores its prefix sum into `blocks` in place (`WRITE_BLOCK`), which is invisible to the compiler unless declared. Both are fixed by making this a custom op with the mutation declared; found via `TORCH_TRACE` + `tlparse`, reading the `user_stack` field of the raw `guard_added_fast` records. 4. **`vllm/models/kimi_k3/amd/latent_moe_runner.py`** — fullgraph capture cannot trace `logging.Logger` methods ([gb0291](https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0291.html)): ``` torch._dynamo.exc.Unsupported: logging.Logger method not supported for non-export cases Developer debug context: method: .info_once from user code: latent_moe_runner.py:67 in _shard_up_proj_tail ``` The one-shot diagnostic in `_shard_up_proj_tail` is hoisted into `__init__` under the same `self._tail_shardable` condition that gates its only call site, so the message is preserved and both the logger call and the `_logged_sharded_tail` state mutation leave the traced hot path. **No changes to any fusion pass or pattern.** The existing DeepSeek-shaped MLA dual-RMSNorm pattern matches K3 unmodified. ## Test Plan Kimi-K3, MI355X, TP8, MXFP4 weights, AITER enabled. Two stacks: - **A — upstream nightly, no spec decoding**, `--kv-cache-dtype fp8_e4m3`. Decode-only torch-profiler capture, three arms on identical settings (ISL 2048 / OSL 96 / concurrency 16, `wait_iterations=40`, `active_iterations=2`). All windows verified pure decode — `execute_context_0(0)_generation_16(16)`, zero prefill-attention kernels. - **B — AMD downstream K3 stack with DSpark MTP + CUDA graphs** (`FULL_AND_PIECEWISE`, `--kv-cache-dtype fp8`, `num_speculative_tokens=2`), i.e. the configuration this model is actually served in. Added because compilation interacts with cudagraph capture and speculative decoding, and stack A exercises neither. Using agentX coni - **Accuracy** — gsm8k via lm-eval against `/v1/completions`. ## Test Result ### Fusion passes now run (stack A) Cumulative match table for one full model forward, per rank: ``` mla_dual_rms_norm_fusion_pass : 24 <- 1 per MLA layer (K3 has 24) rocm_aiter_allreduce_fusion_pass : 92 <- ~1 per layer (K3 has 93) activation_quant_fusion_pass : 0 rocm_aiter_silu_mul_fp8_group_quant_fusion_pass: 0 RocmAiterRMSNormQuantFusionPass : 0 ``` `24` is exactly the MLA layer count, i.e. single coverage with no double counting. Cross-check against the per-graph-piece `Replaced N patterns` lines: 16x1 + 8x2 + 176x4 = 736 across 8 ranks = 92/rank. Identical with and without the KDA custom op, i.e. the new splitting op does not disturb the fusions. ### Kernel counts, pure-decode window (stack A, 2 steps, rank 0) | kernel group | eager | decorator only | **this PR** | |---|---|---|---| | MLA attention (24 x 2) | 48 | 48 | **48** | | MoE `mfma_moe1` (92 x 2) | 184 | 184 | **184** | | KDA `causal_conv1d` + `fused_recurrent_kda` | 276 | **0** ❌ | **276** ✅ | | `aiter::fused_qk_rmsnorm_kernel` (24 x 2) | 0 | 48 | **48** | | `add_rmsnorm_quant` (all) | 282 | 2 | **2** | | `aiter::allreduce_fusion_kernel_1stage` | 0 | 184 | **184** | The middle column is the decorator applied *without* change (2) — included to show the silent KDA loss that change (2) fixes. Corroborated independently of the profiler by the server log: stock and this PR both log `Triton kernel JIT compilation during inference:` for `chunk_kda_fwd_kernel_intra_sub_chunk`, `fused_recurrent_kda_fwd_kernel`, `_causal_conv1d_fwd_kernel` and seven more (223 / 224 lines); the decorator-only build logs **none** of them. ### Norm kernels in detail (stack A) | Norm kernel | eager | this PR | |---|---|---| | `add_rmsnorm_quant` — `q_a_layernorm` | 48 x 230.392 us | — | | `add_rmsnorm_quant` — `kv_a_layernorm` | 48 x 218.353 us | — | | **`aiter::fused_qk_rmsnorm_kernel`** | — | **48 x 231.194 us** | | `add_rmsnorm_quant` — per-layer input/post-attn | 184 x 912.578 us | — (folded into allreduce fusion) | | `add_rmsnorm_quant` — final norm | 2 x 10.718 us | 2 x 11.078 us | 96 launches / 448.745 us collapse into 48 launches / 231.194 us of `aiter::fused_qk_rmsnorm_kernel`, and the 184 standalone per-layer norms are absorbed by the 92 allreduce+RMSNorm fusions. Decode is where this matters: AITER's `_fused_qk_rmsnorm` wrapper deliberately falls back to two separate `rmsnorm` calls at `m >= 16384` tokens, so the fused kernel is the decode / small-batch path. ### Works under CUDA graphs + speculative decoding (stack B) Pure-decode capture, DSpark MTP, concurrency 2 (verify batch `2*CONC*(1+num_spec)` = 12), `FULL_AND_PIECEWISE`, 6 engine steps, rank 0. Window verified pure decode: `execute_context_0(0)_generation_2(6)` x12, no prefill kernels. | kernel group | launches | per step | expected | |---|---|---|---| | `aiter::fused_qk_rmsnorm` | 144 | 24 | 24 MLA layers ✅ | | `allreduce_fusion_kernel_1stage` | 558 | 93 | 93 layers ✅ | | `fused_recurrent_kda` + `causal_conv1d_update` | 828 | 69 + 69 | 69 KDA layers ✅ | | MLA `mla_a8w8_qh16_qseqlen4_gqaratio16_v3_ps` | 144 | 24 | 24 ✅ | The KDA path stays fully alive at 69 layers/step with compilation, cudagraph capture and MTP all active simultaneously, and the QK fusion covers exactly all 24 MLA layers. Trace showing Cuda graph + MTP + compiler pass fusions active \"image\" ### Accuracy — no regression | stack | harness | baseline | this PR | |---|---|---|---| | A (no spec decode) | gsm8k, full 1319, 5-shot, greedy | 0.9098 ± 0.0079 | **0.9257 ± 0.0072** | | B (DSpark MTP + cudagraph) | gsm8k, 250-sample, 5-shot | 0.988 | **0.984** | Stack A's delta is within ~1.5 combined standard errors. Stack B was run with `rejection_sample_method=block`; the `synthetic` variant is a throughput approximation and is not a valid accuracy configuration. ## Known limitation: CUDA graph capture ladder (stack B) On the AMD downstream MTP stack, enabling compilation narrows the range of CUDA graph capture sizes that can be captured. Outside that range the **per-size warmup forward** — before any capture begins, descending from the largest size — dies with `Memory access fault by GPU node-N`. **Root cause is not yet understood.** It is reported here rather than left for a reviewer to hit. Note this is observed only on the downstream stack — the DSpark MTP method and that cudagraph path are not upstream — so it may not be reachable from an upstream configuration. It does not affect stack A, which captures the default ladder without incident.", "url": "https://github.com/vllm-project/vllm/pull/52190", "createdAt": "2026-08-13T16:57:07Z", "updatedAt": "2026-08-13T16:57:46Z", "timestamp": "2026-08-13T16:57:46Z", "metrics": { "reactions": 0, "comments": 0 }, "labels": [ "rocm", "kimi", "k3" ], "author": "rbrugaro-amd", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:4ae146023367f2866069", "signalId": "github:vllm-project/vllm:pull_request:52016", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52016", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X dense linear backends", "text": " ## Purpose This PR integrates [B12X](https://github.com/local-inference-lab/b12x) dense linear kernels for NVIDIA SM120 and SM121 GPUs through the existing vLLM linear backend interfaces. B12X is an optional dependency installed with `vllm[b12x]` and pinned to `b12x==1.2.4`; it is a pure-Python CuTe DSL package and requires no additional vLLM build step. Supported linear paths are: - Per-tensor FP8. - 128x128 block-scaled FP8. - MXFP8. - NVFP4 and MXFP4. B12X participates in automatic kernel selection after established optimized backends and before emulation. Users can also request it with `--linear-backend b12x`; linear families that B12X does not implement, such as dense W4A16, retain normal backend selection so mixed-format models continue to work. Warmup is limited to layers that selected a B12X kernel. It covers every configured CUDA-graph capture size plus `max_num_batched_tokens`. B12X internally deduplicates shapes that select the same kernel policy, while vLLM deliberately avoids duplicating B12X policy-selection heuristics. The documentation covers installation, selection, supported formats, and fallback behavior. This is the linear and shared-integration component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 target FlashInfer-embedded B12X paths or narrower integrations and do not provide this standalone optional linear backend. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused dense-kernel and warmup coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/model_executor/kernels/test_b12x_mxfp4_linear.py \\ tests/model_executor/kernels/test_b12x_nvfp4_linear.py \\ tests/model_executor/kernels/test_b12x_mxfp8_linear.py \\ tests/model_executor/test_b12x_warmup.py \\ tests/kernels/quantization/test_block_fp8.py \\ -k b12x -q ``` Run pre-commit over every changed file: ```bash mapfile -t changed_files < <(git diff --name-only upstream/main...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check upstream/main...HEAD ``` Benchmark `Qwen/Qwen3.6-27B-FP8` on an RTX PRO 6000 Blackwell Max-Q GPU, holding the attention backend constant and comparing B12X with CUTLASS. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10 at the exact rebased PR head: ```text 52 passed, 473 deselected in 24.86s ``` All changed-file pre-commit hooks passed, including Ruff, formatting, mypy, and markdownlint; `git diff --check` also passed. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | Dense block FP8 | Qwen3.6-27B-FP8 | 1 | CUTLASS | 54.3975 | 56.7518 | +4.33% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52016", "createdAt": "2026-08-12T16:04:49Z", "updatedAt": "2026-08-13T16:57:44Z", "timestamp": "2026-08-13T16:57:44Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "ready", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a91d8937e6d761e62f81", "signalId": "github:vllm-project/vllm:pull_request:52018", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52018", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X FP4 MoE backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency, shared lazy imports, packed-storage reuse, and warmup integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) FP4 MoE backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing fused-MoE backend interfaces. It does not introduce a new MoE abstraction. Supported paths include: - Native NVFP4 and MXFP4 W4A4. - W4A16 and supported dynamic W4A8 activation modes. - MXFP4 defaults to MXFP8 activations where supported; NVFP4 retains its checkpoint activation format. - `VLLM_B12X_MOE_FP4_FORCE_A16=1` forces both FP4 formats onto the BF16 activation path. - Tensor parallelism, warmup, reload-stable packed storage, and CUDA graph replay. Expert parallelism, expert maps, EXL3, and NF3 are unsupported. B12X 1.2.4 owns and registers the custom operator used by the backend. Warmup supplies the complete set of serving sizes available to vLLM. B12X deduplicates sizes that resolve to identical internal policy choices; vLLM does not duplicate or guess those package-internal policies. The documentation covers explicit `--moe-backend b12x` selection, supported activation modes, the A16 override, and unsupported expert configurations. This is the MoE component split from #51696, which it supersedes rather than duplicates. Related PRs #41243 and #47577 do not provide this standalone optional-package MoE integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the B12X MoE backend-selection, correctness, warmup, storage-reuse, and CUDA graph coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/kernels/moe/test_b12x_moe.py -q .venv/bin/python -m pytest \\ tests/model_executor/test_b12x_warmup.py \\ tests/quantization/test_auto_round.py::test_inc_mxfp4_moe_method_registers_weights_and_builds_kernel \\ -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `nvidia/MiniMax-M2.7-NVFP4` on an RTX PRO 6000 Blackwell Max-Q GPU, comparing B12X with FlashInfer CUTLASS MoE at TP2. Use five warmup and ten measured requests. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 54 MoE backend tests passed in 23.63s 2 warmup and weight-lifecycle tests passed in 1.09s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | NVFP4 MoE | MiniMax-M2.7-NVFP4 | 2 | FlashInfer CUTLASS MoE | 132.1558 | 137.8276 | +4.29% | ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52018", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:57:21Z", "timestamp": "2026-08-13T16:57:21Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build", "quantization" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:71aadd0f754030e4da98", "signalId": "github:vllm-project/vllm:pull_request:52017", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52017", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Kernel] Add B12X causal paged attention backend", "text": "## Purpose Depends on #52016 for the optional B12X dependency and shared lazy-import integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) causal paged-attention backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing attention backend interface. It does not modify generic model-runner behavior or introduce a new attention abstraction. Supported paths include: - Causal paged MHA, MQA, and GQA with BF16 model dtype. - BF16 and FP8 E4M3 KV caches. - Prefill, decode, and mixed batches. - Speculative verification, sliding-window attention, and attention sinks. - CUDA graph capture and replay. MLA, non-causal attention, ALiBi, logits soft capping, and context parallelism are unsupported. The documentation covers explicit `--attention-backend B12X_ATTN` selection, supported cache formats and head sizes, and unsupported attention modes. This is the causal-attention component split from #51696, which it supersedes rather than duplicates. Related open work targets different backend boundaries and does not provide this standalone optional-package attention integration. AI assistance from OpenAI Codex was used while developing this PR. I reviewed every changed line and am responsible for understanding and defending the integration end-to-end. ## Test Plan Run the focused backend and reference-correctness coverage: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python -m pytest \\ tests/v1/attention/test_b12x_attn.py \\ tests/v1/attention/test_attention_backends.py \\ -k b12x -q ``` Run pre-commit over every file changed by this PR layer: ```bash mapfile -t changed_files < <(git diff --name-only dev/b12x-linear...HEAD) .venv/bin/pre-commit run --files \"${changed_files[@]}\" git diff --check dev/b12x-linear...HEAD ``` Benchmark `Qwen/Qwen3-8B` on an RTX PRO 6000 Blackwell Max-Q GPU with BF16 model weights, TP1, an 8192-token input, and a 512-token output. Hold linear execution constant and compare B12X with FlashInfer using both BF16 and FP8 E4M3 KV caches. Use three warmup and ten measured requests per result. ## Test Result Focused tests against the published `b12x==1.2.4` wheel on GPU 10: ```text 41 passed, 124 deselected in 38.59s ``` All changed-file pre-commit hooks passed, and `git diff --check` reported no errors. Single-request end-to-end decode throughput; higher is better: | Path | Model/configuration | TP | Comparison backend | Comparison tok/s | B12X tok/s | Change | | --- | --- | ---: | --- | ---: | ---: | ---: | | BF16-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 88.5677 | 89.6471 | +1.22% | | FP8-KV attention | Qwen3-8B, 8K context | 1 | FlashInfer | 91.0632 | 92.3668 | +1.43% | The attention measurements exclude prefill and use 5,110 post-first-token outputs across ten requests per result. All 40 measured requests completed successfully. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR. - [x] The test plan, including commands. - [x] The test and performance results. - [x] The necessary documentation update.
", "url": "https://github.com/vllm-project/vllm/pull/52017", "createdAt": "2026-08-12T16:05:50Z", "updatedAt": "2026-08-13T16:57:19Z", "timestamp": "2026-08-13T16:57:19Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "documentation", "ci/build" ], "author": "lukealonso", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1cf2a1eae7fdd2becbe4", "signalId": "github:vllm-project/vllm:pull_request:51778", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51778", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add prediction token usage details", "text": "## Summary - expose per-request speculative decoding acceptance and rejection counts through engine outputs - populate `completion_tokens_details.accepted_prediction_tokens` and `rejected_prediction_tokens` for streaming and non-streaming Chat Completions and Completions responses - preserve per-choice accounting for parallel sampling and add focused scheduler, output processor, and serving tests This PR intentionally does not implement `reasoning_tokens`. Existing PRs, including #35037, already cover that work. `CompletionTokenUsageInfo` includes the field for protocol completeness, but this change leaves it unset. ## Why this is not a duplicate I checked issue #50895 and searched open PRs by the issue number and by `accepted_prediction_tokens rejected_prediction_tokens`. No open PR implements these two fields. The existing related PRs focus on `reasoning_tokens`, while this PR handles the speculative-decoding fields listed as untracked in #50895. ## Tests Passed: ```text .venv/bin/ruff check --ignore ISC004 .venv/bin/ruff format --check .venv/bin/python -m compileall -q git diff --check pre-commit hooks run by `git commit` ``` Focused pytest execution: ```text .venv/bin/python -m pytest \\ tests/v1/core/test_scheduler.py::test_stop_via_update_from_output \\ tests/v1/engine/test_output_processor.py::test_speculative_token_counts_accumulate_per_request \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_prediction_token_details \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_omits_prediction_details_without_spec_decode \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_completion_sums_prediction_details_across_choices \\ tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_chat_streaming_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_prediction_token_details \\ tests/entrypoints/openai/completion/test_completion_error.py::test_completion_streaming_prediction_token_details -q ``` Result: `8 passed, 15 warnings in 2.89s`. ## Model evaluation Not applicable. This change reports existing speculative-decoding accounting and does not affect model output, accuracy, or serving decisions. ## AI assistance AI assistance was used to implement this change. I have reviewed every changed line and understand the implementation end to end. Fixes #50895", "url": "https://github.com/vllm-project/vllm/pull/51778", "createdAt": "2026-08-11T07:22:23Z", "updatedAt": "2026-08-13T16:56:42Z", "timestamp": "2026-08-13T16:56:42Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "frontend", "needs-rebase" ], "author": "luyixiao95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:be071645e84d5f883780", "signalId": "github:vllm-project/vllm:pull_request:50544", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50544", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] strict=false in response_format json_schema", "text": "Per OpenAI semantics, strict=false means best-effort: the schema may guide the model via the prompt, but the output is not grammar constrained. vLLM previously ignored the flag and always applied guided decoding. Now skip the json structured-output override when strict=false, on both Chat Completions and Responses APIs. Default (absent) and strict=true keep full guided decoding, so existing clients are unaffected. ## Purpose ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50544", "createdAt": "2026-07-31T07:25:38Z", "updatedAt": "2026-08-13T16:55:48Z", "timestamp": "2026-08-13T16:55:48Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "frontend", "tool-calling", "cohere" ], "author": "wangln19", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:d9653c6757590038d6af", "signalId": "github:vllm-project/vllm:pull_request:50156", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50156", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Cohere][Spec Decode] Add CohereEagleProposer to support multi layer eagle drafts", "text": "Follow up to #42078 with model released. Cohere Eagle draft models have multipe layers and HybridKVCacheManager split these layers across multiple KV cache groups, which the base EagleProposer does not support (it assumes a single group with one shared block table / slot mapping). This PR adds CohereEagleProposer, which tracks per-group block tables and slot mapping buffers so each group's draft layers read/write their own KV cache correctly. > This is only relevant for MRv1. MRv2 natively supports draft layers spread across different KV groups. This PR also includes a few misc fixes to cohere model definitions. Eagle draft model for `CohereLabs/North-Mini-Code-1.0` uploaded to HF here: `CohereLabs/North-Mini-Code-1.0-eagle`. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/50156", "createdAt": "2026-07-28T15:33:31Z", "updatedAt": "2026-08-13T16:53:44Z", "timestamp": "2026-08-13T16:53:44Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "speculative-decoding", "v1", "verified", "cohere" ], "author": "kkt-cohere", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:fc8fa6ac35de39e354c9", "signalId": "github:vllm-project/vllm:pull_request:49819", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49819", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model] Add Cohere2MoE Eagle3 auxiliary hidden states", "text": "## Purpose Enable Cohere2MoE target models to serve with compatible Eagle3 and DFlash draft models by exposing the auxiliary hidden states required by the speculator. I ~am working on training~ have released a DFlash speculator for North-Mini-Code-1.0 and realized that this was missing! Hopefully it helps others, ~especially when I upload the checkpoints.~ :) ### Update: Public Checkpoints As promised, the checkpoints are now public and there’s a turnkey runtime so this change can be exercised without a custom build. **Weights (auxiliary draft/speculator checkpoints, Apache-2.0):** - DFlash: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dflash - DSpark: https://huggingface.co/sdougbrown/North-Mini-Code-1.0-dspark **Runtime:** a thin overlay on official vLLM v0.27.1 carrying exactly this PR (#49819) plus PR #50937 (skip empty expert bias — required because North-Mini-Code w4a16 has an all-zero per-expert bias, causing vllm `0.27.1` to throw `AttributeError: 'RoutedExperts' object has no attribute 'w2_bias'`). Nothing is compiled from source; it’s two small pure-Python module overlays. - Repo: https://github.com/sdougbrown/north-mini-code-draft-runtime - Published multi-arch image: `ghcr.io/sdougbrown/north-mini-code runtime:v0.27.1-49819-50937` **Validation:** both DFlash (K3) and DSpark (K4) serve and run speculative decoding on NVIDIA GB10. ### Running the Draft Models You can use the pre-built docker runtime I put togther if you like, or apply the patches from https://github.com/sdougbrown/north-mini-code-draft-runtime/tree/main/patches to a fresh `0.27.1` wheel and then serve: ``` vllm serve sdougbrown/North-Mini-Code-1.0-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' ``` The easy docker path is: ``` docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ sdougbrown/North-Mini-Code-1.0-dspark \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --max-model-len 32768 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` Note that by default this fetches the weights from huggingface. If that's not desired, you can get fancier. Here's an example I would use on my local machine assuming `~/Code` is the parent folder for the runtime repo, and `~/Models` is where I've downloaded the weights: ``` # 1) Point the verifier at a local copy of the target and render a bundle: cd ~/Code/north-mini-code-draft-runtime VERIFIER_MODEL=\"$HOME/Models/North-Mini-Code-1.0-w4a16\" \\ RUNTIME_WORK=\"$HOME/Models/north-deploy\" \\ ./scripts/prepare-deployment-model.sh dspark # -> $HOME/Models/north-deploy/deploy/dspark # (config.json now has verifier.name_or_path -> local path; weights are symlinks) # 2) Docker: mount Models at the SAME path and pass the bundle path: docker run --rm --gpus all --ipc host --shm-size 16g -p 8087:8000 \\ -e VLLM_USE_V2_MODEL_RUNNER=1 \\ -v \"$HOME/Models:$HOME/Models\" \\ ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937 \\ \"$HOME/Models/north-deploy/deploy/dspark\" \\ --served-model-name north-mini-code-dspark \\ --tensor-parallel-size 1 \\ --gpu-memory-utilization 0.5 \\ --enable-auto-tool-choice \\ --tool-call-parser cohere_command4 \\ --tokenizer-mode cohere \\ --cohere-format cmd4 \\ --reasoning-config '{\"reasoning_start_str\":\"<|START_THINKING|>\",\"reasoning_end_str\":\"<|END_THINKING|>\"}' \\ --host 0.0.0.0 --port 8000 ``` ## Implementation - implement `SupportsEagle3` for `Cohere2MoeForCausalLM` - capture embedding and post-layer boundaries using global layer indices - preserve Cohere2's parallel-residual behavior without adding the residual twice - preserve baseline model outputs when auxiliary states are disabled - preserve DFlash's `target_layer_ids -> auxiliary boundary + 1` semantics Eagle3, DFlash, and DSpark with pipeline parallelism are already rejected by the V1 GPU model runner. This change does not add auxiliary-state propagation between pipeline stages. ## Duplicate-work check PR #42078 added Cohere Eagle support, but it did not expose auxiliary hidden states from the Cohere2MoE target. Searches of open PRs found no overlapping implementation. ## Tests ```bash .venv/bin/pre-commit run --files \\ vllm/model_executor/models/cohere2_moe.py \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py .venv/bin/python -m pytest \\ tests/model_executor/test_cohere2_moe_eagle.py \\ tests/v1/spec_decode/test_dflash_causality.py -q ``` All pre-commit hooks passed. Pytest result: `15 passed`. ## Model evaluation The original `a1cad4f67` implementation was successfully exercised with: - AMD ROCm 7.2.4, TP=2, INT4/AutoGPTQ - NVIDIA GB10 CUDA 13, TP=1, W4A16 with MARLIN NVFP4 MoE - NVIDIA GB10 CUDA 13, TP=1, FP8 with TRITON FP8 MoE All three configurations completed target-model requests while exposing five finite auxiliary hidden states. The ROCm TP=2 run also verified byte-identical per-layer values across both ranks. These runs validated the target-side auxiliary-state path rather than end-to-end draft-token acceptance. ## AI assistance AI assistance was used to port and test this change. I manually reviewed every changed line and the test results.", "url": "https://github.com/vllm-project/vllm/pull/49819", "createdAt": "2026-07-25T17:06:03Z", "updatedAt": "2026-08-13T16:52:23Z", "timestamp": "2026-08-13T16:52:23Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "speculative-decoding", "v1", "cohere" ], "author": "sdougbrown", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e78df0a10109b7ca468d", "signalId": "github:vllm-project/vllm:issue:51921", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:51921", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Bug] v0.27.0 engine permanently stalls after ~1 min idle on 4-node TP=4 (GB10/sm_121, aarch64): shm_broadcast writer starves, requests never reach scheduler", "text": "## Bug On 4-node TP=4 (GB10/sm_121a, aarch64), a v0.27.0 engine serves its first requests normally, then **permanently stalls after roughly one minute of engine idleness**. Requests after the gap never enter the scheduler (`num_requests_running` stays 0), the API keeps answering `/v1/models` and `/tokenize`, and the head's EngineCore logs repeat: ``` [shm_broadcast.py:802] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work ``` every 60s indefinitely. All four rank containers stay alive; all TP workers' last log lines are healthy post-warmup (graph capture complete). Only a full fleet restart recovers. Reproduced 3/3 boots. Keeping the API warm (`/v1/models` polling every 45s) does NOT prevent it; the stall follows engine-level idleness, not connection idleness. Sequence, timestamped from one boot: warmup collectives complete across all ranks (so the broadcast queue works) → first 1-2 completions succeed (~1.3s each) → ~1-2 min idle → all subsequent requests hang forever. ## Environment Official `vllm/vllm-openai:v0.27.0-aarch64` + two patches required to get this far on sm_121: DeepGEMM repinned to `2fd67329` (#51758) and the `masked_mha_available` fix (#51920). GLM-5.2 (GlmMoeDsaForCausalLM), TP=4 via `--nnodes 4 --node-rank N`, mp executor, `FLASHINFER_MLA_SPARSE_SM120` + FLASH_ATTN MLA prefill, `fp8_ds_mla` KV, MTP k=2 spec decode, dual-rail RoCEv2, `VLLM_USE_BREAKABLE_CUDAGRAPH=0` set. torch 2.13.0+cu130 bundled NCCL. Related: #51758 validated v0.27.0 on 2x Spark (TP=2 and PP=2) with a 400-request soak and zero hangs — consistent with our result that the first requests DO work; the failure needs the multi-node idle gap, and possibly the 4-node width. This looks adjacent to the pre-0.27 TP-hang family that release fixed for 2-node. Full per-rank logs, exact launch argv, and timings available; the cluster reproduces this in ~20 minutes per attempt and I can run instrumented builds or targeted env matrices on request.", "url": "https://github.com/vllm-project/vllm/issues/51921", "createdAt": "2026-08-12T04:03:50Z", "updatedAt": "2026-08-13T16:51:51Z", "timestamp": "2026-08-13T16:51:51Z", "metrics": { "reactions": 0, "comments": 11 }, "labels": [], "author": "joesinvestments", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6e50ef15224bee522c57", "signalId": "github:vllm-project/vllm:pull_request:49602", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49602", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Hoist $defs/definitions in Cohere parser tool schema composition", "text": "## Purpose Fix a bug where any `/v1/chat/completions` request with tools whose JSON Schema contains `$defs`/`definitions` fails with HTTP 500 on Cohere models served with `--reasoning-parser cohere_command3|cohere_command4`. Resolves the bug class reported in #16467 (closed as stale, never fixed) for the Cohere reasoning parser path specifically. Other parsers that compose their own grammars are out of scope (see the GLM case in #47175); the generic tool path in `vllm/tool_parsers/utils.py` already hoists `$defs` and is unaffected. **Root cause.** `collect_tool_schema` in `vllm/reasoning/cohere_command_reasoning_parser.py` builds the constrained-decoding grammar for tool calls by nesting each tool's `parameters` schema inside a `{tool_call_id, tool_name, parameters}` envelope. The tool schema was spliced in verbatim, so a definitions block originally at the schema root ends up under `properties.parameters`, while its `$ref` pointers (`#/$defs/X`) remain root-anchored. xgrammar then correctly rejects the dangling JSON pointer: ```text RuntimeError: Cannot find field $defs in #/$defs/X ``` The request fails during preprocessing, before generation. Agentic frameworks such as pydantic-ai emit `$defs` for any nested model, so this breaks effectively all tools defined via such frameworks (see user reports in #16467). **Minimal repro** against a Cohere model served with `--reasoning-parser cohere_command4 --tool-call-parser cohere_command4 --enable-auto-tool-choice` (any tool schema whose parameters carry `$defs` + `$ref`; a flat schema without `$defs` does not reproduce): ```bash curl -s http://localhost:8000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"CohereLabs/North-Mini-Code-1.0\", \"messages\": [{\"role\": \"user\", \"content\": \"Weather in Prague?\"}], \"tool_choice\": \"auto\", \"tools\": [{ \"type\": \"function\", \"function\": { \"name\": \"get_weather\", \"parameters\": { \"$defs\": { \"Location\": { \"type\": \"object\", \"properties\": { \"city\": {\"type\": \"string\"}, \"country\": {\"type\": \"string\"} }, \"required\": [\"city\"] } }, \"type\": \"object\", \"properties\": {\"location\": {\"$ref\": \"#/$defs/Location\"}}, \"required\": [\"location\"] } } }] }' ``` Before this fix: HTTP 500 with `Cannot find field $defs in #/$defs/Location`. After: HTTP 200 with a valid `tool_calls` response. **Fix.** Build the envelope as a dict and hoist `$defs` (draft 2020-12) and `definitions` (draft-07) from the tool's parameters to the envelope root before calling `xgr.Grammar.from_json_schema`, so root-anchored refs resolve again. This mirrors what the shared tool path already does (`_get_tool_schema_defs` in `vllm/tool_parsers/utils.py`); the Cohere parser bypasses that path for its custom tag/envelope format and had reintroduced the bug. No cross-tool merge handling is needed here because each tool gets its own grammar. The parameters dict is copied so the incoming request object is not mutated. Grammars for schemas without definitions blocks are unchanged. Same bug class was recently fixed for the GLM tool parser path in #47175 (different code path, no overlap). **Not a duplicate.** No open PR references #16467 or touches the Cohere parser's tool-schema composition (checked `gh pr list` searches for \"cohere defs\", \"collect_tool_schema\", \"16467 in:body\"). The nearest related work, #47175, fixes the analogous issue in `glm47_moe_tool_parser.py`. ## Test Plan New regression tests in `tests/reasoning/test_cohere_command_reasoning_parser.py` (`TestCollectToolSchemaDefs`): nested-`$defs` tool through `adjust_request`, recursive `$defs` (can never be inlined away), draft-07 `definitions`, and a no-mutation check on the caller's parameters dict. ```bash python -m pytest tests/reasoning/test_cohere_command_reasoning_parser.py -v pre-commit run --files vllm/reasoning/cohere_command_reasoning_parser.py tests/reasoning/test_cohere_command_reasoning_parser.py ``` ## Test Result - Before the fix: all 4 new tests fail with `RuntimeError: Cannot find field $defs in #/$defs/Location` (the error). - After the fix: 44 passed (4 new + 40 pre-existing) in `tests/reasoning/test_cohere_command_reasoning_parser.py`. - pre-commit (ruff, mypy, etc.): all hooks pass. No model evaluation is included: the change only affects grammar composition for schemas that previously failed with a hard 500 before generation; grammars for all previously-working schemas are unchanged (no `$defs` block to hoist), so there is no before/after output quality to compare. --- AI assistance was used for this PR (Cursor agent): drafting the fix and the regression tests. All changes were reviewed, run, and are understood and defended by the submitter. Commit includes `Co-authored-by: Cursor Agent`. ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49602", "createdAt": "2026-07-23T15:04:30Z", "updatedAt": "2026-08-13T16:51:37Z", "timestamp": "2026-08-13T16:51:37Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "tool-calling", "cohere" ], "author": "vaclavcadek", "state": "open", "assignees": [ "DarkLight1337" ], "change": "new" } }, { "id": "event:3d62816e3642ace5f8e6", "signalId": "github:vllm-project/vllm:pull_request:49038", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49038", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Make `load_weights` completely optional", "text": "Part of https://github.com/vllm-project/vllm/issues/48972 Infrastructure changes: - Add `autoload_weights` and use it in weight loaders and runners. This creates it's own `AutoWeightsLoader` if the model doesn't have a `load_weights` method - When the model has no `load_weights` method, auto apply any `hf_to_vllm_mapper` in `AutoWeightsLoader._load_module` as the model is recursed into - Remove `skip_prefix` and `skip_substrs` from `AutoWeightsLoader`. These arguments duplicate the behaviour achievable with the mapper. Convert all uses of these arguments to mapper mappings - Add automatic `lm_head` skipping for models with `tie_word_embeddings` so every model doesn't need to specify this manually - Add `fused_qkv_interleaved` to `QKVParallelLinear` so that it can de-interleave a fused weight itself", "url": "https://github.com/vllm-project/vllm/pull/49038", "createdAt": "2026-07-18T13:28:20Z", "updatedAt": "2026-08-13T16:48:54Z", "timestamp": "2026-08-13T16:48:54Z", "metrics": { "reactions": 2, "comments": 3 }, "labels": [ "speculative-decoding", "ready", "needs-rebase", "v1", "llama", "qwen", "deepseek", "gpt-oss", "ready-run-all-tests", "mistral", "kimi", "cohere" ], "author": "hmellor", "state": "open", "assignees": [ "sangstar" ], "change": "new" } }, { "id": "event:796c1b3a423080b11049", "signalId": "github:vllm-project/vllm:pull_request:48870", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48870", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model][Quant] Fused WNA16 GEMM for tied quantized lm_head logits", "text": "## Purpose Follow-up to #45535 (compressed-tensors WNA16 input embeddings + tied `lm_head`). This PR speeds up the **logits matmul** when a quantized embedding is reused as a tied `lm_head`. In #45535, `CompressedTensorsEmbeddingWNA16Int.apply()` dequantizes the entire packed `[vocab, hidden]` table into a dense weight on every call and runs `F.linear`. This PR routes that matmul through vLLM's existing WNA16 **Linear** kernel (Marlin/Machete) instead — a fused dequant-GEMM that never materializes the dense weight. Key insight: the embedding's packed weight is **already** in the compressed-tensors layout those kernels expect (packed along the input/hidden dim, identical to a WNA16 Linear), so `process_weights_after_loading` sets up the fused kernel directly. It's set up **only** for tied embeddings (flagged in `tie_weights`), keeping a gather-format copy for the input-lookup path. When no fused kernel is available it falls back to a full-table dequant (no index tensor) + `F.linear`. > [!NOTE] > **Stacked on #45535** — the tied-embedding `apply()` this optimizes only exists there, so the net-new change here is `compressed_tensors_embedding.py`; the rest of the diff belongs to #45535. Please review/merge after #45535. Kept as a separate PR to avoid growing the (already large) plumbing PR. **Not a duplicate:** quantized-`lm_head`-via-Marlin already exists for the *untied / separately quantized* case — a `ParallelLMHead` independently quantized as a GPTQ/AWQ/compressed-tensors **Linear** already dispatches to Marlin (cf. #40999). This PR covers the distinct *tied* case, where the embedding **is** the `lm_head` and its quant method is the embedding method (`CompressedTensorsEmbeddingWNA16Int`), not a Linear method. The open ModelOpt lm_head/embedding PRs (#35660, #42791, #44671, #41000) target a different backend (NVFP4/FP8 in `modelopt.py`) and address loading/dispatch, not this fused tied-logits path. No open PR touches `compressed_tensors_embedding.py`. ## Correctness The fused path matches the dequant reference numerically (max relative diff **0.025%**, fp16). The existing tied fixture test (`tests/quantization/test_quantized_embedding.py::test_tied_quantized_embedding`) exercises the fused Marlin path end-to-end and generates coherently. ## Perf evals RTX 5080 (sm120, CUDA 13), `apply()` microbenchmark, µs/call (lower is better): | dims | M | current (dequant + `F.linear`) | no-gather dequant | **fused (Marlin)** | fused vs current | | --- | --- | --- | --- | --- | --- | | vocab 151936 × 1024, W4-g64 | 1 | 1329 | 1317 | **240** | **5.5×** | | | 32 | 1703 | 2395 | **619** | **2.8×** | | | 256 | 3247 | 2460 | **2266** | 1.4× | | vocab 151936 × 1024, W8-g128 | 1 | 2859 | 1374 | **329** | **8.7×** | | | 32 | 1442 | 575 | **575** | 2.5× | | | 256 | 2971 | 2088 | **2088** | 1.4× | | vocab 50304 × 512, W4-g64 | 1 | 141 | 142 | **19** | **7.6×** | | | 32 | 192 | 190 | **123** | 1.6× | Biggest win at decode (`M=1`), the common serving case for logits. The no-gather dequant (kept as the fallback path) is marginal versus the current dequant, as expected — the fused GEMM is the real lever. ## Test Plan ``` pytest tests/quantization/test_quantized_embedding.py ``` ## Test Result - Both fixture tests pass on an RTX 5080; the tied test confirms dispatch to `MarlinLinearKernel` for the logits path (verified `logits_kernel` is set and gather copies are preserved). - `pre-commit` (ruff + mypy) clean on the changed file. --- This change was developed with AI assistance (Claude Code). All changed lines were reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/48870", "createdAt": "2026-07-16T18:13:30Z", "updatedAt": "2026-08-13T16:48:00Z", "timestamp": "2026-08-13T16:48:00Z", "metrics": { "reactions": 0, "comments": 3 }, "labels": [ "speculative-decoding", "needs-rebase", "llama", "qwen", "deepseek", "gpt-oss", "quantization", "mistral", "kimi", "cohere" ], "author": "KKothuri", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:ebbfed51d40fab9fa33c", "signalId": "github:vllm-project/vllm:pull_request:51159", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51159", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm] Defer `tilelang` import through its import `from vllm.tilelang_utils import tilelang` and relaxed `has_tilelang`", "text": "## Purpose Fixes https://github.com/vllm-project/vllm/issues/51151 This PR introduces `vllm.tilelang_utils` and prevents direct `tilelang` imports. Avoid importing TileLang during ROCm module import, because importing TileLang can load **wrongful/bugged** TVM and HIP stub symbols into the global process scope before AITER loads its JIT modules. This changes `_tilelang_jit` so ROCm applies `tilelang.jit` lazily on first kernel call, while non-ROCm keeps eager JIT decoration. This PR is AI-assisted. Proper fixes in followups need to be: * a PR in `tilelang` or TVM to fix the wrongful symbol load. * a PR in vLLM so that `_has_module` does not import modules https://github.com/vllm-project/vllm/issues/51162 ## Test Plan - `python tools/pre_commit/check_forbidden_imports.py --test-regex` - `pytest tests/kernels/test_mhc_tilelang_jit.py -v` Added new test to verify that CUDA applies `tilelang.jit` eagerly, ROCm defers `tilelang.jit` until first call, and the ROCm wrapper caches the compiled kernel. ```python VLLM_ROCM_USE_AITER=1 HIP_VISIBLE_DEVICES=1 pytest tests/quantization/test_online.py::test_online_quantization[True-fp8_per_tensor-None-Fp8PerTensorOnlineLinearMethod-Fp8PerTensorOnlineMoEMethod] -s -vvvvv ``` that is failing on `main` following https://github.com/vllm-project/vllm/pull/50879, and which surfaces the issue https://github.com/vllm-project/vllm/issues/51151 ## Test Result `test_mhc_tilelang_jit.py` pass on MI350. `tests/quantization/test_online.py` as well on MI300 (previous failing case) `python tools/pre_commit/check_forbidden_imports.py --test-regex` passes as well.", "url": "https://github.com/vllm-project/vllm/pull/51159", "createdAt": "2026-08-05T12:25:31Z", "updatedAt": "2026-08-13T16:45:26Z", "timestamp": "2026-08-13T16:45:26Z", "metrics": { "reactions": 2, "comments": 26 }, "labels": [ "documentation", "rocm", "ready" ], "author": "fxmarty-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:d2ed5b79efbe4e94f0f6", "signalId": "github:vllm-project/vllm:issue:52181", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:issue:52181", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "issue", "title": "[Feature]: FA2 is only supported on devices with compute capability >= 8", "text": "### 🚀 The feature, motivation and pitch I am trying to use the vllm for loading [QWEN3.6 model](https://huggingface.co/Qwen/Qwen3.6-27B) Operating system= rocky linux 9 GPU = 2X Quadro RTX 8000 48 GB As soon as I try to load with the command `vllm serve ./Qwen3.6-27B` there is error below. `(EngineCore pid=1673840) ERROR 08-13 16:42:32 [fa_utils.py:273] Cannot use FA version 2 is not supported due to FA2 is only supported on devices with compute capability >= 8` No matter i use quantized model or the simple, I cannot load. This is bottle neck and this must have some solution to load via some flag or additional parameter. Kindly I need your support and help to use this model on the mentioned GPU. ### Alternatives Tell me the way how to make a quantized version of this above model with vllm which I can load and use on the cuda compute capability 7.5 as my GPU is Quadro RTX 8000 which has power less than 8.0 cuda compute capability ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.", "url": "https://github.com/vllm-project/vllm/issues/52181", "createdAt": "2026-08-13T14:52:13Z", "updatedAt": "2026-08-13T16:45:03Z", "timestamp": "2026-08-13T16:45:03Z", "metrics": { "reactions": 0, "comments": 4 }, "labels": [ "feature request", "quantization" ], "author": "Tortoise17", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:284718cb68f262df2fec", "signalId": "github:vllm-project/vllm:pull_request:49381", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:49381", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method", "text": "## TL;DR ModelOpt linear quantization is implemented today as **six near-duplicate `LinearMethod` classes**, one per format (FP8 per-tensor, FP8 per-channel/per-token, FP8 block-weight-only, NVFP4 W4A4, NVFP4 W4A16, MXFP8). This PR replaces all six with **one generic `ModelOptLinearMethod`**, composed from per-`QuantKey` schemes and driven by a `QuantSpec(weight, activation)` pair. Adding or changing a format becomes *data* (a `resolve()` row + reusable schemes), not a new class. Behavior-preserving for existing checkpoints (byte-level parity + GSM8K on all six formats). **MoE and mixed-precision are untouched.** Two long-standing issues get fixed as a side effect of unifying. 📄 **Design doc:** https://docs.google.com/document/d/14ao-WqOeeMIi0XPvjXXm4baxD4xQm_Jb/edit ## Motivation The six per-format linear classes share ~80% of their structure but each re-implements `create_weights` / `process_weights_after_loading` / `apply` inline, with subtle per-format differences that are **load-bearing and fail silently** — a wrong assumption produces garbage output with no error. Concretely: - **Duplication.** NVFP4 W4A4 and W4A16 are two entire classes that differ only in whether activations are quantized. The three FP8 classes differ mainly in scale shape. - **Scattered kernel selection.** Some classes build their kernel in `__init__`, others in `create_weights` — no single place decides. - **Hidden residue.** Each class silently does its own transpose / requant / reshape, so the format-specific quirks are buried in imperative code and easy to break during unrelated edits. - **Hard to extend.** A new format means a new class, wired into every dispatch site (base config + mixed-precision) and the weight-loader-v2 list. The design question (detailed in the doc): **which parts of a quantized linear layer are derivable purely from the format's numeric description, and which genuinely need per-format code?** The answer is that almost everything is derivable from the `(weight, activation)` QuantKey pair; the true residue is small and can be named explicitly. ## Design ### Flow — before vs after **Before — six self-contained classes** (the config picks one; each re-implements the whole lifecycle): ```text checkpoint (quant_algo) │ ▼ ModelOpt*Config.__init__ ──picks──► self.LinearMethodCls │ ┌─────┬─────┬─────┼─────┬─────┬─────┐ ▼ ▼ ▼ ▼ ▼ ▼ FP8 PcPt PbWo NVFP4 W4A16 MXFP8 ← 6 *LinearMethod classes │ │ │ │ │ │ └─────┴─────┴─────┴─────┴─────┘ │ ▼ each class re-implements its OWN: create_weights · process_weights · apply · kernel-selection (~80% duplicated; per-format quirks inline & silent-failing) ``` **After — one method composed from shared schemes** (each lifecycle phase calls the weight scheme, then the activation scheme, in order): ```text checkpoint (quant_algo) │ ▼ ModelOpt*Config.linear_algo() → \"NVFP4\" │ ▼ resolve(algo) → QuantSpec(weight_key, activation_key) CkptCtx(serialized, group_size) │ ▼ ModelOptLinearMethod (one generic class) ├─ wkey = SCHEME_FOR[weight_key] └─ akey = SCHEME_FOR[activation_key] │ ▼ fixed 3-phase lifecycle — each phase runs wkey THEN akey, in order: create_weights(layer) 1. wkey.create_weights(WEIGHT) register weight + scales 2. akey.create_weights(ACT) register activation params (if any) 3. select_linear_kernel(spec) pick fp4 / mxfp8 / fp8 kernel 4. expose_input_quant_key process_weights_after_loading(layer) 1. wkey.process(WEIGHT) transpose / requant / reshape 2. akey.process(ACT) 3. maybe_fuse_global_scales alpha = weight_gs × input_gs (if both present) 4. kernel.process_weights_after_loading apply(layer, x) 1. kernel.apply_weights shared schemes: NVFP4 W4A4 & W4A16 → same KNvfp4Static (differ only by activation: kNvfp4Dynamic vs None) ``` **Dispatch identity is the `(weight_key, activation_key)` pair.** A ModelOpt linear format is fully described by two `QuantKey`s. `resolve(algo, config, prefix)` maps the checkpoint's algo string to a `QuantSpec(weight, activation)` plus a `CkptCtx(serialized, group_size)`. `resolve` is strictly read-only over the config. **`QuantKeyScheme` — one scheme per QuantKey, content-keyed and role-parameterized.** Each scheme knows how to allocate and post-process *its key* in either the weight or activation slot, and rejects any role it hasn't validated (no silent wrong-role fall-through). Schemes are shared across formats — e.g. NVFP4 W4A4 and W4A16 use the *same* `KNvfp4Static` weight scheme. Seven schemes cover all six formats. **One generic `ModelOptLinearMethod`** holds `(wkey, akey)` from the pair and runs a fixed lifecycle: - `create_weights` → `wkey.create_weights(WEIGHT)` → `akey.create_weights(ACT)` → `select_linear_kernel(spec)` → expose input-quant-key - `process_weights_after_loading` → `wkey.process` → `akey.process` → `maybe_fuse_global_scales` → `kernel.process_weights_after_loading` - `apply` → one line to the kernel **`select_linear_kernel(spec, …)`** picks the kernel family from the weight key (fp4 / mxfp8 / fp8) in a single place. **Front-end.** Each config now exposes `linear_algo()` (just the algo string), and `get_quant_method`'s `LinearBase` arm calls `build_linear_method(config, algo, prefix)`. The config no longer names a method class — it names a format, and the behavior is *derived*. **Only three things don't fit a single key** (everything else composes): the cross-key `alpha = weight_gs × input_gs` (`maybe_fuse_global_scales`), the W4A16 placeholder-`input_scale` drop, and the MXFP8 idempotency guard. **Two extension seams** so future formats have tools, not forks: - **`FormatScheme`** — optional per-format hooks (`extra_weights` / `pre_process` / `post_process`) that compose *around* the key schemes, for residue that belongs to the layer as a whole. Default no-op. - **`LINEAR_METHOD_BUILDERS`** — a registry (empty by default) so a format that genuinely can't be a `(weight, activation)` pair registers its own `LinearMethodBase` by algo, via the single `build_linear_method` indirection point — without re-growing per-format branching in either dispatch arm. **MoE and mixed-precision are unchanged.** MoE keeps its per-format methods (the `ModelOpt*` class-name coupling that `routed_experts.py` gates on is preserved), and the mixed-precision MoE fork is byte-identical. Only the linear arm changed; a mixed checkpoint runs new-linear beside old-MoE with no new plumbing. ## Behavior preservation + two intended fixes Behavior-preserving for all six formats (evidence below). Two deltas fall out of unifying, both improvements: - **FP8 PbWo bug fix.** The old class skipped the block kernel's post-load via a misnamed guard, so under compiled serving it ran the GEMM on un-repacked weights → **garbage** (acc 0.000, invalid-rate 1.0). The generic method runs the post-load unconditionally → correct (0.897). (Note: old PbWo works under *eager*, so this only surfaces in real compiled serving.) - **MXFP8 → weight_loader_v2.** MXFP8 was the one linear format missing from `WEIGHT_LOADER_V2_SUPPORTED` (v1 loader); the generic method self-registers for v2. Proven byte-identical. ## W4A16 NVFP4 checkpoints that carry an on-disk `input_scale` (called out loud) A `W4A16_NVFP4` (weight-only) checkpoint can still ship activation `input_scale` tensors on disk — e.g. a W4A4 checkpoint relabeled `W4A16_NVFP4`, or scales injected by a downstream tool. Because `resolve(\"W4A16_NVFP4\")` sets `activation=None`, no `input_scale` param is registered and the on-disk tensor is orphaned. **This PR drops the orphan on load.** Simple linears (`down_proj`, `o_proj`) skip the unmatched key via `_ignore_unexpected_suffixes`; merged linears (`qkv_proj`, `gate_up_proj`) load shards through `LinearBase.load_weights`, which previously looked the param up unconditionally and crashed — now guarded by a `param is self` skip (the one shared-`linear.py` touch here). Native W4A16 (no scale on disk) is unaffected. Verified: a qwen3 W4A16 checkpoint carrying 108 `input_scale` loads clean weight-only (Marlin) with no behavior change vs the already-validated weight-only path. > A commit earlier on this branch also explored two follow-ons — a Scope-A > `_DropInputScale` FormatScheme (register a placeholder, then discard) as an alternative > to the load-time drop, and an `activation_dtype: nvfp4` config override that runs such a > checkpoint as *pure W4A4* (consuming the `input_scale`; validated at GSM8K 0.89310). > Both are intentionally **excluded from this PR's diff** and left for separate proposals; > the tip carries only the drop above. ## Adding a format (developer guide, in-code) - **Composes as a `(weight, activation)` pair** → add a `QuantKeyScheme` per new key to `SCHEME_FOR` + a `resolve()` row. No new method class. (How all six formats are built.) - **Needs format-wide residue, same lifecycle** → also return a `FormatScheme` from that `resolve()` row. - **Genuinely can't be a key pair** → write a `LinearMethodBase` and register it in `LINEAR_METHOD_BUILDERS` by algo. ## Testing Kept current with upstream `main` by rebasing; `modelopt.py` and `linear.py` have had no upstream changes, so the cutover applies cleanly. **Unit** — `pytest tests/quantization/test_modelopt.py -k \"not checkpoint_setup\"`: **20 passed.** Dispatch/config tests updated to assert the resolved `QuantSpec` + `ModelOptLinearMethod` (instead of the old classes), a new builder-registry test for the extension seam, and the existing upstream mixed-precision/mapper tests. **Lint**: `ruff check` + `ruff format --check` (0.14.0) clean on all changed files. **Per-format evidence** — Qwen3-8B. *Parity* is the deterministic gate (`VLLM_BATCH_INVARIANT=1` + `--enforce-eager`): per-layer weight/kernel/expose byte-hash + per-token prefill logit-diff (old vs generic) against a proven-`0.0` old-vs-old baseline. *GSM8K* (1319 Q · 5-shot · T=0 · 512 tok, batch-invariant) is the coarse end-to-end check. | Format | weight/kernel/expose hash | logit Δ | GSM8K generic | GSM8K old | Verdict | |---|---|---|---:|---:|---| | NVFP4 W4A4 | byte-identical | `0.0` | 0.89765 | 0.89765 | ✅ identical | | W4A16 NVFP4 | byte-identical | `0.0` | 0.87794 | 0.87642 | ✅ identical (+2 Q autotune) | | FP8 per-tensor | byte-identical | `0.0` | 0.88476 | 0.88628 | ✅ identical (−2 Q autotune) | | FP8 PcPt | byte-identical | `0.0` | 0.89689 | 0.89689 | ✅ identical | | MXFP8 | byte-identical | `0.0` | 0.89538 | 0.89538 | ✅ identical | | FP8 PbWo | weights differ ¹ | `0.0` (eager) | 0.89689 | **0.00000** | ✅ **bug fix** | Logits are **byte-identical for all six** → behavior-preserving. GSM8K deltas are ≤2 Q (torch.compile cross-load autotune noise, not math — weights are provably identical), invalid-rate 0. NVFP4 was re-confirmed **after rebasing onto current main** (`0.89765`, bit-identical) — upstream changes don't affect the cutover's output. ¹ PbWo stored weights differ *by design*: the generic method runs the block-kernel post-load the old class skipped (a misnamed-guard bug). Output is identical under eager; under real compiled serving the old class produces garbage (`0.000`, invalid-rate 1.0) while the generic path is correct — this is the fix noted above. ## Follow-up (not in this PR) To keep this diff reviewable, `modelopt.py` stays a single file. A natural next step is to split it into a package by tier — schemes/method vs MoE: ```text modelopt/ ├── configs.py # base + Fp8/NvFp4/MxFp8/Mixed configs (+ linear_algo) ├── kv_cache.py ├── linear/ # types.py · schemes.py · kernels.py · method.py │ # (ModelOptLinearMethod, FormatScheme, resolve, builders) └── moe/ # fp8.py · nvfp4.py · mxfp8.py ``` Constraints: keep the `ModelOpt*` class names (`routed_experts.py` coupling) and re-export the config classes from `modelopt/__init__.py` for import stability. ## Not duplicating existing work This **replaces** the in-tree ModelOpt linear methods; it is not a parallel implementation. #48422 targets the FP8_PB_WO post-load bug specifically; this PR fixes it as a consequence of unifying the linear methods. If #48422 merges first, the PbWo behavior here becomes a no-op vs the fixed class and we rebase accordingly. #50617 (serve `modelopt_mixed` checkpoints with FP8_PB layers) edits the same `modelopt.py` in the pre-refactor structure, so it will conflict; whichever lands first, the other rebases. Its two fixes — `has_blocked_weights()` and loading the 2-D `weight_scale_inv` block scale — are complementary and will be carried into the generic path on rebase. ## Notes - `mypy` currently hits an environmental internal-crash in the dev container (unrelated to this diff; reproduces with the changes stashed). CI's `mypy-3.12` hook runs it properly. ## AI assistance AI assistance (Claude Code) was used for this change; the diff and results were human-reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49381", "createdAt": "2026-07-21T23:53:04Z", "updatedAt": "2026-08-13T16:42:50Z", "timestamp": "2026-08-13T16:42:50Z", "metrics": { "reactions": 2, "comments": 18 }, "labels": [ "ready", "quantization", "verified" ], "author": "juhi10071998", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:93d066e8f69073425228", "signalId": "github:vllm-project/vllm:pull_request:49585", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49585", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[EC Connector] Added Build Connector Worker Meta for EC Connector", "text": "## Purpose **Why this is still needed after #38390.** The PR implemented the V2 model runner EC Connector, but EC still has no worker -> scheduler metadata channel. KV connectors have a complete one: `build_connector_worker_meta()` -> `KVConnectorOutput.kv_connector_worker_meta` -> `KVOutputAggregator` -> scheduler-side KV connector. EC had none of those three pieces, so a worker-side`ECConnector` has no way to report per-step state — e.g. \"this `mm_hash` is now durably stored\" — back to its scheduler-side EC connector. That blocks the CPU-offload based on CUDA events and NIXL P2P EC connectors, which need exactly this report (We are retiring the StepTracker in the CPU EC Connector, because using CUDA events is the right way to handle \"copy done\" notifications, instead of waiting until the batch is done). This PR adds that channel, and fixes the exit paths where the worker's `ec_connector_output` was computed and then dropped before reaching the scheduler. ### The channel - `ec_connector/base.py`: `ECConnectorWorkerMetadata` ABC with an abstract `aggregate()`, plus `ECConnectorBase.build_connector_worker_meta()`. - `v1/outputs.py`: `ec_connector_worker_meta` field on `ECConnectorOutput`. - `ec_connector/utils.py`: `ECOutputAggregator`, mirroring `KVOutputAggregator` - merges the EC output of the rank that ran the connector onto the rank whose `ModelRunnerOutput` reaches the scheduler. - `engine/core.py`, `executor/abstract.py`, `executor/multiproc_executor.py`: wiring so the aggregator runs when an EC connector is configured. - Both runners (MRv1, MRv2) populate the field via `build_connector_worker_meta()`. Default behavior is unchanged for existing connectors: nothing overrides `build_connector_worker_meta()` yet, so it returns `None`. ## Test Plan ```bash # Existing EC connector unit tests (unaffected areas) .venv/bin/python -m pytest tests/v1/ec_connector/unit/test_metadata.py \\ tests/v1/ec_connector/unit/test_ec_example_connector.py \\ tests/v1/ec_connector/unit/cpu/test_connector.py -v ## Test Result ``` tests/v1/ec_connector/unit/test_metadata.py ..................... PASSED tests/v1/ec_connector/unit/test_ec_example_connector.py ......... PASSED tests/v1/ec_connector/unit/cpu/test_connector.py ................ PASSED 36 passed, 2 warnings pre-commit: ruff check / ruff format / mypy / SPDX headers / etc. — all Passed ```", "url": "https://github.com/vllm-project/vllm/pull/49585", "createdAt": "2026-07-23T12:33:50Z", "updatedAt": "2026-08-13T16:42:37Z", "timestamp": "2026-08-13T16:42:37Z", "metrics": { "reactions": 2, "comments": 7 }, "labels": [ "documentation", "ready", "ci/build", "v1", "kv-connector", "mrv2" ], "author": "omerpaz95", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:888d8b78c29fa501c164", "signalId": "github:vllm-project/vllm:pull_request:51886", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51886", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[KVConnector] Add retention interval to OffloadingConnector", "text": "## Purpose Add retention interval support to `OffloadingConnector`. The Mooncake KV connector supports retention interval support via the `envs.VLLM_PREFIX_CACHE_RETENTION_INTERVAL` env variable. It is used to avoid storing/loading blocks that are outside of the sliding window of the attention mechanism. This is implemented via block masks. The implementation in `OffloadingConnector` uses the same block mask mechanism to implement retention interval support. cc @varun-sundar-rabindranath Note: parts of the code generated with claude ## Test Plan - Run kv_offloading unit tests - Add new unit tests - run gptoss w/retention interval enabled. ## Test Result gpt-oss-20b results ``` Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-20b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":4294967296,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-20b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.315 │ 0.185 │ 36.5 │ 1284.8 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.305 │ 0.145 │ 33.4 │ 1434.2 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.325 │ 0.170 │ 25.2 │ 1840.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.325 │ 0.160 │ 29.8 │ 1577.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.345 │ 0.200 │ 22.3 │ 2138.7 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.320 │ 0.165 │ 28.1 │ 1678.6 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` gpt-oss-120b results ``` ● gpt-oss-120b — Retention Interval Eval Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve openai/gpt-oss-120b \\ --enforce-eager \\ --max-model-len 4096 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 5 \\ --host http://localhost \\ --port 8001 Results — gpt-oss-120b (GSM8K, 200 questions, 5-shot) ┌──────────────────────────┬────────┬──────────┬──────────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Invalid Rate │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.540 │ 0.145 │ 58.9 │ 725.4 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.540 │ 0.100 │ 51.7 │ 851.9 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ fresh │ 0.575 │ 0.085 │ 57.0 │ 769.3 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Sparse (interval=1024) │ reload │ 0.555 │ 0.080 │ 50.9 │ 870.1 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.570 │ 0.095 │ 40.3 │ 1059.6 │ ├──────────────────────────┼────────┼──────────┼──────────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.570 │ 0.080 │ 50.9 │ 851.3 │ └──────────────────────────┴────────┴──────────┴──────────────┴─────────────┴──────────┘ ``` DeepSeekV4 results showing reduction in memory stores for different intervals. ``` DeepSeek-V4-Flash (GSM8K, 200 questions, 8-shot, max_model_len=16384) Server: VLLM_SERVER_DEV_MODE=1 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=${INTERVAL} \\ vllm serve deepseek-ai/DeepSeek-V4-Flash \\ --enforce-eager \\ --max-model-len 16384 \\ --enable-prefix-caching \\ --no-disable-hybrid-kv-cache-manager \\ --kv-transfer-config '{\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_conf ig\":{\"spec_name\":\"CPUOffloadingSpec\",\"cpu_bytes_to_use\":17179869184,\"eviction_policy\":\"lru\"}}' \\ --trust-remote-code \\ --port 8001 \\ --disable-uvicorn-access-log \\ --tensor-parallel-size 4 \\ --enable-expert-parallel \\ --kv-cache-dtype fp8 \\ --block-size 256 Where INTERVAL is unset (baseline), 512 (sparse), or 0 (most sparse). Eval: python tests/evals/gsm8k/gsm8k_eval.py \\ --num-questions 200 \\ --num-shots 8 \\ --host http://localhost \\ --port 8001 Accuracy: ┌──────────────────────────┬────────┬──────────┬─────────────┬──────────┐ │ Configuration │ Run │ Accuracy │ Latency (s) │ Tokens/s │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ fresh │ 0.960 │ 48.5 │ 371.1 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Baseline (no interval) │ reload │ 0.970 │ 53.0 │ 345.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ fresh │ 0.965 │ 54.3 │ 331.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Sparse (interval=512) │ reload │ 0.955 │ 47.9 │ 380.6 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ fresh │ 0.970 │ 52.6 │ 343.0 │ ├──────────────────────────┼────────┼──────────┼─────────────┼──────────┤ │ Most sparse (interval=0) │ reload │ 0.960 │ 46.3 │ 389.6 │ └──────────────────────────┴────────┴──────────┴─────────────┴──────────┘ Prometheus Offload Metrics: ┌──────────────────────────┬─────────────────────────┬───────────────────────┐ │ Configuration │ Store Bytes │ Reduction vs Baseline │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Baseline (no interval) │ 4,610,304,000 (4.61 GB) │ — │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (interval=512) │ 192,430,080 (192 MB) │ 95.8% │ ├──────────────────────────┼─────────────────────────┼───────────────────────┤ │ Most sparse (interval=0) │ 104,232,960 (104 MB) │ 97.7% │ └──────────────────────────┴─────────────────────────┴───────────────────────┘ ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/51886", "createdAt": "2026-08-11T23:11:05Z", "updatedAt": "2026-08-13T16:42:21Z", "timestamp": "2026-08-13T16:42:21Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "kv-connector" ], "author": "bnellnm", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:ecc36151518e07a59cbe", "signalId": "github:vllm-project/vllm:pull_request:48556", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48556", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Bump astral-sh/setup-uv from 7.6.0 to 9.0.0", "text": "Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 9.0.0.
Release notes

Sourced from astral-sh/setup-uv's releases.

v9.0.0 🌈 Change prune-cache default to false

Changes

This release disables the default cache cache pruning to ease the load on the PyPi infrastructure. Since users might experience more GitHub Actions cache usage which might result in higher costs this is marked as a breaking change. To read more on why we did this (now) you can read the detailed analysis and reasoning in #967

Besides this big breaking change we also have a small bugfix while building caches for linux distributions that behave a big different than the "big ones" and a speed up in version resolution by only reading the version manifest until a matching version is found saving runtime and network bandwith.

🚨 Breaking changes

🐛 Bug fixes

  • fix: fall back to distribution ID when os-release has no version field @​cxzhong (#961)

🚀 Enhancements

🧰 Maintenance

📚 Documentation

⬆️ Dependency updates

v8.2.0 🌈 New inputs quiet and download-from-astral-mirror

Changes

This release brings two new inputs and a few bug fixes.

New inputs

Lets talk about the new inputs first.

quiet

Pretty simple. It turns of all info loggings. Useful if you use this in a composite action and are not interested in all the details. In the upcoming releases we will add log groups to fully implement support for "less noise"

[!NOTE]
Warnings and errors are always logged.

... (truncated)

Commits
  • c771a70 chore(deps): roll up Dependabot updates (#970)
  • 2f537ca chore: update known checksums for 0.11.30 (#968)
  • 2269552 Speed up version client by partial response reads (#807)
  • 47a7f4f Change prune-cache default to false (#967)
  • 71966ef chore(deps): roll up Dependabot updates (#962)
  • f12b1f0 fix: fall back to distribution ID when os-release has no version field (#961)
  • ecd24dd chore: update known checksums for 0.11.29 (#960)
  • 6a19136 docs: update version references to v8.3.2 (#949)
  • 11f9893 chore: roll up Dependabot updates (#948)
  • f798556 docs: update version references to v8.3.1 (#946)
  • Additional commits viewable in compare view

> **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days.", "url": "https://github.com/vllm-project/vllm/pull/48556", "createdAt": "2026-07-14T04:16:57Z", "updatedAt": "2026-08-13T16:41:44Z", "timestamp": "2026-08-13T16:41:44Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "ci/build", "github_actions", "dependencies" ], "author": "dependabot[bot]", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:3494129942a6ec000b5f", "signalId": "github:vllm-project/vllm:pull_request:51714", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51714", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][DSV4] Opt-in AITER gluon kernel for sparse-MLA decode on gfx950", "text": "## Purpose Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair (`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). ## Changes Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): - `VLLM_ROCM_DSV4_SPARSE_GLUON` (**default off**). Read once at import — this sits on the per-step decode path, so re-reading the environment per call would be pure overhead. With the flag unset the decode path is byte-for-byte unchanged. - The gluon entrypoint is resolved lazily behind `functools.cache`, so builds whose AITER lacks the kernel are unaffected. - A failed launch latches a process-wide flag and degrades to Triton for the rest of the process, rather than taking the server down mid-request. - The first successful dispatch logs once, so enabling the flag is observable rather than silent. - Additionally gated on `_ON_GFX950`. ## Requires ROCm/aiter#4673 The kernel needs [ROCm/aiter#4673](https://github.com/ROCm/aiter/pull/4673) to be correct on large KV pools. Without it the gluon driver decides `use_buffer_load` from `nelement() * element_size()`, which undercounts a strided pool whose addressable span exceeds a 32-bit offset; past 2 GiB the `buffer_load` offsets wrap and the gather silently returns zeros. An earlier revision of this PR refused to run under DP-attention, citing [ROCm/aiter#4667](https://github.com/ROCm/aiter/issues/4667) (gsm8k 1.00 → 0.30 on EP8 + DP-attention). That guard has been removed: #4667 is the overflow above, and DP-attention was only ever the thing that grew the pool far enough to cross the boundary — not a property of the kernel. Gating on `data_parallel_size` therefore disabled the kernel on exactly the configuration it was written for. With #4673 applied, DP-attention runs clean. ## Test Plan - DeepSeek-V4 on MI355X (gfx950) with `VLLM_ROCM_DSV4_SPARSE_GLUON=0` and `=1`, comparing decode outputs for parity and measuring decode throughput. - Cover both `swa_only` (single-segment) and two-loop layers. - Confirm the fallback latch by forcing a launch failure. - Cover DP-attention (DEP8) with aiter#4673 applied. ## Test Result | Concurrency | Output tok/s (base → +PR) | Δ | Median TPOT (base → +PR) | Δ | |---|---|---|---|---| | 8 | 284.79 → 291.33 | **+2.30%** | 24.67 → 24.16 ms | −2.04% | | 16 | 483.69 → 488.59 | **+1.01%** | 29.32 → 29.02 ms | −1.03% | | 32 | 708.51 → 712.89 | **+0.62%** | 40.78 → 40.52 ms | −0.65% | | 64 | 955.23 → 958.04 | **+0.29%** | 62.55 → 62.36 ms | −0.30% | ### AgentX agentic replay (fast mode, TP=8, concurrency 32, MTP on) | Metric | base (Triton) | +PR (gluon) | Δ | |---|---|---|---| | Output tok/s | 430.76 | 453.23 | **+5.22%** | | Output tok/s/GPU | 53.84 | 56.65 | **+5.22%** | | Per-user tok/s | 32.72 | 39.96 | **+22.12%** | | ITL avg | 39.83 ms | 34.60 ms | **−13.11%** | | ITL p50 | 35.87 ms | 26.44 ms | **−26.30%** | | Time-to-2nd-token p50 | 38.19 ms | 21.79 ms | **−42.95%** | | TTFT p50 | 656.7 ms | 700.6 ms | +6.68% (regression) | ### DP-attention, with aiter#4673 applied DEP8 (DP-attention + EP8) + MegaMoE + gluon, MI355X, agentic replay at concurrency 96: **12,244 tok/s/GPU**. All eight DP ranks log the gluon path engaging. This is the configuration the removed guard used to refuse.", "url": "https://github.com/vllm-project/vllm/pull/51714", "createdAt": "2026-08-10T18:00:26Z", "updatedAt": "2026-08-13T16:40:41Z", "timestamp": "2026-08-13T16:40:41Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "rocm" ], "author": "jiacao-amd", "state": "open", "assignees": [ "shen-shanshan" ], "change": "updated" } }, { "id": "event:70d92e5cf47567a78c33", "signalId": "github:vllm-project/vllm:pull_request:52091", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Auto-ping Cohere on related issues", "text": "## Purpose As discussed offline ## Test Plan ## Test Result ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52091", "createdAt": "2026-08-13T03:37:59Z", "updatedAt": "2026-08-13T16:40:39Z", "timestamp": "2026-08-13T16:40:39Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready", "ci/build" ], "author": "DarkLight1337", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:c254aa60559186b97e6a", "signalId": "github:vllm-project/vllm:pull_request:52188", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52188", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec decode] Support Kimi-K3 DCP with DSpark", "text": "## Purpose This PR adds support for running Kimi-K3 decode context parallel with DSpark with FlashinferMLA and Tokenspeed as target causal attention backend and Tokenspeed as the draft non-causal backend. ## Test Plan Kimi K3 GSM8k with the different backend combination. ## Test Result Default (no backend specified): ``` vllm serve moonshotai/Kimi-K3 \\ --tensor-parallel-size 8 \\ -dcp 8 \\ --load-format fastsafetensors \\ --no-enable-flashinfer-autotune \\ --trust-remote-code \\ --language-model-only \\ --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\ --kv-cache-dtype fp8 \\ --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"rejection_sample_method\":\"standard\"}' |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9606|± |0.0054| | | |strict-match | 5|exact_match|↑ |0.9606|± |0.0054| ``` FlashinferMLA target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9613|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9613|± |0.0053| ``` Tokenspeed target + Tokenspeed draft: ``` |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9621|± |0.0053| | | |strict-match | 5|exact_match|↑ |0.9621|± |0.0053| ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52188", "createdAt": "2026-08-13T16:10:03Z", "updatedAt": "2026-08-13T16:39:35Z", "timestamp": "2026-08-13T16:39:35Z", "metrics": { "reactions": 2, "comments": 0 }, "labels": [ "nvidia", "mrv2", "kimi", "k3" ], "author": "wzhao18", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6b0a016416c60df366b1", "signalId": "github:vllm-project/vllm:pull_request:48040", "event": "discovered", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:48040", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] test_batch_inference_correctness now uses batch invariance", "text": "## Purpose Partially Fixes [48038](https://github.com/vllm-project/vllm/issues/48038). An example PR fixing a test flagged by our audit. The test now enforces batch invariance to prevent accidental numerical changes causes falsely failing tests. The test is no longer coincidentally correct and enforces a strong contract: Spec decode exact matching only when the test explicitly forces batch-invariant mode/kernels. ## Test Plan N/A ## Test Result The test still passes with batch in variance turned on. ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/48040", "createdAt": "2026-07-08T20:17:50Z", "updatedAt": "2026-08-13T16:29:46Z", "timestamp": "2026-08-13T16:29:46Z", "metrics": { "reactions": 0, "comments": 2 }, "labels": [ "bug", "v1" ], "author": "morrison-turnansky", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:96dc33f602d9dc54956c", "signalId": "github:vllm-project/vllm:pull_request:50082", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50082", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Add Kimi K3 MoE support to benchmark_moe.py", "text": "## Purpose `benchmarks/kernels/benchmark_moe.py` cannot tune the fused MoE Triton kernel for Kimi K3. `get_model_params()` does not recognize `KimiK3ForConditionalGeneration`, so it falls through to the Mixtral default branch and raises `AttributeError` (Kimi's config has no `num_local_experts`). Kimi K3 is multimodal and stores its MoE parameters in a nested `KimiLinearConfig` `text_config`, and it exposes the routing top-k as `num_experts_per_token` rather than the more common `num_experts_per_tok` used by the existing handlers. This adds a dedicated branch, following the same pattern as the Qwen3.5 handler added in #37068. It also covers the text-only `KimiLinearForCausalLM`, for which `get_text_config()` returns the config itself. ## Test Plan Parse the released Kimi K3 config and confirm the extracted MoE parameters: ```python from vllm.transformers_utils.config import get_config from benchmark_moe import get_model_params cfg = get_config(\"\", trust_remote_code=True) print(get_model_params(cfg)) ``` ## Test Result ``` architectures = ['KimiK3ForConditionalGeneration'] (text_config: KimiLinearConfig) get_model_params -> (E=896, topk=16, moe_intermediate_size=3072, hidden_size=7168) ``` The values match the model config. With `--tp-size 8 --enable-expert-parallel` this yields `E_local=112` and shard `N=6144`. Before this change the script raised `AttributeError` before tuning could start.", "url": "https://github.com/vllm-project/vllm/pull/50082", "createdAt": "2026-07-28T04:27:24Z", "updatedAt": "2026-08-13T16:28:14Z", "timestamp": "2026-08-13T16:28:14Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "performance", "ready", "kimi", "k3" ], "author": "vanshbhatia-amd", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8ac0b24663bdc5239393", "signalId": "github:vllm-project/vllm:pull_request:43249", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:43249", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Fix MRV2 Gumbel sampling for non-finite logits", "text": "## Summary Fix MRV2 Gumbel sampling when the candidate logits contain `-inf` entries. The current Gumbel path only clamps the uniform draw away from zero. If the draw reaches the upper endpoint after floating-point rounding, the Gumbel noise becomes `inf`. Adding that noise to a masked/non-candidate `-inf` logit produces `NaN`, and a subsequent Triton reduction can return an invalid token index. This change: - clamps fp32/fp64 uniform draws away from both `0` and `1`; - only adds Gumbel noise to finite logits; - adds a CUDA/Triton regression test that deterministically reproduces the `-inf + inf -> NaN` edge case. ## Why this matters We hit this downstream in a Kimi K2.6 MRV2 speculative decoding deployment with DCP8 and Eagle/MTP. The failure surfaced during spec-decode/logprobs warmup as an invalid sampled token id and then a CUDA illegal memory access: ```text Invalid sampled token ids for spec logprobs: count=1 vocab=163840 idx=[508] vals=[9223372034707292159] ``` The DCP8/Kimi setup makes the bug easy to trigger, but the root cause is not DCP-specific. It is the generic Gumbel sampler being asked to sample from residual logits where rejected tokens are represented as `-inf`. A minimal deterministic reproduction of the arithmetic issue is: ```python logits = -inf u = 1.0 gumbel_noise = -log(-log(u)) # inf logits + gumbel_noise # NaN ``` After this patch, the same case stays `-inf` and does not poison the reduction. ## Tests ```text python3 - <<'PY' import importlib.util path = \"tests/v1/sample/test_gumbel.py\" spec = importlib.util.spec_from_file_location(\"test_gumbel_direct\", path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(False) mod.test_gumbel_noise_does_not_turn_negative_inf_logits_into_nan(True) print(\"direct test passed\") PY python3 -m py_compile vllm/v1/worker/gpu/sample/gumbel.py tests/v1/sample/test_gumbel.py git diff --check ``` I could not run the repository pytest entrypoint in this local container because `tests/conftest.py` imports `tblib`, which is not installed in the environment: ```text ModuleNotFoundError: No module named 'tblib' ```", "url": "https://github.com/vllm-project/vllm/pull/43249", "createdAt": "2026-05-20T20:58:39Z", "updatedAt": "2026-08-13T16:25:59Z", "timestamp": "2026-08-13T16:25:59Z", "metrics": { "reactions": 1, "comments": 1 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "voipmonitor", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:3471cf58cccc1ea41c8f", "signalId": "github:vllm-project/vllm:pull_request:43091", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:43091", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Model Runner V2][Spec Decode] Support spec decode with draft model", "text": "## Purpose Support spec decode with draft models in model runner v2 ## Test Plan ``` python examples/features/speculative_decoding/spec_decode_offline.py \\ --method draft_model \\ --model-dir /shared/models/modelscope/models/Qwen/Qwen3-32B \\ --draft-model /shared/models/modelscope/models/Qwen/Qwen3-1.7B \\ --num-spec-tokens 3 \\ --dataset-name hf \\ --dataset-path philschmid/mt-bench \\ --num-prompts 400 \\ --temp 0 \\ --output-len 256 \\ --max-model-len 10000 ``` ## Test Result **Model Runner V1:** ``` -------------------------------------------------- total_num_output_tokens: 102355 num_drafts: 36593 num_draft_tokens: 109779 num_accepted_tokens: 65181 mean acceptance length: 2.78 -------------------------------------------------- acceptance at token 0: 0.75 acceptance at token 1: 0.57 acceptance at token 2: 0.45 ``` **Model Runner V2:** ``` -------------------------------------------------- total_num_output_tokens: 102312 num_drafts: 36394 num_draft_tokens: 109182 num_accepted_tokens: 65379 mean acceptance length: 2.80 -------------------------------------------------- acceptance at token 0: 0.76 acceptance at token 1: 0.58 acceptance at token 2: 0.46 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/43091", "createdAt": "2026-05-19T11:35:21Z", "updatedAt": "2026-08-13T16:25:24Z", "timestamp": "2026-08-13T16:25:24Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "ready", "needs-rebase", "ci/build", "v1", "mrv2" ], "author": "wxsIcey", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:501e9e50817daa8fd16a", "signalId": "github:vllm-project/vllm:pull_request:42963", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:42963", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ModelRunnerV2] Support prompt embeds", "text": "## Purpose Support prompt embeds for ModelRunnerV2. ## Test Plan ```bash VLLM_USE_V2_MODEL_RUNNER=1 pytest -sv tests/basic_correctness/test_basic_correctness.py::test_models -k \"True-uni or True-mp\" ``` Before ```bash E pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig E Value error, VLLM_USE_V2_MODEL_RUNNER does not yet support: prompt embeds [type=value_error, input_value=ArgsKwargs((), {'model_co... 'shutdown_timeout': 0}), input_type=ArgsKwargs] E For further information visit https://errors.pydantic.dev/2.13/v/value_error vllm/engine/arg_utils.py:2171: ValidationError ============================================================ warnings summary ============================================================ :488 :488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute :488 :488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute ../.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings /root/vllm-workspace/.venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================================================== short test summary info ========================================================= FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-uni-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-True-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-hmellor/tiny-random-Gemma2ForCausalLM] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig FAILED tests/basic_correctness/test_basic_correctness.py::test_models[True-mp-False-False-5-FLASH_ATTN-meta-llama/Llama-3.2-1B-Instruct] - pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig ======================================== 8 failed, 8 deselected, 16 warnings in 237.83s (0:03:57) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` After ```bash ======================================== 8 passed, 8 deselected, 16 warnings in 443.73s (0:07:23) ======================================== sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ``` vllm serve --enable-prompt-embeds ``` ``` \"\"\"Smoke test for prompt_embeds over the OpenAI-compatible HTTP server. Usage: # Terminal 1 (server): see vllm serve command in the chat. # Terminal 2: .venv/bin/python test_embeds_serve.py \"\"\" import io import openai import pybase64 as base64 import torch from transformers import AutoModelForCausalLM, AutoTokenizer MODEL_PATH = \"/root/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B\" SERVED_NAME = \"Qwen/Qwen3-0.6B\" # must match --served-model-name BASE_URL = \"http://localhost:8000/v1\" PROMPT = \"The capital of France is\" def to_b64_embed(tensor: torch.Tensor) -> str: buf = io.BytesIO() # torch.save is what the server expects (it calls torch.load on the bytes). torch.save(tensor, buf) return base64.b64encode(buf.getvalue()).decode(\"utf-8\") def main() -> None: tok = AutoTokenizer.from_pretrained(MODEL_PATH) hf = AutoModelForCausalLM.from_pretrained(MODEL_PATH, torch_dtype=torch.bfloat16) with torch.no_grad(): ids = tok(PROMPT, return_tensors=\"pt\").input_ids embeds = ( hf.get_input_embeddings()(ids) .squeeze(0) .to(torch.bfloat16) .cpu() .contiguous() ) del hf encoded = to_b64_embed(embeds) client = openai.OpenAI(base_url=BASE_URL, api_key=\"EMPTY\") # Case 1: prompt_embeds only. out = client.completions.create( model=SERVED_NAME, prompt=None, # leave empty so the server falls through to prompt_embeds max_tokens=16, temperature=0.0, extra_body={\"prompt_embeds\": encoded}, ) print(f\"[prompt_embeds] {out.choices[0].text!r}\") # Case 2: same prompt via text path, for sanity comparison. out_text = client.completions.create( model=SERVED_NAME, prompt=PROMPT, max_tokens=16, temperature=0.0, ) print(f\"[text ] {out_text.choices[0].text!r}\") if out.choices[0].text == out_text.choices[0].text: print(\"MATCH: prompt_embeds output equals text output\") else: print(\"DIVERGE: outputs differ (expected only if tokenizer/embedding \" \"scaling differs from raw lookup)\") if __name__ == \"__main__\": main() ``` ## Test Result ``` python test_embeds_serve.py [transformers] `torch_dtype` is deprecated! Use `dtype` instead! Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████| 311/311 [00:00<00:00, 6652.97it/s] [prompt_embeds] ' Paris. The capital of France is also the capital of the Republic of France.' [text ] ' Paris. The capital of France is also the capital of the Republic of France.' MATCH: prompt_embeds output equals text output ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/42963", "createdAt": "2026-05-18T10:04:55Z", "updatedAt": "2026-08-13T16:25:01Z", "timestamp": "2026-08-13T16:25:01Z", "metrics": { "reactions": 1, "comments": 13 }, "labels": [ "needs-rebase", "v1", "mrv2" ], "author": "gcanlin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:724d1f9aff77320612b9", "signalId": "github:vllm-project/vllm:pull_request:52171", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52171", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Declare SupportsEagle3 on KimiLinearForCausalLM", "text": "### Purpose `KimiK3ForConditionalGeneration` (multimodal) declares `SupportsEagle3`; the text-only `KimiLinearForCausalLM` does not — even though both serve the same inner `KimiLinearModel`, which already inherits `EagleModelMixin` and implements the aux-hidden-state tap machinery. Serving a text-only Kimi-K3 checkpoint with EAGLE3-family speculative decoding (e.g. `dspark`) therefore dies at startup: ``` RuntimeError: Model does not support EAGLE3 interface ``` Adding the interface to the class bases is the whole fix: the protocol's default `set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` delegate to `self.model`, which satisfies their `EagleModelMixin` requirement. ### Test plan - `tests/models/kimi_k3/test_eagle3.py` gains `test_kimi_linear_advertises_eagle3_support`, mirroring the existing multimodal assertion; it fails on `main` and passes with this change. - Runtime-validated: a text-only Kimi-K3 checkpoint served with a `dspark` draft on 8× RTX 3090 using an equivalent patch (that is how we hit the error). --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52171", "createdAt": "2026-08-13T13:55:11Z", "updatedAt": "2026-08-13T16:22:51Z", "timestamp": "2026-08-13T16:22:51Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "ready", "verified", "kimi", "k3" ], "author": "nickus", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:129203b08ef65f33cd48", "signalId": "github:vllm-project/vllm:pull_request:51459", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51459", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[CI] Fix and extend PR/issue auto-labeling", "text": "## Purpose Auditing the labeling setup against the actual repo turned up rules that had stopped matching the code they were written for, plus issue-side labeling that covered only 6 of 62 labels. ### Rules that were missing real work - **`deepseek` missed the `vllm/models/` tree.** Newer models live in `vllm/models//` rather than `vllm/model_executor/models/`. The rule only knew the old location, so only the title match applied — **64 of 99 PRs** touching `vllm/models/deepseek_v4/` went unlabeled. Same gap for `deepseek_v32`. - **`speculative-decoding` missed the Model Runner V2 subtree.** `vllm/v1/worker/gpu/spec_decode/` has seen 94 PRs in 6 months, more than the original `vllm/v1/spec_decode/` at 81. This also folds `dflash` and `dspark` under the umbrella label, per review feedback that both are spec decoding techniques. ### Rules that could never fire Four conditions used `files=` (exact match) with regex syntax, or carried a doubled backslash that YAML passes through literally: ```yaml - files=^examples/features/speculative_decoding/ # never matches - files~=^\\\\.buildkite/intel_jobs/ # never matches ``` A further 13 conditions pointed at files that have since moved or been deleted — `harmony` moved to the Rust frontend, `pallas` is gone. Repointed where the code moved, removed where it is gone. ### Retired: `new-model` The rule fired on any PR touching a model file **and** `registry.py`. `registry.py` is edited for renames, aliases, refactors and removals, not just additions. Of the 25 most recent PRs carrying the label, 10 added a model. The rest included: - `[Model] Remove Ouro`, `[Model] Remove Plamo2`, `Remove PersimmonForCausalLM and FuyuForCausalLM` — PRs that **delete** models - `[model registry] some simple typos` - `[Perf] Optimize clamp to clamp_`, `[Kernel] ReplaySSM: cache SSM inputs`, `Revert \"Blackwell decode optimizations\"` Whether a new architecture was registered depends on what changed *inside* `registry.py`. Mergify matches file paths, not diff contents, so this rule cannot be made accurate — tightening the paths does not help. 111 open PRs currently carry the label. The label remains on issues, applied by the New Model template, where the requester states the intent explicitly. ### Rule that fired too often `nvidia` matched `requirements/cuda.txt`, so dependency bumps were tagged as NVIDIA backend work. Measured across 200 merged PRs, 17 carried the label and this was the one clear miss. ### New rules `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`, `DSv4`, `dflash`, `torch.compile`, `ray`, `vllm-ir`. ### Issue labeling Extended from 6 labels to 31. Terms match **titles only** by design: issue bodies carry pasted `collect_env` output that names unrelated hardware and libraries, which tags the reporter's environment rather than the topic. ### Guard against recurrence `tools/pre_commit/check_label_rules.py` fails when a file condition matches nothing in the tree, and runs via pre-commit when `mergify.yml` changes. It caught two bugs in this change while it was being written. ### Also removed: the New Model issue template `.github/ISSUE_TEMPLATE/600-new-model.yml` is deleted. - Usage fell from 155 issues in 2025 to 30 in 2026, only 3 since July - It applied a label named `new model`, but the label is `new-model`. GitHub silently skips a label that does not exist, so requests went untagged from the March rename onward — five months, unnoticed - 292 of the 304 ever filed are closed - Model support arrives as planned work rather than as community requests `blank_issues_enabled` is `false`, so model requests route to Feature Request. **Nothing applies `new-model` after this change.** The label is left in place so it stays on the 917 PRs and 304 issues carrying it; deleting the label itself is a separate call. ## Before merging **The five new labels must be created first** — `scheduler`, `kv-cache-manager`, `glm`, `minimax`, `inkling`. Mergify errors on a label that does not exist. Two related items a PR cannot do: - `mrv1-only`'s description currently reads \"apply only to Model Runner V2 (not applicable to Model Runner V2)\" — says V2 twice. - `DSv4` has a rule here mirroring the `kimi`/`k3` split, but whether to keep it as a distinct label is a maintainer call. ## Test Plan Rules were validated against the real file tree and against real issues and PRs, rather than by inspection. ```bash # every file condition in mergify.yml matches at least one tracked file python tools/pre_commit/check_label_rules.py pre-commit run check-label-rules --files .github/mergify.yml pre-commit run ruff-check --files tools/pre_commit/check_label_rules.py pre-commit run ruff-format --files tools/pre_commit/check_label_rules.py ``` ## Test Result ``` tracked files: 6430 rules: 44 file conditions matching >=1 file: 154 file conditions matching NOTHING: 0 Check auto-label rules still match real files............................Passed ruff check...............................................................Passed ruff format..............................................................Passed ``` Negative test: injecting a condition pointing at a nonexistent path makes the check exit 1 and name the offending rule. New rules were spot-checked against the tree — `glm` matches 21 files and correctly ignores `midashenglm.py`, which contains the substring \"glm\" but is an unrelated model. Issue rules were replayed over 600 real issues: 43% now receive at least one label, and the existing rules agreed with hand-applied labels (`kimi` 35 fires / 31 already labeled, `rocm` 34 / 33, `k3` 27 / 26). No model evaluation is included because this change touches only CI labeling configuration. It does not affect inference, model output, accuracy, or serving. ## Note on AI assistance AI assistance (Claude Code) was used to audit the existing rules, measure PR and issue volumes, and draft these changes. Every measurement quoted above is reproducible with the commands shown. I have reviewed each changed line.", "url": "https://github.com/vllm-project/vllm/pull/51459", "createdAt": "2026-08-07T22:57:38Z", "updatedAt": "2026-08-13T16:22:38Z", "timestamp": "2026-08-13T16:22:38Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "documentation", "ready", "ci/build" ], "author": "jcotant-inferact", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:6a83a29ab5c314d188ac", "signalId": "github:vllm-project/vllm:pull_request:49793", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "text", "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Spec Decode][Perf] Fuse the MTP trailing all-reduce; local-argmax draft tokens", "text": "Two optimizations on the DeepSeek-V3.2 / GLM-5.2 MTP draft path. - **Fuse the trailing all-reduce into the final RMSNorm** on the non-sequence-parallel path, as the main model already does at layer boundaries. The sequence-parallel path is unchanged. - **Greedy draft tokens via vocab-parallel local argmax** (`get_top_tokens`), skipping the full-vocab all-gather in `compute_logits`. The proposer already probes for this via `use_local_argmax_reduction`; this adds the model side. Not a duplicate: an earlier revision also carried the `index_share_for_mtp_iteration` gates and the V2 speculator lifecycle hooks, both of which have since landed on main; that part is dropped. Nothing open overlaps with what remains. ```bash gh pr list --repo vllm-project/vllm --state open --search \"MTP all-reduce RMSNorm fuse\" gh pr list --repo vllm-project/vllm --state open --search \"local argmax draft tokens\" ``` ## Benchmarks 2×GB300, TP8 across 2 nodes, `nvidia/GLM-5.2-NVFP4`, MTP=5, `VLLM_USE_V2_MODEL_RUNNER=1`, fp8 KV cache, no prefix caching. The baseline arm is the same tree with the fusion reverted to an explicit `tensor_model_parallel_all_reduce` + `RMSNorm`; local argmax is toggled with `--speculative-config.use_local_argmax_reduction`. **Batch 1 (8192 in / 1024 out, c=1) — no measurable change.** Three runs per arm, output tok/s and MTP acceptance length: | arm | fusion | local argmax | output tok/s | acceptance length | | --- | :-: | :-: | --- | --- | | baseline | ✗ | ✗ | 456.7 / 451.6 / 463.3 | 4.78 / 4.70 / 4.86 | | +fusion | ✓ | ✗ | 487.0 / 459.0 / 461.3 | 5.05 / 4.76 / 4.79 | | +both | ✓ | ✓ | 460.1 / 472.8 / 470.2 | 4.80 / 4.95 / 4.90 | The arms overlap completely. At c=1 the saved all-gather is one vocab row and the fused all-reduce is one small tensor, so this is the expected result; median ITL sat at 9.08–9.12 ms across all nine runs. **Concurrency 64 (1024 in / 1024 out, 256 prompts) — this is where it pays.** One A/B pair on the same node pair: | arm | output tok/s | acceptance length | median ITL | | --- | --- | --- | --- | | baseline | 2923.2 | 4.86 | 31.83 ms | | +both | 3321.1 (**+13.6%**) | 4.69 | 31.32 ms | The gain cannot be attributed to speculation luck: acceptance length is 3.5% *lower* in the faster arm. Normalizing throughput by acceptance length (steps/s) gives +17.8%. At c=64 the draft's per-step full-vocab all-gather is 64 × 151k × 2 B ≈ 19 MB, five times per accepted token — that is what `get_top_tokens` removes. Caveats, stated plainly: the c=64 result is a single A/B pair, and a repeat was lost to cluster contention. Cross-node variability on this cluster is large (the +fusion arm measured 4226 tok/s on a different node pair with the same acceptance length as the baseline), so only within-pair comparisons above are meaningful — do not compare numbers across the two tables. ## Accuracy gsm8k, 5-shot, full 1319 questions, via `lm_eval --model local-completions`: | arm | flexible-extract | strict-match | | --- | --- | --- | | baseline | 0.9439 ± 0.0063 | 0.9424 ± 0.0064 | | +both | 0.9431 ± 0.0064 | 0.9409 ± 0.0065 | Unchanged within stderr, which is the point: the fused kernel accumulates in fp32 and local argmax breaks vocab-boundary ties differently, so both are output-affecting in principle. AI assistance (Claude) was used; every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/49793", "createdAt": "2026-07-25T08:16:24Z", "updatedAt": "2026-08-13T16:22:17Z", "timestamp": "2026-08-13T16:22:17Z", "metrics": { "reactions": 2, "comments": 15 }, "labels": [ "speculative-decoding", "ready", "v1", "mrv2" ], "author": "zhou9402", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e2d1b7ae632fcce13b70", "signalId": "github:vllm-project/vllm:pull_request:52173", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52173", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Apply logit softcapping in Transformers modelling backend", "text": "This is mainly used by Gemma models and only affects workloads where the exact logit values are important. Generation is unaffected because it does not reorder anything.", "url": "https://github.com/vllm-project/vllm/pull/52173", "createdAt": "2026-08-13T14:01:25Z", "updatedAt": "2026-08-13T16:21:56Z", "timestamp": "2026-08-13T16:21:56Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ready" ], "author": "hmellor", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:9a450613f16800c00840", "signalId": "github:vllm-project/vllm:pull_request:50983", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:50983", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Preserve the audio duration-limit error through the PyAV fallback", "text": "## Purpose load_audio wraps every failure on its PyAV fallback path as \"Invalid or unsupported audio file.\" including the rejection raised by the VLLM_MAX_AUDIO_DECODE_DURATION_S guard itself. For containers soundfile cannot probe (like WebM), audio over the duration limit therefore reports a misleading invalid-file error, while soundfile-readable formats (mp3, ogg, flac) report the actionable duration message. This PR types the guard's rejection as AudioDurationExceededError and re-raises it ahead of the generic wrap, so only genuine decode failures are normalized. No behavior change for callers catching ValueError. ## Test Plan ``` pytest tests/multimodal/media/test_audio.py ``` New tests synthesize an opus WebM in memory to exercise the PyAV fallback while a garbage-bytes control verifies genuine decode failures keep the generic error. ## Test Result All 12 tests pass (Linux x86_64, CUDA torch). Without the fix, both new rejection tests fail: the duration error is masked as \"Invalid or unsupported audio file.\". With it, WebM over the limit reports the same actionable message as mp3.", "url": "https://github.com/vllm-project/vllm/pull/50983", "createdAt": "2026-08-04T06:33:26Z", "updatedAt": "2026-08-13T16:21:20Z", "timestamp": "2026-08-13T16:21:20Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "bug", "needs-rebase", "multi-modality" ], "author": "ieaves", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:a071385d5c45495ea5e5", "signalId": "github:vllm-project/vllm:pull_request:49357", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "updatedAt", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:49357", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[BugFix] bound FlashMLA sparse decode intermediate tensors size", "text": "## Purpose Fix #44545 - not in an ideal way though. When FlashMLA sparse kernel is selected with FP8 KV cache dtype, FlashMLA sparse decode kernel(https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_decode.h#L184) is called in mixed batch mode. This kernel internally allocates intermediate tensors (`o_accum`) on call, which quickly dominates the usual unallocated workspace size (~2GB per the comment in `flashmla_sparse.py`) for long prefill: For 32k input on H200, the bytesize of `o_accum` is ``` # b = 1 # num_sm_parts = 1 # s_q = 32768 # h_q = 64 # d_v = 512 (b + num_sm_parts) * s_q * h_q * d_v * 4 bytes = 8 GiB ``` For BF16 KV cache, a different kernel (FlashMLA sparse prefill kernel - https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_fwd.h#L101) is called, which does not allocate a massive VRAM on kernel call and thus does not suffer an OOM contrary to the FP8 code branch. (Note the additional `d_v` factor in the formular for `o_accum`, no other scratch tensors in FlashMLA have this factor thus negligible in terms of VRAM occupancy) Hence we chunk the mixed-batch input into manageable bytes (`VLLM_FLASHMLA_SPARSE_MAX_SCRATCH_MB`) and loop over the chunks to aggregate the output attention tensor, which is in the same spirit as the indexer budget management logic in #36178. Due to the mechanism (looping the kernel over sub-chunks instead of single big shot), it may incur a throughput penalty as a tradeoff, but I have no better idea yet. As the context window for MLA model families keep growing, FP8 KV cache is kind of inevitable, so I think this PR can work as a bandaid. ## Test Plan No OOM for the reproducer in #44545 ## Test Result pass ---
Essential Elements of an Effective PR Description Checklist - [x] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [x] The test plan, such as providing test command. - [x] The test results, such as pasting the results comparison before and after, or e2e results - [x] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/49357", "createdAt": "2026-07-21T18:06:42Z", "updatedAt": "2026-08-13T16:20:26Z", "timestamp": "2026-08-13T16:20:26Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "v1" ], "author": "cjackal", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1b5b7fdd08f628d195a0", "signalId": "github:vllm-project/vllm:pull_request:41834", "event": "changed", "observedAt": "2026-08-13T17:43:20.785491Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:41834", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes", "text": "## Summary This PR enables DeepSeek V4 Flash on SM120/SM121 Blackwell client hardware by carrying the SM12x fallback and tuning stack needed for the current vLLM V1 path. It targets RTX PRO 6000 Blackwell Workstation Edition, RTX 5090-class SM120, and GB10 / DGX Spark SM121 users who cannot use SM100-only TMEM / `tcgen05` kernels. The branch is **reconciled on top of the merged #43477** and provides the **stock-deps path**: DeepSeek V4 on SM120/121 that builds and serves on *released* FlashInfer / DeepGEMM wheels, complementing #43477's route that needs the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches. It is kept **synced onto current `upstream/main`**. **Latest validated head: tag `sm120-pr-41834-stable-preview-20260809` (`aa0d513027`), synced onto `upstream/main` as of 2026-08-09 (`f18e10a7e1`)** — see [Update 2026-08-09](#update-2026-08-09) below. **The default model runner is now V2**; `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Model / speculative-decode status.** `deepseek-ai/DeepSeek-V4-Flash-0731` is the checkpoint this branch is validated on. It **removed the MTP heads** and folded the DSpark draft into the main checkpoint, so **DSpark (`method: \"dspark\"`, `num_speculative_tokens: 5`) is the speculative path**; MTP is supported only for older checkpoints that still carry those weights. Running without speculation is also fully supported and validated. ## Change footprint — model kernels vs. core-vLLM touch points 187 files, ~+29.1k / −1.3k against `upstream/main`, of which ~10.7k added lines are tests. The branch splits cleanly into model/kernel code and a small set of core-vLLM integration points: - **DeepSeek-V4 model + SM12x kernels — the enablement itself.** Everything under `vllm/models/deepseek_v4/**` plus the SM12x sparse-MLA decode / indexer / DeepGEMM kernels that live in shared dirs (`v1/attention/backends/mla/sparse_mla_kernels.py`, `model_executor/layers/sparse_attn_indexer.py`, `v1/attention/backends/mla/{indexer,sparse_swa}.py`, `utils/deep_gemm.py`, `kernels/mhc/tilelang.py`), the new DSv4 reasoning parser / tokenizer, and device tuning JSONs. - **C128A metadata device→host sync removed** (`models/deepseek_v4/sparse_mla.py`, perf) — `_c128a_effective_topk_width` takes the max position from the CPU-side `CommonAttentionMetadata.max_seq_len` instead of a per-step `int(positions.max().item())` device sync, dropping a launch-stream stall on every C128A metadata step. Decode is identical (`max_seq_len-1 == positions.max()`); only chunked prefill sees a safe, slightly-wider 128-aligned top-k. - **Core-vLLM integration — the hooks below.** Almost all are gated by model architecture / quant config / an env flag and are inert for other models. | Subsystem | Files | What it does | | --- | --- | --- | | KV-cache core | `single_type_kv_cache_manager.py`, `kv_cache_coordinator.py`, `kv_cache_manager.py`, `sched/scheduler.py` (+1) | prefix-cache correctness for DSv4 sparse-MLA + speculative decode: an MLA cache-manager with prompt-block protection, a hybrid-coordinator `cache_blocks` tail-block-reuse rewrite | | Speculative decode | `v1/spec_decode/{dspark,dspark_sampling,llm_base_proposer,dflash}.py`, `config/speculative.py` | DSpark self-drafting proposer + sampling; DSv4 probabilistic draft sampling and per-step draft-layer routing in the shared proposer base; DSpark/MTP method detection and validation | | MoE quantization | `fused_moe.py`, `oracle/mxfp4.py`, `routed_experts.py`, `experts/flashinfer_cutlass_moe.py`, `quantization/mxfp4.py`, `oracle/nvfp4.py` | MXFP4 / NVFP4 backend selection; the one-line NVFP4 fix (FLASHINFER_CUTLASS into the SwiGLU-clamp allow-list) lets DSv4-Flash-NVFP4 serve | | FP8 / Marlin GEMM | `quantization/utils/fp8_utils.py`, `linear/scaled_mm/{cutlass,marlin}.py`, `csrc/.../marlin_moe_wna16/ops.cu` (the only C++) | SM12x e8m0→fp32 upcast + Marlin MoE SM12.0a cudagraph hardening | | cudagraph / compile / config | `config/vllm.py`, `compilation/breakable_cudagraph.py`, `passes/utility/fix_functionalization.py`, `config/compilation.py` | breakable-cudagraph auto-enable gate (MiniMax-only; DSv4 deliberately excluded), DSv4 custom-op defunctionalization + splitting-op registration | | OpenAI entrypoints / parsers | `chat_completion/protocol.py`, `serve/render/serving.py`, `tool_parsers/structural_tag_registry.py`, `chat_utils.py`, `engine/protocol.py`, `chat_completion/{serving,batch_serving}.py`, `reasoning/__init__.py` | expose DSv4 API semantics — `reasoning_content` / `thinking` param / tool-call streaming (jasl/vllm#19 instruction-following) | | Kernel warmup | `model_executor/warmup/deepseek_v4_sm12x_warmup.py` (new), `kernel_warmup.py` (+11) | DSv4 warmup passes (D512-split prefill precompile, paged-MQA rowwise, draft path) that avoid JIT-during-inference wedges. Kept in a separate module so `kernel_warmup.py` stays a two-line hook on upstream's file | | Weight loading | `weight_utils.py`, `default_loader.py` | fast-safetensors weight filter + EP-skip (lowers DSv4 load overhead on GB10) | | env / utils | `envs.py`, `utils/flashinfer.py`, `utils/import_utils.py`, `v1/worker/{gpu_model_runner,ubatch_utils}.py` | `VLLM_DEEPSEEK_V4_*` flags + `has_cutedsl` / `has_flashinfer_trtllm_sparse_mla` probes | Two notes for review: - The most invasive generic edits were **removed** in the 2026-06-21 audit cleanup: the scheduler carries a single +1-line change (the prefill-fairness heuristics were dropped) and the prefix-cache write-fence is gone. - A few hooks touch code paths shared with non-DSv4 models and are worth a closer look: the `kv_cache_coordinator` `cache_blocks` rewrite (affects hybrid-KV models; validated ≥ prior behavior), the proposer base-class change, and the OpenAI-entrypoint plumbing. Everything else (MoE oracle, fp8_utils, cudagraph gate, warmup, envs) is arch / quant / env-gated and inert for other models. ## Duplicate-work check The nearest open/merged PRs are related but not duplicates: | PR | Difference | | --- | --- | | #43477 | **Merged 2026-06-22.** Enables DeepSeek V4 + GLM-5.1 on SM120 via the FlashInfer-SM120 sparse-MLA route, but on its merged form **requires the unreleased FlashInfer #3395 + DeepGEMM #324 dependency branches** — on released/stock wheels its SM12x path raises at model construction. This PR is **reconciled on top of #43477** (merge `42657aca65`) and carries the **stock-deps** DSv4 SM120/121 path that runs on released wheels. | | #40929 | Earlier WIP Triton fallback effort. This PR is the maintained replacement branch with the broader scheduler, prefix-cache, parser, quant, warmup, and harness-validated fixes carried forward. | | #42856 | Focused workspace-bound fix that explicitly depends on / references this PR; a subset-style bugfix, not the full DeepSeek V4 SM12x enablement branch. | | #49335 | mxfp8 activation-scale swizzle after DP/EP dispatch — **carried in this branch** (unclaimed upstream). Inert at DP=1; taken for this branch's multi-node DP users. | | #50686 | Consecutive-assistant-message merging in DSv4 prompt encoding — **carried in this branch** (reproduced here before taking it). | | #50693 | B300-targeted prefill-workspace fix. **Test carried, code not needed**: this branch's `_prefill_workspace_topk_bound` returns early for `compress_ratio <= 1` and never reaches the affected buffer. | Upstream PRs whose fixes this branch previously carried as local deltas and has since **retired** in favour of upstream's own version: #48304, #48911, #48959 (via #49052). ## Fixed preview tags These tags are in `jasl/vllm` and give users stable pins while the PR is still moving: | Tag | Commit | Notes | | --- | --- | --- | | `sm120-pr-41834-stable-preview-20260809` | `aa0d513027` | **latest validated head** — 87 upstream commits incl. FlashInfer 0.6.16.post3; the V2 recall collapse root-caused as a prefix-cache ghost-block race and fixed (port of #42359); **default runner switched to V2**. See **Update 2026-08-09**. | | `sm120-pr-41834-stable-preview-20260804` | `0f59188db1` | 35 further upstream commits, four fixes from community reports (DSpark out-of-vocab draft token, eager scratch pool), two contributor PRs. Validated on **both** SM121 and SM120. See **Update 2026-08-04**. | | `sm120-pr-41834-stable-preview-20260802` | `9a94c54292` | 234 upstream commits, `DeepSeek-V4-Flash-0731` support, two DSpark config fixes, #49335 / #50686 absorbed. See **Update 2026-08-02**. | | `sm120-pr-41834-stable-preview-20260727d` | `d64074e6f0` | 209-commit upstream sync + torch 2.13 (tag `…-20260727`, `70a33886bd`); DSpark VRAM work (jasl/vllm#27) merged; bounded block-table gather in `compute_global_topk_indices_and_lens`. | | `sm120-pr-41834-stable-preview-20260721` | `832775efd1` | 79-commit upstream sync; #48911 dropped in favour of upstream's merged version; compact CPU KV offload (opt-in). | | `sm120-pr-41834-stable-preview-20260717` | `f63bfd3d7b` | 195-commit upstream sync; prefill ctx_pp +4.7% @ d8192. | | `sm120-pr-41834-stable-preview-20260711` | `b5c0d43b96` | 181-commit upstream sync; #48304 MTP unscaled-draft-rope; ~1097-line dead-kernel cleanup. | | `sm120-pr-41834-stable-preview-20260704` | `b43470e871` | @GanyX19 GB10 fixes: per-shape `constexpr`→runtime (stops the Triton recompile → unified-memory leak → hard-freeze) + fp8-einsum `tl.multiple_of(16)` (~24% decode @256K). | | `sm120-pr-41834-stable-preview-20260703` | `444fe3ac8b` | DSpark spec-decode (self-drafting block-5), V2 padded-Q OOM fix (jasl/vllm#26), exact non-cooperative `persistent_topk` for <128 KB-smem parts. | Older tags (`…-20260705` back to `…-20260612…`) remain in `jasl/vllm` for history. ## Update 2026-08-02 — `DeepSeek-V4-Flash-0731`, 234 upstream commits, two DSpark fixes Validated head **`9a94c54292`** (tag `sm120-pr-41834-stable-preview-20260802`), 234 upstream commits absorbed, level with `upstream/main` as of 2026-08-02. **What's in it** - **`DeepSeek-V4-Flash-0731` support.** The new checkpoint ships **no MTP heads** — `enorm`, `hnorm`, `e_proj`, `h_proj` and `shared_head` are absent from the weight index, and `mtp.{0,1,2}.*` now carries the DSpark-style `main_norm` / `main_proj` structure (matching `dspark_target_layer_ids: [40, 41, 42]`). DSpark is the speculative path going forward; the MTP code is retained for older checkpoints. - **`num_speculative_tokens` vs `dspark_block_size`** — *this rule was relaxed on 2026-08-04; see Update 2026-08-04. It now errors only BELOW the block size and warns above it.* The original reasoning and measurements follow. - The validator was tightened to require equality. The validator previously accepted `>=` and its error message recommended exceeding it. The drafter emits exactly one block per pass, so the extra slots are structurally unreachable — measured on a prose workload, the **7th draft position accepted 0.000 in every sample** (the 6th in all but one, 0.004 there), and `nst=7` drafts 40% more tokens per step for strictly worse acceptance: | configuration | mean acceptance length (3 samples) | avg draft acceptance rate | | --- | --- | --- | | nst=5 probabilistic | 2.15 / 2.16 / **2.19** | 22.9 / 23.2 / **23.8%** | | nst=7 probabilistic | 1.61 / 1.75 / 1.95 | 8.7 / 10.7 / 13.6% | | nst=5 greedy | 1.82 / 2.06 / **2.23** | 16.4 / 21.2 / **24.5%** | | nst=7 greedy | 1.57 / 1.66 / 1.75 | 8.2 / 9.5 / 10.8% | All samples are shown rather than a single figure: the probe reads whatever `SpecDecoding metrics` lines vLLM flushed inside its window, so a low sample means \"not much steady traffic in that slice\", not a worse drafter. Both `nst=7` runs also hit connection errors partway through, so their spread is noisier. - **`method: \"mtp\"` is no longer silently rewritten to `\"dspark\"`.** Auto-detection preserved an explicitly requested method only for `eagle`/`eagle3`/`dflash`/`dspark`. Since 0731 puts `dspark_block_size` in every DSv4 config, `method: \"mtp\"` fell through to the dspark branch, was rewritten, and then failed validation with a DSpark message the user never asked for. - **Absorbed** #49335 (mxfp8 activation-scale swizzle after DP/EP dispatch, verified non-vacuous) and #50686 (consecutive assistant messages merged in prompt encoding — it also removes a spurious mid-turn `<|end_of_sentence|>` on the default `drop_thinking=True` path). #50693's regression test is carried; its code fix is not reachable here. - **Three community reports re-tested and not reproduced** on this head: nst=7 draft-position collapse, TP=4 + expert-parallel `Check failed: num_tokens > 64`, and `FLASHMLA_SPARSE_DSV4` missing `tile_sched`. Details in [this comment](https://github.com/vllm-project/vllm/pull/41834#issuecomment-5152562124). **Validation (GB10 SM121, 2-node TP=2, `DeepSeek-V4-Flash-0731`, torch 2.13.0, FlashInfer 0.6.15.post1, nccl 2.30.7)** | | DSpark nst=5 | no speculation | | --- | --- | --- | | GSM8K 8-shot (flexible) | 0.9394 | 0.9500 | | GSM8K 8-shot (strict) | 0.9363 | 0.9484 | | instruction-following (jasl/vllm#19, JSON-only) | PASS | PASS | | long-context recall (arthur needle, c=1) | 2/2 | 2/2 | | illegal-access / assertion in serve log | 0 | 0 | | draft acceptance (prose) | mean 2.08, 21.7% | — | The GSM8K difference (1.06 pp flexible / 1.21 pp strict) is within this gate's measured single-run spread (~1.1 pp). **Resolved:** three runs per cell were collected and the arms interleave, so it was noise. `0731` is the first checkpoint where the strict and flexible extractors disagree at all; on every prior baseline they were identical. Perf — pinned llama-benchy standard (fp8 KV, prefix-cache on, `FULL_AND_PIECEWISE`, mml 49152, util 0.85; C=1, 3 runs), against the **full** recorded range of the prior MTP2 baselines. This crosses a checkpoint boundary, so read it as a sanity band rather than a controlled A/B: | metric | prior MTP2 range (n=10) | 0731 + DSpark | vs band | | --- | --- | --- | --- | | pp2048 @ d8192 | 1339.11 – 1400.81 | **1432.23** ± 11.74 | **above** | | pp2048 @ d16384 | 1308.77 – 1344.68 | **1356.56** ± 11.78 | **above** | | pp2048 @ d32768 | 1089.05 – 1226.63 | **1250.75** ± 2.18 | **above** | | ctx_pp @ d8192 | 1757.16 – 1876.01 | 1816.97 ± 5.89 | inside | | ctx_pp @ d16384 | 1769.85 – 1842.16 | 1817.43 ± 1.43 | inside | | ctx_pp @ d32768 | 1595.87 – 1756.01 | 1740.22 ± 2.87 | inside | | tg128 @ d8192 | 36.27 – 43.08 | 41.72 ± 5.09 | inside | | tg128 @ d16384 | 34.59 – 43.14 | 37.92 ± 9.92 | inside | | tg128 @ d32768 | 32.77 – 42.91 | 34.88 ± 5.78 | inside | | ctx_tg @ d8192 | 38.52 – 43.01 | 39.37 ± 2.34 | inside | | ctx_tg @ d16384 | 39.29 – 43.07 | **35.07** ± 0.67 | **below, −10.7%** | | ctx_tg @ d32768 | 38.02 – 42.73 | 40.70 ± 6.85 | inside | **Batched prefill (pp2048) is above the historical band at all three depths** (+2.2% / +0.9% / +2.0%) — the only consistent directional move here. Clearing the max of ten prior runs at all three depths says more than any single one of those margins would: +0.9% is inside this metric's own resolution, so read the consistency rather than the magnitudes. No sign of DSpark being slower than MTP2 was. One caveat reported rather than buried: **`ctx_tg @ d16384` sits 10.7% below its historical minimum**, the only metric outside its band. It is non-monotonic against our own neighbouring depths (39.37 at d8192, 40.70 at d32768, where history has d16384 ≈ d8192), which points at a single-run artifact rather than a depth-specific regression. **Resolved:** repeated on later heads and it did not recur. A measurement caveat for anyone benchmarking this branch: the `±` in a benchy row is the spread of the three runs *inside one invocation*, and it runs 5–30× smaller than the build-to-build spread. This branch's own history spans 31% on tg128 @ d32768 and ~1.3% on ctx_pp, so anything under ~15% on tg or ~2% on ctx_pp is not resolvable this way. ## Update 2026-08-04 — four fixes from community reports, 35 upstream commits, and first SM120 validation Validated head **`0f59188db1`** (tag `sm120-pr-41834-stable-preview-20260804`). **This is the first head validated on both SM121 and SM120.** Every SM120 discrete-GPU result on this PR up to now was a contributor's measurement we could not reproduce. We have since rebuilt a 2× RTX PRO 6000 Blackwell box as a first-party SM120 target. ### Fixes - **DSpark's fused Markov sampler could emit an out-of-vocab token id** (`e171c51036`). `_dspark_markov_probs_blocks_kernel` stores `vocab_size` as the filler for a block with no active lane. On a fully-masked row — every candidate `-inf`, which structured-output constraints can produce — *no* block has an active lane, so every block stores the filler and the reduce kernel returns it verbatim as the sampled token. Nothing downstream bounded it: the runner clamped `input_ids` with `min=0` only, and the DSv4 hash-MoE router indexes `tid2eid[token_id * 6 + lane]` on a `[vocab_size, 6]` table. Result is an illegal memory access on every TP rank. This is the producer on the **V1** path, which is this branch's default. @alexbi29's report traced the same class of defect to the V2 samplers (#50843) — a real defect, but a different tree. Fixed by folding out-of-range to `0` (matching `torch.argmax` on such a row, so the fused kernel stays bit-identical to the eager reference) and making the runner clamp two-sided. Worth stating plainly for anyone with similar gates: **our own gates could not have caught this**. The fused path is skipped when `all_greedy`, and both our long-context recall gate and GSM8K are greedy, so they are structurally incapable of executing that kernel. The new regression test is explicitly non-greedy. - **Adopted #50844** (`3df857ba50`) — bound `token_id` before the `tid2eid` gather. Defence in depth; `prompt_token_ids` reach that gather directly when `--skip-tokenizer-init` disables the engine's vocab check. **Not** taking #50843 (V2-tree only, inert on our default) or #50845, which has a defect reported on its own thread. - **Eager scratch pool is now OFF by default** (`d42b8d9f55`, `b1ef3033f4`), opt-in via `VLLM_DEEPSEEK_V4_EAGER_SCRATCH_POOL=1`. @tobymao bisected output corruption under concurrent mixed prefill+decode to it: pool active 7/7 rounds corrupt, disabled 0/2. We first removed the cross-template aliasing (`max()` → `sum()` sizing with per-family offsets); **they tested that commit directly and it was still corrupt in round 1**. Their diagnosis is the useful part: the pre-pool code was race-free *for free* because per-call transients go through the caching allocator, whose cross-stream reuse is event-guarded — the pool reuses memory without that machinery, so no static partitioning fixes it. Making the cross-layer reuse safe needs producer-waits-on-consumer events against the real stream graph; until then, off by default. - **Two contributor PRs merged** — jasl/vllm#37 (tuned FP8 W8A8 config for `N=4096,K=12288` on RTX PRO 6000) and jasl/vllm#38 (hoist the E8M0 block-scale upcast out of the FP8 GEMM hot path, 13,561 kernel launches removed per 25 decode steps), both from @alexbi29. - **`num_speculative_tokens` rule relaxed.** Upstream removed its own assertion in #50869 as \"invalid\". They were right that erroring *above* `dspark_block_size` is wrong — two users on this thread run `nst=7` against `block_size=5` and it demonstrably works. The two directions are not symmetric, so this branch now **errors below** the block size (that genuinely garbles output) and **warns above** it, quoting the acceptance cost. Strictly more permissive than what shipped before. ### Validation Full gate battery on **both** architectures, same branch: | gate | SM121 (2× GB10, 2-node TP=2) | SM120 (2× RTX PRO 6000, TP=2) | | --- | --- | --- | | serve, DSpark nst=5, `--block-size 256` | ✅ | ✅ | | instruction-following (jasl/vllm#19) | PASS | PASS | | long-context recall, arthur c=1 | 2/2 | 2/2 | | long-context recall, arthur c=12 | 22, 23, 22 / 24 | 22, 23 / 24 | | GSM8K 8-shot flexible | 0.9484 / 0.9507 / 0.9492 | 0.9371 | | GSM8K 8-shot strict | 0.9462 / 0.9477 / 0.9462 | 0.9303 | | tool-calling, 135 cases | 256/270 (94.8%) | — | | illegal-access / assertion lines | 0 | 0 | The ~1.1 pp GSM8K difference between architectures sits inside this gate's measured single-run spread and spans different silicon, different memory architecture and a 3× smaller KV cache (6.25 GiB vs ~18.5 GiB). We are not claiming a difference from it. **`Check failed: num_tokens > 64` does not reproduce on this branch.** @fuzzifikation reported stock 0.26.0 dying there on SM120 at `--block-size 256`, correctly tracing it to the DSv4 decode dispatch requiring `page_block_size == 64`. On our SM120 box, at the same `--block-size 256`, the serve comes up and the assertion never appears — the DSv4 packed KV cache is laid out in 64-token pages independent of vLLM's logical block size, and FlashInfer derives `page_block_size` from tensor geometry rather than the engine config. The SM120 packed decode path is confirmed engaged in the same run. Note the same assertion has **two** distinct gates (`page_block_size` and `(num_heads, topk)`, the latter being #50720 / flashinfer#3989), so patching one and still seeing it means checking the other. ### Prefill: V1 vs V2 model runner The 2026-08-02 V1-vs-V2 comparison never measured throughput. It has now been measured, blocked and pre-registered — 10 blocks, both arms inside each node pair, exact sign-flip permutation test, Holm-corrected across the six prefill cells, with the decision rule committed before any data was collected: | metric | V2 / V1 | 95% CI | exact p | | --- | --- | --- | --- | | ctx_pp @ d8192 | **+1.18%** | [+0.82, +1.53] | 0.0020 | | ctx_pp @ d16384 | **+1.11%** | [+0.35, +1.87] | 0.0137 | | ctx_pp @ d32768 | **+1.61%** | [+1.17, +2.04] | 0.0020 | | pp2048 @ d8192 | **+4.18%** | [+3.28, +5.09] | 0.0020 | | pp2048 @ d16384 | **+4.18%** | [+3.36, +5.00] | 0.0020 | | pp2048 @ d32768 | **+4.33%** | [+3.56, +5.11] | 0.0020 | All six survive Holm; both node pairs agree in direction on every cell. **Decode is not resolved in either direction** — `tg128` was declared unresolvable before the run (its within-build spread equals its entire historical range) and is reported for the record only. **V1 was the default when this was written; that was reversed on 2026-08-09 — see Update 2026-08-09.** The reasoning below was correct on the evidence available at the time, and the collapse it describes was real; it turned out not to be a property of the runner. Kept unedited because how the conclusion failed is the useful part. V2 is ahead on prefill, KV headroom (+4.70 GiB) and draft acceptance (+6.6%), but its long-context recall under concurrency is unreliable in a way that is worse than a consistent deficit: across 14 independent serves on the same build and configuration, roughly two thirds land in a state that loses most of the needles (arthur c=12 as low as 3/24), while the rest match V1 at 22–24/24. The mode is fixed at startup and stable within a serve, and nothing we have found predicts or detects it. A deployment could run clean for days and restart into the bad mode. The cause is **not identified**. Eliminated so far: the eager-scratch cross-template aliasing, the upstream merges, and the eager scratch pool as a whole (pool on 2 good / 6 bad vs pool off 3 good / 3 bad over 14 serves — no effect). The startup logs of a good and a bad serve are structurally identical, which rules out \"a different code path was taken\". Anyone opting into V2 with `VLLM_USE_V2_MODEL_RUNNER=1` should know this. ### Measurement note Two errors from our own process, since they affect how the numbers above should be read. The n=8 sampling that originally established V2's recall deficit took eight gate runs from **one serve** — it measured within-serve variance while the quantity that actually varies is across-serve. Raising n on the wrong axis. The 14-serve figures above use the inverted design: many serves, few gates each. And the `±` in a benchy row is the spread *within one invocation*; it runs 5–30× smaller than the build-to-build spread. The blocked design above exists because of that: a coarse range screen over the same 10 blocks returns \"no measurable difference\" on all six prefill cells, while the paired test finds all six. Had the screen been the decisive statistic, this section would have concluded the opposite and been wrong. ## Update 2026-08-09 — the V2 recall collapse was a prefix-cache race, not the runner; V2 becomes the default 70 upstream commits (to `643c125fab`), and the long-standing reason this branch pinned V1 is gone: it was **an unfixed upstream bug**, not a property of the V2 model runner. ### The defect `FullAttentionManager.cache_blocks()` commits prefix block hashes to the shared `BlockPool` **at scheduling time, before the forward pass writes their KV**. A request admitted later in the same step can match those hashes and read unwritten values. `MambaManager` has guarded this since #29387; no other manager does. This is **vllm-project/vllm#42359**, open and unmerged. Two more reports look like the same triple on different models — **#50188** (prefix caching + MTP spec decode + fp8 KV, byte-identical repeat requests, RTX 5090 / Qwen3.6-27B-NVFP4) and **#43559** (closed without a merged fix). Anyone on `--enable-prefix-caching` with speculative decoding is exposed; DeepSeek-V4 is not special here. What makes it hard to catch: **the damage persists**. A serve that loses the race keeps serving from the poisoned blocks for its lifetime, so a later *serial* request fails too — which is why it looked like a per-serve \"mode\" rather than a race. It is also stochastic, roughly half of cold serves. ### Evidence Same binary, `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT` the only variable, cache populated by the real gate, **4 fresh serves per arm** (a single clean serve proves nothing at ~50% incidence), 3 arthur c=12 runs each: | | serve 1 | serve 2 | serve 3 | serve 4 | mean | min | | --- | --- | --- | --- | --- | --- | --- | | guard **off** | 22/23/24 | **6/5/3** | **14/9/10** | **7/8/7** | 11.5 | 3 | | guard **on** | 23/22/21 | 20/22/20 | 22/23/23 | 23/22/23 | 22.0 | 20 | Mann-Whitney U, p = 0.0043. Every serve's runner and guard state was read back from the serve log rather than assumed. **It also fixes V1**, which was not expected: V1's arthur c=12 goes 20.7 → 23.0 with the guard on (24.0 with prefix caching disabled entirely). V1's own 2–4 needle shortfall was the same defect, not an inherent concurrency margin. ### Runner arbitration, re-run on the fixed tree Same tree, same guard mode, **runner the only variable**: | | V1 | V2 | | --- | --- | --- | | arthur c=12, 4 serves × 3 | 22.3 / 21.7 / 21.7 / 21.0 → **21.67** | 22.0 / 20.7 / 22.7 / 22.7 → **22.00** | | pp2048 d8192 / 16384 / 32768 | 1427 / 1363 / 1216 | **1472 / 1421 / 1303** (+3.1% / +4.2% / +7.1%) | | tg128 mean | 39.95 / 41.16 / 35.25 | **41.56 / 49.61 / 45.18** | | e2e TTFT | 1437 / 1506 / 1689 ms | **1393 / 1444 / 1576 ms** | | GPU KV cache | 339,194 tok | **423,752 tok** (+24.9%) | | GSM8K strict / flexible | 0.9378 / 0.9401 | 0.9401 / 0.9439 | | issue19 · multi-needle · c=1 | PASS · 48/48, 0 leaks · 2/2 | PASS · 48/48, 0 leaks · 2/2 | Recall: p = 0.697, neither side with a single-digit serve. GSM8K differs by <0.4 pp against ~1.1 pp single-run noise. **V2 is not behind anywhere and leads on throughput, latency and KV headroom, so it becomes the default.** `VLLM_USE_V2_MODEL_RUNNER=0` still selects V1, which stays supported. **Correction:** the *\"V2 +6.6% draft acceptance\"* figure in Update 2026-08-04 does **not** survive re-measurement — 2.772 (V1) vs 2.710 (V2) on the same formula and sample size, i.e. a tie. It was measured while V2 was poisoned. ### If you are running this branch **Nothing to set — the guard is on by default where it matters.** `KVCacheCoordinator` enables it whenever prefix caching and speculative decoding are both active, which is the only configuration in which a block hash can be published before its KV is written *and* a second request admitted in the same step to match it. This correction matters: an earlier revision of this update shipped V2 as the default while leaving the guard off by default, which would have handed a plain serve the exact combination measured at mean 11.5 with a 3/24 floor. Both changes looked like improvements in isolation. If you pulled `c054feedac`, take `aa0d513027` instead, or set the variable yourself. To turn it **off** (it is a real escape hatch, pinned by a test): ```bash VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0 ``` `1` is upstream's semantics, gated on `use_eagle`; on DeepSeek-V4 that covers only **2 of 5** managers and leaves the main MLA path unguarded — measured, not assumed, via a startup log line this branch adds that reports how many managers are actually guarded. `2` covers every group and is what the engine selects. Regression sweep on the merged tree: 261 passed, 1 failed, that one failing identically on the pre-merge tree `4ebd1fb698`. ### The two endpoints disagreed about the same model, twice `ResponsesRequest.reasoning` took its type from the OpenAI SDK, whose `ReasoningEffort` stops at `xhigh`, so DeepSeek's documented top tier `max` was rejected by schema validation on `/v1/responses` while `/v1/chat/completions` accepted it. Worse, and on the default path: with no thinking kwarg, `DeepSeekV4Tokenizer.apply_chat_template` defaults thinking **on** while `DeepSeekV4ReasoningParser` defaults it **off** and selects `IdentityReasoningParser`. The model reasoned and its reasoning, with a bare ``, came back inside `output_text` as though it were the answer — whenever a request omitted `reasoning`, which is exactly what a stock OpenAI SDK sends. Chat was immune only because it normalises thinking state at the protocol boundary, and its own docstring says why: *\"so the tokenizer and reasoning parser see the same effective state\"*. Responses never called that hook. The derivation now lives in `deepseek_v4_chat_kwargs` and both request types call it, so a third endpoint cannot repeat it. Measured on both checkpoints with no workaround flag set, 21/21 each: | | `DeepSeek-V4-Flash-0731` | `DeepSeek-V4-Flash` | | --- | --- | --- | | silence: reasoning in its own field | PASS | PASS | | silence: no `` in the answer | PASS | PASS | | `effort: none` disables thinking | PASS | PASS | | six spellings × two endpoints | PASS | PASS | | `high` reasons deeper than `low` | +85% / +110% | +19% / +35% | 26 unit cases accompany it, 11 of which fail on the unpatched tree. `tests/reasoning` 440 passed, `tests/tokenizers_/test_deepseek_v4.py` 45 passed. ### Acceptance on the exact published SHA Everything above was re-measured on `d44e224ab9` — the commit this tag points at, after a second upstream sync (17 further commits, FlashInfer 0.6.16.post3) — not on an ancestor assumed to be equivalent. 17 of 18 checks pass: | check | result | | --- | --- | | four nodes clean at the SHA, FlashInfer 0.6.16.post3 | PASS | | `tests/v1/core` | 509 passed, 1 pre-existing failure | | default serve with **nothing set**: boots, runner V2, guard 5/5, no NameError | PASS | | arthur c=12 ×3 / c=1 | 22 / 20 / 23 · 2/2 | | GSM8K strict · issue19 · multi-needle | 0.9363 · PASS · 48/48, 0 leaks | | pp2048 d8192 | 1480.77 (arbitration V2 arm 1471.8) | | `VLLM_USE_V2_MODEL_RUNNER=0` → V1, guard still 5/5, c=1 2/2 | PASS | | `VLLM_ALLOW_SPEC_DEC_SAME_STEP_PREFIX_HIT=0` → guard 0/5, c=1 2/2 | PASS | **The one non-pass, and what it turned out to be.** `tests/v1/spec_decode` does not complete on this hardware — it wedges under a 30-minute bound on this head and on `4ebd1fb698` alike. Narrowed to `test_max_len.py` and measured both ways: | how it is run | result | | --- | --- | | whole file, one pytest process | wedges after ~7 min, 5 of 11 done | | each case in its own process | **11 of 11 pass**, free memory steady at 117 GiB | No individual case is broken. Each stands up a full engine, and repeated create/tear-down inside one process does not release resources fast enough on a **single-GPU** unified-memory node. That also explains why both trees wedge and why they stop at different points. It remains **unverified coverage rather than a pass**; running one process per case produces a verdict instead of a hang. `test_async_scheduling_pp_allows_rescheduling_with_output_placeholders` is the same class: it builds `pipeline_parallel_size=2`, and a GB10 node has one GPU, so it fails at config construction. It is the only case in `tests/v1/core` that needs more than one GPU; the other 509 pass. ### What this arbitration does and does not cover Everything above was measured on **one configuration**: 2-node TP=2, `DeepSeek-V4-Flash-0731`, DSpark `num_speculative_tokens: 5`, fp8 KV, `max_model_len` 131072, prefix caching on, GB10 (SM121). The default now applies to **every** DSpark config, including shapes not measured here — TP=4, other context lengths, the NVFP4 checkpoint, single-node setups. The reasoning generalises better than the numbers do: the race is in block publication and is not specific to a model shape, and V2's advantage comes from KV headroom and scheduling rather than anything config-specific. But if you run a materially different shape and see something worse, `VLLM_USE_V2_MODEL_RUNNER=0` returns you to V1 and a report would be welcome — that is a gap in our coverage, not a claim we have ruled out. ## Running DSpark DSpark is DeepSeek's self-drafting speculative-decode variant; on `0731` the draft weights are carried in the main checkpoint, so no separate `--speculative-model` is needed. ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \\ --trust-remote-code \\ --tokenizer-mode deepseek_v4 \\ --tool-call-parser deepseek_v4 --enable-auto-tool-choice \\ --reasoning-parser deepseek_v4 \\ --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --block-size 256 \\ --max-model-len 49152 \\ --max-num-seqs 64 \\ --max-num-batched-tokens 8192 \\ --gpu-memory-utilization 0.85 \\ --enable-prefix-caching \\ --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":5,\"draft_sample_method\":\"probabilistic\"}' ``` - `num_speculative_tokens` **must equal** the checkpoint's `dspark_block_size` (5). Larger values are rejected: they are never accepted and only waste draft compute. - `--kv-cache-dtype fp8` is mandatory — DSv4's `fp8_ds_mla` attention asserts an fp8 KV layout, so the default `auto` fails at model construction. Not DSpark-specific. - Runs on the **V1 runner by default** (correct long-context recall). `VLLM_USE_V2_MODEL_RUNNER=1` opts into the V2 DSpark speculator; V2's long-context recall is correct after the jasl/vllm#26 padded-Q fix. - If you measure draft acceptance yourself, **use prose**. On counting or repeated text the Markov head alone reaches 68–100% acceptance even with the neural draft path degraded, which hides real regressions entirely. ## Dependencies (stock-deps path) Pins on the current head: torch 2.13.0 (triton 3.7.1) · `flashinfer-python` / `flashinfer-cubin` 0.6.15.post1 · tilelang 0.1.12 · `nvidia-cutlass-dsl[cu13]` 4.6.0 · `quack-kernels>=0.6.1` · nvidia-nccl-cu13 2.30.7 (multi-node, see below). - **FlashInfer** is pinned in `requirements/cuda.txt` (`flashinfer-python` **and** the GitHub-release `flashinfer-cubin`, which must be the same version); it ships the SM120 packed sparse-MLA kernels, so a stock build picks them up with no manual install dance. - **GB10 / multi-node**: pin `nvidia-nccl-cu13==2.30.7` on every node. A rebuild silently reverts it to torch's bundled version, and a per-node mismatch hangs the NCCL handshake. - The SM120 **decode** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_DECODE`) and **prefill** (`VLLM_DEEPSEEK_V4_FLASHINFER_SM120_PREFILL`) FlashInfer sparse-MLA paths **default on**; set either `=0` to fall back to the FlashMLA / Triton path. Both are availability-gated, so stock installs without the kernel degrade gracefully rather than raising. ## Running the NVFP4 checkpoint This branch also serves `nvidia/DeepSeek-V4-Flash-NVFP4` on SM12x (RTX PRO 6000 / GB10). The NVFP4 MoE auto-selects the FlashInfer CUTLASS backend (the SwiGLU-clamp model gate accepts it), so no `--moe-backend` flag and no special FlashInfer build are required: ```bash vllm serve nvidia/DeepSeek-V4-Flash-NVFP4 \\ --trust-remote-code --tensor-parallel-size 2 \\ --kv-cache-dtype fp8 \\ --tokenizer-mode deepseek_v4 ``` Expert-parallel off (plain TP) is the supported path. Accuracy matches MXFP4 (GSM8K 8-shot ~0.96 on both SM120 and SM121). On SM12x NVFP4 is **not** a memory or throughput win versus MXFP4: NVFP4 weights are ~4 GiB/GPU larger, leaving less KV-cache room; single-stream prefill is marginally faster and aggregate decode marginally slower. Its value here is checkpoint availability / parity with the SM100 datacenter path — MXFP4 remains the better practical choice on consumer Blackwell. ## AI assistance disclosure AI assistants, including OpenAI Codex/GPT models and Anthropic Claude models, were used for code review, refactoring support, regression-script writing, and benchmark analysis. The branch was validated through human review plus the commands and harness artifacts listed above; every performance and accuracy number quoted was measured on real SM120/SM121 hardware.", "url": "https://github.com/vllm-project/vllm/pull/41834", "createdAt": "2026-05-06T15:17:15Z", "updatedAt": "2026-08-13T14:36:24Z", "timestamp": "2026-08-13T14:36:24Z", "metrics": { "reactions": 41, "comments": 441 }, "labels": [ "new-model", "structured-output", "frontend", "speculative-decoding", "needs-rebase", "ci/build", "v1", "tool-calling", "deepseek", "kv-connector", "nvidia", "quantization", "mrv2" ], "author": "jasl", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:8903dad0307b6a3042ec", "signalId": "github:vllm-project/vllm:pull_request:51746", "event": "discovered", "observedAt": "2026-08-13T17:47:07.884300Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:51746", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add explicit model info cache preparation", "text": "## Purpose PR #23558 by @manoelmarques added the source-hash-validated runtime ModelInfo cache and reported model-class inspection at 10.1034 seconds uncached versus 0.6127 seconds cached, n=1 per state. During review, @hmellor objected to mirroring common runtime requirements into the build. The merged design therefore populated the cache at runtime. Without a matching entry, first activation still pays the subprocess inspection. This PR adds an explicit preparation command for the existing cache format: ```bash vllm prepare-model-info Qwen3ForCausalLM ``` The command runs in the final installed environment and ensures a readable entry exists under `VLLM_CACHE_ROOT/modelinfos`. It does not download model weights. It accepts only an unchanged built-in lazy registration, then verifies that the entry can be read after it is written. Unknown, plugin, eager, and overridden registrations fail. Normal startup behavior is unchanged: missing or stale entries still use the existing subprocess inspection and safe replacement path, and ordinary cache writes remain best-effort. This deliberately does not generate entries during a generic wheel or source build. Model inspection can depend on runtime packages, platform modules, and compiled extensions, so the entry must be prepared in the target runtime or final deployment image. The tradeoff is time shifting, not free work. In the hardware gate, the preparation command took 9.7860 seconds, n=1, and produced an 870-byte entry. It is useful when that prepared cache is baked or distributed to matching fresh deployments with the same installed environment, or when readiness latency matters more than image preparation time. A writable deployment that starts once will populate the same cache on its own. ## Test Plan Focused registry and public CLI coverage: ```bash pytest -q \\ tests/entrypoints/unit_tests/test_launch_cli.py \\ tests/models/test_registry.py::test_prepare_model_info_inspects_builtin_lazy_registration \\ tests/models/test_registry.py::test_lazy_prepare_model_info_requires_readable_cache \\ tests/models/test_registry.py::test_inspect_model_info_keeps_best_effort_cache_write \\ tests/models/test_registry.py::test_lazy_prepare_model_info_accepts_readable_cache \\ tests/models/test_registry.py::test_prepare_model_info_rejects_unsupported_registration ``` All applicable changed-file pre-commit hooks were also run from exact base `b2506d62aec7e6bccc5959b829221a7ae217abf3` to exact head `13f5279875da8dd5e90c9a5a3f779fe839689e54`. The end-to-end gate used Qwen3-0.6B FP16 TP1 on one RTX 4090 Laptop GPU, maximum model length 256, warm model pages, and identical non-ModelInfo generated state. The interleaved sequence was empty, valid, valid, empty, empty, valid. ## Test Result Focused public CLI and registry coverage passed 20 tests. All applicable changed-file pre-commit hooks passed, including Ruff, formatting, markdownlint, Python 3.10 mypy, SPDX, lazy-import, forbidden-import, and configuration checks. | Starting state | Process launch to HTTP ready | Process launch to first correct token | | --- | ---: | ---: | | Empty ModelInfo cache | 20.2773 s median | 20.4360 s median | | Valid prepared entry | 16.0033 s median | 16.1558 s median | | Reduction | 4.2739 s, 21.1% | 4.2802 s, 20.9% | Each arm used n=3. All 18 exact-token requests passed, the deliberately wrong-token control failed, and all six cells passed GPU engagement, cleanup, source-origin, page-cache, and generated-state controls. This hardware result used exact source `8a90f1ab7565b6b6ad824b5643c203bb40bddcac` on base `8bcc916a98a90822882455ab30aba02c409da2a6`. The same feature patch was then rebased without conflicts onto base `b2506d62aec7e6bccc5959b829221a7ae217abf3` at current head `13f5279875da8dd5e90c9a5a3f779fe839689e54`; its stable patch ID is unchanged, and the intervening upstream commits overlap none of the six feature files. It is a Qwen3-0.6B installed-runtime integration result, not an exact-current-head, dense-model, cross-platform, or wheel-build performance claim. The measured source used a recorded compatible precompiled runtime payload. AI assistance was used. I reviewed the changes and can defend the implementation, constraints, and measurements.", "url": "https://github.com/vllm-project/vllm/pull/51746", "createdAt": "2026-08-11T01:35:33Z", "updatedAt": "2026-08-13T17:45:30Z", "timestamp": "2026-08-13T17:45:30Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "documentation", "new-model", "frontend" ], "author": "matteso1", "state": "closed", "assignees": [], "change": "new" } }, { "id": "event:6eea789bcd87348eaffb", "signalId": "github:vllm-project/vllm:pull_request:52197", "event": "changed", "observedAt": "2026-08-13T17:47:07.884300Z", "changedFields": [ "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Support DSpark configs with `architectures=DSparkDraftModel` + `model_type=qwen3`", "text": "## Purpose Generic Qwen `DSparkDraftModel` is now normalized to `Qwen3DSparkModel` so models like https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark can now run on vLLM ## Test Plan ## Test Result Tested gsm8k with ``` vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned75 -tp=4 --spec-model RadixArk/Qwen3.8-2.4T-A95B-DSpark --spec-method dspark --spec-tokens 7 ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52197", "createdAt": "2026-08-13T17:37:50Z", "updatedAt": "2026-08-13T17:38:26Z", "timestamp": "2026-08-13T17:38:26Z", "metrics": { "reactions": 3, "comments": 0 }, "labels": [ "qwen" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:084f456ac0594ea1f844", "signalId": "github:vllm-project/vllm:pull_request:52117", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52117", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix] Do not require SupportsPP of draft models under pipeline parallelism", "text": "Fixes #52069. `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so `_verify_args` demands `SupportsPP` from the draft model. No head in `MTPModelTypes` implements it, so every MTP speculative config dies in `create_engine_config` under PP > 1. A draft model always runs whole on a single PP rank (the V2 runner builds it on the last rank only), so this verifies it as single-stage, on a shallow copy of the config. **Test plan:** without the change, `--pipeline-parallel-size 4 --speculative-config '{\"method\":\"mtp\",...}'` fails at config time on GLM-4.5-Air; with it, the server loads and serves (8× RTX 3090, three machines, nightlies dev678–dev693). Complementary to #46994, which does not touch this file. --- Assisted-By: Claude", "url": "https://github.com/vllm-project/vllm/pull/52117", "createdAt": "2026-08-13T07:22:01Z", "updatedAt": "2026-08-13T18:01:09Z", "timestamp": "2026-08-13T18:01:09Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug" ], "author": "nickus", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:e2538f01d7b29d3f5919", "signalId": "github:vllm-project/vllm:pull_request:51360", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51360", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Frontend] Add reusable TP1 initialized-engine snapshots", "text": " ## Purpose @elizabetht's RFC #34303 and #37921/#37925 split CUDA checkpoint work into in-process primitives and leave CRIU disk persistence for a later phase. #35934 proposes a CRIU-safe engine suspend and resume path, while PRs #42790 and #46089 propose provider-owned pod snapshot hooks. @fergusfinn's data in #34303 also showed that full initialized snapshots can work, with large images and stale external TCP connections as central costs. @simon-mo's closed, reference-only #40323 autoresearch showcase also included inert snapshot scaffolding for a pre-model image of Python imports and a CUDA context. It explicitly excluded model weights, model configuration, compile state, and CUDA graphs. This PR instead implements and measures a host-owned artifact captured after TP1 engine initialization and a deterministic generation canary, before HTTP bind. It is also the measured successor to my closed PR #48996. PR #48996 captured only the import stage. Its matched warm comparison was flat at about 29.08 seconds versus 29.09 seconds, n=3 per arm, because the expensive engine state still had to be constructed after restore. This PR moves the boundary after engine initialization and a deterministic generation canary, but before the HTTP listener binds. The user-facing workflow is: ```bash vllm snapshot create MODEL --snapshot-dir PATH --revision MODEL_SHA [serve args] vllm snapshot inspect PATH vllm snapshot restore PATH --host 127.0.0.1 --port 8000 ``` Creation captures the initialized TP1 process tree and CUDA state with CRIU and NVIDIA's CUDA plugin. Restore validates the saved host and software identity before CRIU runs, restores the tree, binds the requested HTTP address, and checks the saved token again before returning. It does not silently fall back to ordinary startup. The lightweight `vllm_bootstrap` wrapper is intentional. Importing the normal CLI loads the vLLM runtime and PyTorch before CRIU can restore the saved process, consuming much of the activation win. Only `vllm snapshot restore` takes the lightweight path. Every other `vllm` command delegates to the existing CLI entry point unchanged. This is adjacent to, but does not replace, the in-process suspend and resume work. The current product boundary stays TP1 so it can preserve the initialized process tree without claiming multi-GPU communicator restore. ## Product envelope and tradeoffs This feature is opt-in and intentionally exchanges local disk and host privilege for activation time. - Linux x86-64, one NVIDIA GPU, TP1, PP1, and DP1 only. - Same host, GPU, driver, kernel, Python, PyTorch, vLLM source and binary, model revision, engine arguments, selected environment, and plugin identity. - Root or passwordless `sudo`, CRIU, `cuda-checkpoint`, and NVIDIA's CRIU CUDA plugin. - `io_uring` disabled before vLLM starts. For an unprivileged process, `kernel.io_uring_disabled=1` is sufficient; mode `2` disables it host-wide. - One plaintext HTTP frontend without API authentication, TLS, custom middleware, or a Unix domain socket. - No external established TCP peer at capture. Tree-owned internal connections are recorded. - Private local artifact that may contain process memory, model state, and secrets. - No concurrent restore of one artifact and no automatic normal-start fallback. The documentation explains these constraints, the sensitive-artifact boundary, and the disk, preparation-time, privilege, compatibility, and portability costs. ## Update, August 8 After the original `16999dbe788f` head passed full CI, I added one opt-in artifact-size path to the same product. `vllm snapshot create --minimize-snapshot-state` uses the existing sleep level 2 lifecycle after the correctness canary and before capture. It releases the model-weight and KV-cache allocations, then reloads the weights and recreates the KV cache after restore and before binding HTTP. The default full-state path is unchanged. The dense A10 decision cell used experimental source `4154f9b61775048f7cb679c4581706fd620426ff`, Qwen3-8B FP16 TP1, warm model-weight pages, and a cold snapshot artifact: | Path | Allocated artifact | Activation to first correct token | | --- | ---: | ---: | | ordinary | 24,231,231,488 B | 23.7633 s median, n=3 | | minimized | 5,144,281,088 B | 21.2396 s median, n=3 | The minimized path removed 19,086,950,400 allocated bytes, or 78.8 percent, and reduced the activation clock by 2.5237 seconds, or 10.6 percent. Every accepted restore returned the exact oracle token, completed at least six post-restore requests, and left no GPU process behind. The public option was then tested on source `e790b9b2018a6727bea069fa11bf1d1e64626e2b` with Qwen3-0.6B FP16 TP1 on an RTX 4090 Laptop GPU. It reduced the allocated artifact from 11,623,870,464 to 3,698,221,056 bytes, or 68.2 percent. The ordinary artifact restored once and the minimized artifact restored twice. Each restore passed the exact-token oracle plus three additional requests and drained the GPU. Page residency was uncontrolled, so this is a mechanism smoke and not another performance claim. Feature production-code changes since that tested source are a reviewer-requested hardening of existing CRIU link-remap validation and the lightweight package rename from `vllm_cli` to `vllm_bootstrap` after a reviewer found a collision with the independent `vllm-cli` PyPI project. Neither changes the measured activation path. Deterministic regressions cover the prior remap check/read race and the console entry-point/package-discovery boundary. Current head `f86a1e269b` rebases the tested implementation plus those two review fixes onto vLLM main `8a9f9f762a`; the remaining feature changes since the tested source are documentation and tests. The five commits added since the previous reviewed base touched none of this PR's 16 files. The state-minimization measurements use a different model and starting state from the original 9.4234-second A10 restore result below. The savings are not additive. Warm model-file pages can make the smaller artifact faster, while cold or slow storage can make reloading weights slower than restoring the full process image. ## Test plan Focused CPU tests: ```bash PYTHONPATH=$PWD pytest -q \\ tests/snapshot \\ tests/entrypoints/unit_tests/test_launch_cli.py ``` Changed-file repository checks: ```bash pre-commit run \\ --from-ref 8a9f9f762a47e24ed8b48a8472f6757346cb30ae \\ --to-ref HEAD ``` The RTX 4090 Laptop GPU cell used: - source `6d4507450d6ad2b226b5883904e009b9d11617ec` - base `dd856e48bbf969e3f0e561e8c76f6e92c76e0795` - RTX 4090 Laptop GPU, 16,376 MiB, driver 595.71.05 - Ubuntu kernel 7.0.0-28, Python 3.12.3, torch 2.13.0+cu132, CRIU 4.2.1 - `Qwen/Qwen3-0.6B` at revision `c1899de289a04d12100db370d81485cdf75e47ca` - FP16, maximum model length 512, GPU memory utilization 0.50, TP1, PP1, and DP1 - warm Hugging Face, TorchInductor, Triton, and vLLM caches - `VLLM_NO_USAGE_STATS=1` The Python source was the exact clean feature commit. The precompiled native payload was reused from `adc3e03517d2e7333a3bb2083bb4d394a2986876` after a path-limited diff found no C++, CUDA, Rust, FlashAttention, or vendored native-source change. The only `setup.py` change was an optional dependency minimum. The final NVIDIA A10 acceptance cell used: - exact source `16999dbe788f2a1d7e6b0de243e874193388e352` - base `c810e5ee9976ad86b81d1277b53e76d0ee639414` - NVIDIA A10, 23,028 MiB, driver 570.148.08 - Ubuntu 22.04.5, kernel 6.8.0-60, Python 3.10.12, torch 2.13.0+cu129, CUDA 12.9, CRIU 4.2.1 - the same model revision and engine settings as the 4090 cell - warm Hugging Face, TorchInductor, Triton, and vLLM caches - documented unprivileged `kernel.io_uring_disabled=1` This exact head contains the two repairs selected by the first A10 run. It captures the caller's environment before vLLM injects internal defaults, and it rejects active `io_uring` descriptors with an actionable error before CRIU runs. ## Test result Focused tests: ```text 65 passed ``` All applicable changed-file pre-commit hooks passed, including Ruff, formatting, typos, markdownlint, Python 3.10 mypy, SPDX, lazy imports, forbidden imports, and configuration checks. Each hardware cell restored one artifact five times and ran five matched ordinary warm starts. Both arms used the same source, model revision, caches, engine settings, console entry point, clock boundary, and correctness request within that cell. ### RTX 4090 Laptop GPU | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 15.7790 s | 5 | | initialized snapshot restore | 5.5488 s | 5 | | reduction | 10.2302 s, 64.8% | 5 per arm | Snapshot creation took 37.50 seconds and recorded 11,595,128,620 artifact bytes, about 11.60 GB. The measured break-even was the fourth repeated activation. ### NVIDIA A10 | Path | Median to checked token | Samples | | ---------------------------- | ----------------------: | --------: | | ordinary warm start | 23.9121 s | 5 | | initialized snapshot restore | 9.4234 s | 5 | | reduction | 14.4888 s, 60.6% | 5 per arm | Snapshot creation took 42.3594 seconds and recorded 15,580,855,788 artifact bytes, about 15.58 GB. The measured break-even was the third repeated activation. Every restore and warm control in both cells returned token ID `12095` with text value \" Paris\" for the prompt `The capital of France is`. The final A10 cell exercised both product defects found by the first A10 run. With the host default `kernel.io_uring_disabled=0`, creation stopped before CRIU and named the owning PIDs. Mode `1` then completed creation and all five restores. The saved manifest omitted vLLM-injected internal defaults, while changing caller-provided `VLLM_USE_FLASHINFER_SAMPLER` still produced `snapshot mismatch: environment` before CRIU ran. The runner restored the host sysctl and left the GPU process list empty. These break-even points belong only to their measured machine, model, and cache state. Larger models, slower disks, or different GPU memory settings can change both artifact size and restore time. Both hardware cells disabled usage reporting. #51219 separately closes the default telemetry HTTP session, so this PR does not duplicate that fix or claim a default-telemetry hardware result. This PR does not claim a model-quality improvement. The generation canary checks that the restored engine reproduces the saved deterministic token before it is declared ready. AI assistance was used. I reviewed the changes and can defend the implementation, constraints, and measurements. ## Checklist - [x] The purpose names the RFC and adjacent open work. - [x] The test plan includes focused tests, repository checks, and hardware. - [x] The result includes matched before and after measurements with n labels. - [x] The documentation covers setup, security, compatibility, and tradeoffs.", "url": "https://github.com/vllm-project/vllm/pull/51360", "createdAt": "2026-08-07T05:06:49Z", "updatedAt": "2026-08-13T18:00:06Z", "timestamp": "2026-08-13T18:00:06Z", "metrics": { "reactions": 2, "comments": 9 }, "labels": [ "documentation", "frontend", "ready" ], "author": "matteso1", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:1c6a752a5acf781257e9", "signalId": "github:vllm-project/vllm:pull_request:50280", "event": "discovered", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:50280", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Spec Decode] Make EAGLE weight sharing TP-consistent", "text": "## Purpose Addresses the rank-divergent EAGLE weight-sharing decision described in #50158. The issue stays open: real-hardware runs (see discussion below) show the acceptance collapse reported there persists after this fix and has a separate root cause that is still under investigation. EAGLE currently decides whether to share the target model's `embed_tokens` and `lm_head` by comparing each tensor-parallel rank's local shards independently. If equality differs across ranks, some ranks replace the draft module with the target module while others keep the draft weights. The ranks then build different drafters, which can severely reduce speculative acceptance. This change requires unanimous TP-wide agreement before sharing either module. Each rank contributes its local decision to a CPU-group `MIN` reduction, so any mismatch keeps the draft weights on every rank. TP size 1 preserves the existing fast path without a collective. The regression test covers both unanimous agreement and one-rank mismatch for `embed_tokens` and `lm_head`. The change is limited to the existing EAGLE weight-sharing decisions. The collectives occur in the same fixed model-load order on every TP rank and do not affect inference-time execution. ## Test Plan ```bash pytest -q tests/v1/spec_decode/test_eagle.py::test_eagle_weight_sharing_requires_all_tp_shards_to_match pre-commit run --files vllm/v1/spec_decode/llm_base_proposer.py tests/v1/spec_decode/test_eagle.py ``` ## Test Result - Targeted regression test: `2 passed`. On macOS, the platform backend list was patched to a non-empty value before module collection; the test itself mocks the TP CPU-group reduction. - All applicable pre-commit hooks passed on both changed files, including Ruff, formatting, typos, mypy, SPDX, import checks, and forbidden-import checks. - Real-hardware validation (community, 2x RTX 3090, TP=2, see discussion): the consensus mechanism works as designed — both ranks now log the same verdict where stock vLLM disagreed on every boot; no measurable perf change. The acceptance collapse from #50158 is unaffected by this change (separate root cause, tracked in the open issue). AI assistance: OpenAI Codex was used for analysis, implementation, testing, and PR preparation. All changes were reviewed by the submitter.", "url": "https://github.com/vllm-project/vllm/pull/50280", "createdAt": "2026-07-29T14:27:13Z", "updatedAt": "2026-08-13T17:59:54Z", "timestamp": "2026-08-13T17:59:54Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "bug", "speculative-decoding", "v1" ], "author": "sdflkjssl", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:790af83f5dbec8eb434a", "signalId": "github:vllm-project/vllm:pull_request:51915", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51915", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness", "text": "## Purpose Enables GLM-5.2 (`GlmMoeDsaForCausalLM`) end-to-end on `vllm/models/deepseek_v32/amd/` for gfx942/gfx950. Routing is opt-in via `--model-class-overrides`; the registry entry is unchanged, so the default path for GLM-5.2 and DeepSeek-V3.2 is untouched. The following issues were also fixed as a result of this overall enablement since it surfaced dormant bugs from a prior deepseek_v32 porting work: - **`launch_pdl` forwarded to a Triton runtime that doesn't accept it** — the shared `common/kernels.py` passed this CUDA-only kwarg unconditionally; ROCm's Triton rejects it by name even when `False`, so the engine died at startup with `KeyError`. Now passed only when the platform actually supports PDL. - **fp4 bmm invoked with mismatched positional args** — the output tensor landed in the kernel's `dtype` slot (`Expected 'Optional[int]' for 'dtype' but found Tensor`). Routed through the existing `rocm_aiter_ops.batched_gemm_a16wfp4` wrapper rather than calling aiter directly, so the argument contract lives in one place. - **fp4 MLA bmm silently wrong for non-power-of-two K** — `batched_gemm_a16wfp4` only supports power-of-two K, and returns wrong values without erroring otherwise. GLM-5.2's `qk_nope_head_dim=192` lands in the unsupported set; DeepSeek-V3.2's 128 does not. Guarded on both `qk_nope_head_dim` and `kv_lora_rank` to fall back to the fp8 bmm, with a warning. `W_K`/`W_K_scale` are also made contiguous after the transpose, which otherwise produced NaN. - **Indexer-K cache written in the plain layout while its reader expects the shuffled one** — affects DeepSeek-V3.2 identically, not just GLM-5.2. The AMD path writes this cache from the shared `fused_norm_rope`, which only implemented `[n_blocks, blk, head_dim]`, while the reader selects `SHUFFLE` above block size 1: the write kept storing row-major, but the reader expected shuffled, so the indexer scored garbage K and output degenerated once a sequence exceeded `index_topk=2048`, with no error raised. Refactored so the shared kernel stays platform-agnostic — it writes whichever layout the cache reports via a new `uses_shuffled_layout` property — and the AMD-specific shuffled write (`[n_blocks, blk/16, head_dim/16, 16, 16]`, used above block size 1) lives in the AMD backend. The base class returns `False`, so CUDA is unchanged. - **Sparse-MLA block sizes narrowed instead of extended** — the ported backends replaced the inherited `get_supported_kernel_block_sizes()` with `[16, 32]`, which made `select_common_block_size` silently downgrade a requested 64 to 32 via its largest-divisor fallback. Now concatenated with the base list. - **Kernel tests gated on `is_cuda()` and running against dims that matched neither model** — the tests were skipped entirely on CDNA. The gate is now `supports_fp8()`, so the suite runs on gfx950. The hardcoded attention dims were written as if one set served both GLM-5.2 and DeepSeek-V3.2, which it did not; they are now taken from each model's published config and parametrized over both. ## Test Plan Server: VLLM_ROCM_USE_AITER=1 \\ HF_HUB_OFFLINE=1 SAFETENSORS_FAST_GPU=1 \\ vllm serve /path-to-your/GLM-5.2-MXFP4 \\ --model-class-overrides '{\"GlmMoeDsaForCausalLM\": \"vllm.models.deepseek_v32:DeepseekV32ForCausalLM\"}' \\ --tensor-parallel-size 8 \\ --kv-cache-dtype auto \\ --max-model-len 65536 \\ --no-enable-prefix-caching \\ --trust-remote-code \\ --port 8000 Variants: --kv-cache-dtype fp8 for the fp8 KV row; --block-size {16,64} for the block-size matrix. Drop --model-class-overrides to run the legacy deepseek_v2 path for comparison. Accuracy: lm_eval --model local-completions \\ --model_args \"model=/models/GLM-5.2-MXFP4,base_url=http://localhost:8000/v1/completions,num_concurrent=32,tokenized_requests=False,max_retries=3,timeout=3600\" \\ --tasks gsm8k --num_fewshot 5 --gen_kwargs max_gen_toks=2048 --limit 200 Throughput: vllm bench serve --backend vllm --model /shareddata/models/GLM-5.2-MXFP4 \\ --dataset-name random --random-input-len 4096 --random-output-len 1024 \\ --num-prompts 200 --max-concurrency 32 --ignore-eos --port 8000 \\ --save-result --result-filename glm52_v32amd_bench.json Kernel unit tests: pytest tests/kernels/test_fused_deepseek_v32_norm_rope.py -v ## Test Result **gsm8k** — GLM-5.2 200 samples, 5-shot, `exact_match`, `deepseek_v32` AMD path | KV cache | Before (flexible / strict) | After (flexible / strict) | |-------------|----------------------------|------------------------------| | auto (bf16) | 0.0000 / 0.0000 | **0.9400 / 0.9400** ± 0.0168 | | fp8 | 0.0000 / 0.0000 \\* | **0.9350 / 0.9350** ± 0.0175 | \\* Not separately measured. fp8 KV goes through the same absorbed-weight bmm where the bug was, so it fails identically; only the bf16 run was evaluated pre-fix.", "url": "https://github.com/vllm-project/vllm/pull/51915", "createdAt": "2026-08-12T02:59:15Z", "updatedAt": "2026-08-13T17:59:08Z", "timestamp": "2026-08-13T17:59:08Z", "metrics": { "reactions": 3, "comments": 1 }, "labels": [ "bug", "rocm", "needs-rebase", "deepseek" ], "author": "jhu960213", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:01f58712abae1393fddc", "signalId": "github:vllm-project/vllm:pull_request:51793", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt", "state" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51793", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Quantization] Remove dead `QuantizationConfig.is_mxfp4_quant`", "text": "## Purpose This is dead code following https://github.com/vllm-project/vllm/pull/37128. This was originally added in https://github.com/vllm-project/vllm/pull/29008 that supported padding for gpt-oss / certain MXFP4 backends, see: https://github.com/xuebwang-amd/vllm/blob/c62f664e97977ee54ab1d1c77604ebb45081bc06/vllm/model_executor/layers/fused_moe/layer.py#L259-L277 This is now handled in: https://github.com/vllm-project/vllm/blob/490259c1f63faf025b8050504db63d81c817d781/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py#L640 ## Test Plan N/A ## Test Result N/A", "url": "https://github.com/vllm-project/vllm/pull/51793", "createdAt": "2026-08-11T09:46:17Z", "updatedAt": "2026-08-13T17:58:31Z", "timestamp": "2026-08-13T17:58:31Z", "metrics": { "reactions": 2, "comments": 5 }, "labels": [ "ready", "quantization" ], "author": "fxmarty-amd", "state": "closed", "assignees": [], "change": "updated" } }, { "id": "event:46b0c6e4778fc489233a", "signalId": "github:vllm-project/vllm:pull_request:52183", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt", "metrics" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52183", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel)", "text": "## Purpose Update the PyTorch ecosystem to the **2.14.0** release candidate, resolving wheels from the PyTorch **test channel** (`download.pytorch.org/whl/test/...`): - `torch`: → **2.14.0** - `torchvision`: → **0.29.0** - `triton`: → **3.8.0** 2.14.0 is published on `download.pytorch.org/whl/test/` but not yet on the release index / PyPI, so the index URLs point at the test channel. This mirrors the 2.13.0 test PR #45731. ### Verified availability (test channel) before pinning - `torch-2.14.0+cu130`: present - `torchvision-0.29.0`: present - `triton==3.8.0`: present, and is exactly what `pytorch` `release/2.14` pins in `.ci/docker/triton_version.txt` - `torchaudio`: newest published is still **2.11.0** on every channel, so that pin is unchanged ### ROCm `requirements/build/rocm.txt` moves from the `rocm7.1` index to `test/rocm7.2`. `rocm7.1` has no torch 2.14.0 wheels — 2.14 is built for `rocm7.2` and `rocm7.14` only. That file was also still on torch 2.11.0 / triton 3.6.0, so this brings it in line with `CMakeLists.txt`'s `TORCH_SUPPORTED_VERSION_ROCM`. ### Ray dependency-compatibility check `.buildkite/scripts/check-ray-compatibility.sh` now offers `uv` both the stable and test channels (it already runs with `--index-strategy unsafe-best-match`), so an RC `torch` pin coming from a transitive dep resolves during the RC window instead of failing against a stable-only index. Same fix as #45731. ### Deliberately *not* carried over from #45731 That PR also carried fixes reacting to breakages found during 2.12/2.13 CI. Those are not copied here, since 2.14 should get whatever it actually needs once CI reports: - xfails for pytorch#184431, pytorch#187735, pytorch#187727 - the ROCm source-build override + `install_flash_attn_rocm.sh` (needed because ROCm/pytorch had no `release/2.13` branch) - `verify_torch_wheel.py` / `UV_NO_VERIFY_HASHES` RC-respin handling `requirements/build/tpu.txt` is also left alone — TPU torch is bumped separately (#50412). ## Test Plan Full CI (apply the `ready` label to trigger the Buildkite run). ## BC-breaking? No — test-channel version bump only. *Authored with AI assistance; modeled on #45731.*", "url": "https://github.com/vllm-project/vllm/pull/52183", "createdAt": "2026-08-13T14:56:49Z", "updatedAt": "2026-08-13T17:57:44Z", "timestamp": "2026-08-13T17:57:44Z", "metrics": { "reactions": 2, "comments": 2 }, "labels": [ "ci/build", "cpu", "nvidia" ], "author": "atalman", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:4e5b0776be7d485c80a0", "signalId": "github:vllm-project/vllm:pull_request:52198", "event": "discovered", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52198", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts", "text": "# [Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts ## Problem PEFT 0.18+ can train a LoRA against a fused 3D MoE expert parameter via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`), reading it `[E, dim1, dim2]` as `[E, in, out]`. vLLM's `add_lora_w13` / `add_lora_w2` punica kernels use the opposite `nn.Linear` `[out, in]` convention, so per expert `ΔW_vllm = ΔW_peft^T`. On stock vLLM, `_stack_moe_lora_weights` reshapes with the wrong semantics and `set_lora` raises a shape mismatch that drops the adapter (e.g. Qwen3.5-35B-A3B `down_proj`: `512 vs 2048`). ## Fix - `peft_helper.py`: parse `target_parameters` from `adapter_config.json`. - `lora_model.py`: propagate it onto `LoRAModel` (ctor, `clone()`, `from_lora_tensors`). - `model_manager.py`: in `_stack_moe_lora_weights`, detect the PEFT 3D layout and swap `lora_A`/`lora_B` + transpose the trailing two dims **after** the EP-aware expert slicing. Detection is required, not assumed: `create_dummy_lora_weights` builds dummies in the native layout with the same flat shape, so an unconditional transpose would corrupt them. Primary signal is `target_parameters` naming a fused expert weight; fallback is `lora_a` trailing dim != `hidden_size`. ## Testing ``` python -m pytest -v \\ tests/lora/test_peft_helper.py::test_peft_helper_target_parameters_direct \\ tests/lora/test_lora_manager.py::test_target_parameters_indicates_3d_lora \\ tests/lora/test_lora_manager.py::test_from_lora_tensors_propagates_target_parameters \\ tests/lora/test_peft_helper.py::test_peft_helper_pass \\ tests/lora/test_lora_manager.py::test_from_lora_tensors ``` → **10 passed** (the discriminator test is parametrized over 6 cases). **End-to-end (Qwen3.5-35B-A3B expert LoRA).** The raw PEFT adapter crashes the unpatched loader with the `512 vs 2048` error. With the fix, the directly-loaded adapter is **bit-identical** (chosen-token logprobs, 32 tokens × 5 prompts) to an independent offline-converted oracle that self-checks `max|ΔW_vllm − ΔW_peft^T| = 0` over 320 experts, while both shift logits measurably vs base — so the adapter takes effect and the in-loader transpose matches the verified transform. Logprobs, not decoded text: for an experts-only (8-of-256, rank-32) adapter, greedy text is a false-negative trap since the adapter shifts logits without always flipping the argmax on short prompts. ## AI assistance disclosure AI assistance was used to author this change. A human submitter has reviewed every changed line and run the tests above.", "url": "https://github.com/vllm-project/vllm/pull/52198", "createdAt": "2026-08-13T17:48:23Z", "updatedAt": "2026-08-13T17:55:50Z", "timestamp": "2026-08-13T17:55:50Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "bug" ], "author": "darrenpeng-amazon", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:8f10133a5e6a894c7732", "signalId": "github:vllm-project/vllm:pull_request:52197", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "text", "updatedAt", "metrics", "labels" ], "signal": { "id": "github:vllm-project/vllm:pull_request:52197", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "Support DSpark configs with `architectures=DSparkDraftModel` + `model_type=qwen3`", "text": "## Purpose Generic Qwen `DSparkDraftModel` is now normalized to `Qwen3DSparkModel` so models like https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark can now run on vLLM ## Test Plan ## Test Result Tested gsm8k with ``` vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned75 -tp=4 --spec-model RadixArk/Qwen3.8-2.4T-A95B-DSpark --spec-method dspark --spec-tokens 7 ... (APIServer pid=2078258) INFO 08-13 17:48:18 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.90, Accepted throughput: 1537.03 tokens/s, Drafted throughput: 2757.00 tokens/s, Accepted: 15372 tokens, Drafted: 27573 tokens, Per-position acceptance rate: 0.901, 0.795, 0.667, 0.550, 0.436, 0.327, 0.225, Avg Draft acceptance rate: 55.8% ``` ---
Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as \"Fix some issue (link existing issues this PR will resolve)\". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and after, or e2e results - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
", "url": "https://github.com/vllm-project/vllm/pull/52197", "createdAt": "2026-08-13T17:37:50Z", "updatedAt": "2026-08-13T17:53:53Z", "timestamp": "2026-08-13T17:53:53Z", "metrics": { "reactions": 3, "comments": 2 }, "labels": [ "speculative-decoding", "ready", "qwen" ], "author": "mgoin", "state": "open", "assignees": [], "change": "updated" } }, { "id": "event:aad9c4995adccc9c7f1e", "signalId": "github:vllm-project/vllm:pull_request:52199", "event": "discovered", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:52199", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Feature] Add local/external prefix-cache hit breakdown to prompt_tokens_details (additive)", "text": "## Summary A **strictly additive** change that exposes the **local vs external** prefix-cache hit breakdown per request in `prompt_tokens_details` (closes #52137). No existing field or behavior changes: `cached_tokens` keeps its exact meaning (total = `local + external`), and the two new fields are `None` by default / omitted from the response unless the flag is on. ## Background The scheduler already tracks `num_local_cached_tokens` / `num_external_cached_tokens` separately in `PrefillStats`, but `OutputProcessor` collapsed them to their sum (`num_cached_tokens`) before forwarding, so users of distributed prefix caching (Mooncake / lmcache / PD disaggregation) could not tell, per request, where cache hits came from (relevant for billing and diagnosing hit-rate changes). ## Changes (all additive) - **Data pipeline**: `RequestState` and `RequestOutput` gain `num_local_cached_tokens` / `num_external_cached_tokens`; `OutputProcessor` threads them from `PrefillStats` (same assignment site, so `cached_tokens == local + external` holds by construction). - **Protocol**: `PromptTokenUsageInfo` gains `local_cached_tokens` / `external_cached_tokens` (both default `None`). - **Endpoints**: chat and completion, streaming and non-streaming, all forward the breakdown. Other endpoints (responses / cohere / scale_out) are intentionally out of scope. - **Compatibility**: existing fields are untouched; new fields are `None` by default and omitted from the response when `--enable-prompt-tokens-details` is off — existing clients and dashboards keep working unchanged. ## Tests - `tests/entrypoints/openai/chat_completion/test_serving_chat.py`: `_make_prompt_tokens_details` with the new signature — defaults, gated-off behavior, multimodal coexistence, and the `cached_tokens == local + external` invariant. - `tests/v1/engine/test_output_processor.py`: `PrefillStats` → `RequestState` threading with non-zero local/external values.", "url": "https://github.com/vllm-project/vllm/pull/52199", "createdAt": "2026-08-13T17:48:30Z", "updatedAt": "2026-08-13T17:49:11Z", "timestamp": "2026-08-13T17:49:11Z", "metrics": { "reactions": 2, "comments": 1 }, "labels": [ "frontend" ], "author": "SayHelloToWorld", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:7ed414fea7a9bf53b28c", "signalId": "github:vllm-project/vllm:pull_request:46424", "event": "discovered", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [], "signal": { "id": "github:vllm-project/vllm:pull_request:46424", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[Bugfix][Mamba2] Fix assert crash when prefill-reclassified-as-decode occurs with no concurrent spec tokens", "text": "## Summary When a Mamba2-attention hybrid model (e.g. NemotronH) is served with both `--enable-prefix-caching` and `--speculative-config`, the server can crash with: ``` AssertionError File \"vllm/model_executor/layers/mamba/mamba_mixer2.py\", line 963, in conv_ssm_forward assert block_idx_last_scheduled_token_prev_step_d is not None ``` ### Trigger condition 1. `--enable-prefix-caching` auto-sets `mamba_cache_mode=all` for models implementing `SupportsMambaPrefixCaching` (`config.py:352`). This makes `is_mamba_cache_all=True` in the model. 2. A request arrives whose prompt is exactly `N × block_size + 1` tokens and the N-block prefix is already cached. Only 1 token remains to prefill. `mamba_attn.py` reclassifies any single-query-token prefill with prior state (`seq_len > 1`) as a decode step. 3. No other concurrent request has active speculative tokens that scheduler step, so `use_spec_decode = len(scheduled_spec_decode_tokens) > 0` is **False**. The current code gates `prev_last_scheduled_idx` inside this check, so it is never forwarded to `Mamba2AttentionMetadataBuilder`. 4. `mamba_mixer2.py:963` asserts this value is non-None whenever `has_decode=True`, `is_mamba_cache_all=True`, and `num_spec > 0`. The assert fires and kills all TP ranks. The crash is intermittent under heavy traffic: if any concurrent decode request happens to have active spec tokens in the same step, `use_spec_decode=True` and the bug is silently avoided. ### Root cause PR #41233 added `mamba_prev_last_scheduled_idx` and per-step preprocessing but left the per-step `use_spec_decode` gate in place. The fix is incomplete: `prev_last_scheduled_idx` must be forwarded whenever the buffer exists, not only on steps where spec tokens are scheduled. In v0.21.0 the crash was impossible because `config.py` forced `mamba_cache_mode=align` (not `all`) when spec decode was also active, so `is_mamba_cache_all` was always False. ### Fix Move the `prev_last_scheduled_idx` assignment outside the `use_spec_decode` gate. The guard `self.mamba_prev_last_scheduled_idx is not None` ensures this only runs for spec-decode deployments with `mamba_cache_mode=all`. `Mamba2AttentionMetadataBuilder` already handles `prev_last_scheduled_idx < 0` via the fallback `(num_computed - 1) // mamba_block_size`, so passing unconditionally is safe. ## Duplicate check Searched open PRs for `mamba_mixer2 assert`, `mamba prev_last_scheduled_idx`, `mamba cache mode spec decode` — no existing PR found. Related: #41233 (introduced the incomplete fix this PR completes). ## Test plan Requires a Mamba2-attention hybrid model implementing `SupportsMambaPrefixCaching` (NemotronH family) served with both `--enable-prefix-caching` and `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":N}'`: 1. Start vLLM. Find the effective `BLOCK_SIZE` in the startup log: `\"Setting attention block size to N tokens\"`. 2. Send a prompt of exactly `BLOCK_SIZE + 1` tokens. The server caches one full block. 3. Send the same prompt again while the server is otherwise idle. Without this fix: the assert fires immediately and all TP ranks crash. With this fix: the request succeeds. Verified on `vllm/vllm-openai:v0.22.0`, 4-way tensor parallel, NemotronH-class model. ## AI assistance disclosure This fix was developed with AI assistance (Claude Sonnet 4.6). The crash sequence, root cause analysis, and fix were verified by the human submitter against the live codebase and against a running deployment. Every changed line has been reviewed.", "url": "https://github.com/vllm-project/vllm/pull/46424", "createdAt": "2026-06-22T23:20:12Z", "updatedAt": "2026-08-13T17:47:48Z", "timestamp": "2026-08-13T17:47:48Z", "metrics": { "reactions": 2, "comments": 4 }, "labels": [ "bug", "ready", "v1" ], "author": "sytianhe", "state": "open", "assignees": [], "change": "new" } }, { "id": "event:f881f967df91e2b8dad9", "signalId": "github:vllm-project/vllm:pull_request:51338", "event": "changed", "observedAt": "2026-08-13T18:01:55.420671Z", "changedFields": [ "updatedAt" ], "signal": { "id": "github:vllm-project/vllm:pull_request:51338", "source": "github", "group": "ai-infrastructure", "project": "vllm-project/vllm", "kind": "pull_request", "title": "[RFC][Bug Fix][Spec Decode] Require explicit speculative methods", "text": "This PR removes speculative-method guessing and makes the method an explicit contract. ## Problem On current `main`, vLLM guesses the speculative method from model path/name substrings such as `eagle3`, `dflash`, and `dspark`. A DFlash or DSpark checkpoint at a neutral training path can therefore be treated as the generic autoregressive `draft_model`, while renaming or moving the same checkpoint changes serving behavior. For example: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{\"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15}' ``` Previously, this could silently select `draft_model`. In the original end-to-end Qwen3-8B + DFlash reproduction, the misdetected path produced acceptance length 1.02; adding explicit `\"method\": \"dflash\"` produced 3.88. Serving a specialized checkpoint through the wrong proposer is worse than requiring one extra field, so this PR now fails closed. ## Design - A method is required for every explicit speculative configuration. It can be supplied as `\"method\"` in `--speculative-config` or with `--spec-method`. - The explicit method selects the proposer and any structural loader normalization. vLLM does not replace or second-guess it from the model path, checkpoint architecture, structural `model_type`, or other metadata. - Checkpoint metadata is used only after method selection for implementation details, such as choosing the embedded DeepSeek-V4 DSpark loader for `method=\"dspark\"` and defaulting its proposal depth from `dspark_block_size`. - Known checkpoint schemas that explicitly declare their algorithm may populate the method before `SpeculativeConfig` is constructed. The speculators format uses `speculators_model_type`; PEagle is normalized to `method=\"eagle3\"` with parallel drafting. - `num_speculative_tokens` remains optional when the selected checkpoint format declares a proposal depth, such as speculators `speculative_tokens`, `n_predict`, or DSpark `dspark_block_size`. Otherwise the existing clear error remains. - DFlash and DSpark parallel-drafting behavior is derived from the explicit method. - Explicit shorthand values are folded into the config before checkpoint-declared defaults, so `--spec-method`, `--spec-model`, and `--spec-tokens` follow the same precedence rule as JSON fields. For an embedded DeepSeek-V4 checkpoint, the same checkpoint metadata is therefore handled according to the requested method: `draft_model` keeps the ordinary model loader, `mtp` uses `DeepSeekV4MTPModel`, and `dspark` uses `DSparkDraftModel`. The metadata does not choose among them. ## Breaking change Configurations that relied on implicit selection must add a method. Before: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` After: ```bash vllm serve Qwen/Qwen3-8B \\ --speculative-config '{ \"method\": \"dflash\", \"model\": \"/path/to/checkpoints/6\", \"num_speculative_tokens\": 15 }' ``` For a generic autoregressive draft model, use `\"method\": \"draft_model\"`. Legacy EAGLE/EAGLE3, Medusa, MLP-speculator, n-gram, and custom-proposer configurations must likewise state their method. A speculators-format checkpoint used as the top-level model still works without an explicit `--speculative-config`, because its schema declares both the algorithm and proposal depth. ## Validation - End-to-end reproduction recorded before this design change: - misdetected `draft_model`: acceptance length 1.02 - explicit `dflash`: acceptance length 3.88 - This PR now rejects the missing-method configuration instead of silently serving it autoregressively. - Explicit method, schema default, loader selection, and callable override tests: 18 passed. - Full draft HF override module: 11 passed. - DeepSeek EAGLE loader compatibility regression: 1 passed. - Earlier focused config, DSpark, engine-argument, scheduler, and structured-output suites: 50 passed. - Full `tests/test_config.py` before the final refinement: 179 passed; 2 failed only because this environment cannot access gated `meta-llama/Meta-Llama-3-8B-Instruct` (HTTP 403). - All pre-commit hooks passed on every changed file, including Ruff, mypy, Markdown lint, SPDX, DCO sign-off, and configuration validation. - GPU model evaluation was not rerun for this follow-up; the serving change is fail-fast configuration and loader-selection behavior, and the earlier DFlash measurement above is retained as the motivating model result.
Commands run ```bash .venv/bin/python -m pytest -q \\ tests/config/test_speculative_draft_hf_overrides.py \\ tests/test_config.py \\ -k 'method_selects_draft_override or callable_overrides_reach_the_draft_config or arch_mapping_applies_before_callable_override or composed_override_is_picklable or explicit_method_selects_deepseek_v4_loader or speculative_config_requires_method or speculators_format' .venv/bin/python -m pytest -q \\ 'tests/config/test_model_arch_config.py::test_draft_model_arch_config[eagle618/deepseek-v3-random-eagle618/eagle-deepseek-v3-random-eagle-True]' .venv/bin/python -m pytest -q tests/config/test_speculative_draft_hf_overrides.py .venv/bin/pre-commit run --files ```
## Contribution notes - This is not a duplicate of #47490: that PR preserves auto-detection and validates explicit-method mismatches; this PR removes automatic method selection and fails when an explicit configuration omits the method. - This is not a duplicate of #44566: that PR only propagates token depth from a speculators-format draft; this PR changes method-selection semantics while preserving declared token defaults. - This is not a duplicate of #52165: that PR detects embedded DeepSeek-V4 DSpark metadata and uses it for automatic routing; this PR makes the explicit method authoritative and uses metadata only to configure the selected loader. - AI assistance was used. I reviewed every changed line and ran the validation listed above.", "url": "https://github.com/vllm-project/vllm/pull/51338", "createdAt": "2026-08-07T00:53:10Z", "updatedAt": "2026-08-13T17:47:21Z", "timestamp": "2026-08-13T17:47:21Z", "metrics": { "reactions": 2, "comments": 6 }, "labels": [ "bug", "documentation", "new-model", "structured-output", "speculative-decoding", "rust" ], "author": "WindChimeRan", "state": "open", "assignees": [], "change": "updated" } } ] } }